Simulation-based evaluation of the performance of the F test in a linear multilevel model setting with sparseness at the level of the primary unit

Abstract

In a linear multilevel model, significance of all fixed effects can be determined using Ftests under maximum likelihood (ML) or restricted maximum likelihood (REML). In this paper, we demonstrate that in the presence of primary unit sparseness, the performance of the Ftest under both REML and ML is rather poor. Using simulations based on the structure of a data example on ceftriaxone consumption in hospitalized children, we studied variability, typeI error rate and power in scenarios with a varying number of secondary units within the primary units. In general, the variability in the estimates for the effect of the primary unit decreased as the number of secondary units increased. In the presence of singletons (i.e., only one secondary unit within a primary unit), REML consistently outperformed ML, although even under REML the performance of the Ftest was found inadequate. When modeling the primary unit as a random effect, the power was lower while the typeI error rate was unstable. The options of dropping, regrouping, or splitting the singletons could solve either the problem of a high typeI error rate or a low power, while worsening the other. The permutation test appeared to be a valid alternative as it outperformed the Ftest, especially under REML. We conclude that in the presence of singletons, one should be careful in using the Ftest to determine the significance of the fixed effects, and propose the permutation test (under REML) as an alternative.