Research

A permutation-based random forest diagnostic for linear mixed models

Independent research · Sole author · Department of Statistics, University of Washington · 2025

A linear mixed model can be misspecified in several ways at once. Standard diagnostics are often aimed at one particular failure or depend heavily on what an analyst thinks to check. RandomForestSpecCheck takes a broader approach. It asks a random forest to predict the fitted model's residuals from the covariates, then uses a permutation test that respects the clustered design to decide whether the predictive signal is stronger than noise.

The result is a single screening test for systematic mean structure left in the residuals. The analyst specifies the variables to examine, but not the functional form of the possible misspecification.

The problem with checking a mixed model

Linear mixed models are used for correlated data such as repeated measurements on a subject, students within classrooms, or patients within clinics. They model that correlation directly. In doing so, they introduce several assumptions about the fixed effects, random effects, and residual covariance structure.

A model can be wrong in any of these places. Some failures bias the coefficient estimates. Others leave the estimates relatively stable but distort their uncertainty.

The usual diagnostic tools each see only part of the problem. Residual plots rely on visual judgment and can hide structure in a dense cloud of points. Information criteria compare only the models an analyst chose to fit. Formal tests usually target one alternative at a time. In each case, the analyst has to make some prior guess about what may be wrong.

The method

The procedure begins by fitting the mixed model and extracting its conditional residuals. Those residuals are then regressed on the predictors the analyst wants to examine using a random forest.

The forest is scored with its out-of-bag coefficient of determination,

R²_oob = 1 − MSE_oob / var(ê_ij)

where each tree is evaluated only on observations excluded from its own bootstrap sample. This matters because an in-sample forest can fit residual noise extremely well. Out-of-bag scoring gives a much more useful measure of whether the residual structure generalizes.

Under correct mean specification,

H₀: E[e | X] = 0   versus   H₁: E[e | X] ≠ 0

The analyst supplies the variables in X. The random forest is left to search for the functional form.

The sampling distribution of the statistic is not available in closed form, so the null is generated by permutation. Residuals are permuted within clusters rather than across the full dataset. This breaks the relationship between the residuals and predictors while preserving the clustered structure of the data.

The forest is refit B times to produce the permutation distribution.

Large observed values indicate residual structure. The procedure flags a model only when two conditions are met,

R²_oob > Q_0.975   and   R²_oob > τ

with τ = 0.05 by default.

The percentile criterion asks whether the observed value is unusual under the permutation null. The practical threshold asks whether the remaining structure is large enough to matter. This second condition is important in large samples, where a very small effect can be statistically detectable without being substantively useful.

The default value of τ is a judgment rather than a theoretical result. It is exposed as an argument for that reason.

The diagnostic, running

The panel below is a working browser implementation of the procedure. It simulates clustered longitudinal data with a quadratic time trend, fits a random-intercepts model that omits the quadratic term, grows a forest on the residuals, constructs the permutation null within clusters, and applies the decision rule.

Nothing is precomputed.

Running.

What the method detects, and what it does not

Four simulation studies were run. Each generated data under a known model violation and then fit a model that omitted it.

Three studies concern the conditional mean.

• an omitted quadratic fixed effect
• an omitted random slope
• unmodeled AR(1) correlation

The fourth concerns the variance. Residual variance increases with time while the fitted model assumes constant variance.

Each cell was replicated one hundred times.

For the three mean-structure violations, the diagnostic behaves as intended. False positives stay between one and three percent under correct specification. Power rises with the size of the omitted effect and with a stronger signal-to-noise ratio.

Large quadratic departures are detected in roughly eighty to one hundred percent of replicates. Weak departures are often missed. Omitted random slopes show a similar pattern, with additional dependence on cluster size. Unmodeled autocorrelation is the most difficult of the three and reaches only about one quarter of replicates at ρ = 0.6.

Heteroscedasticity is different. It is essentially never flagged.

That result follows from the statistic itself. The forest predicts the residual, not its magnitude. If the conditional variance changes while the conditional mean remains zero, there is no mean signal for the forest to recover.

Application to the Framingham Heart Study

The diagnostic was applied to a subset of the Framingham Heart Study containing more than 2,600 cholesterol measurements taken at two-year intervals over ten years. Baseline age, sex, and body mass index were included.

Five models were fit. Four were plausible specifications an analyst might reasonably consider. The fifth deliberately omitted time from the fixed effects even though time is known to matter biologically.

The intended separation is therefore clear. The four reasonable models are not flagged. The deliberately misspecified model is.

Residual-versus-fitted plots do not make that distinction nearly as clearly. All five produce dense clouds centered around zero. The misspecification is in the mean structure, and the forest is more sensitive to that structure than the visual residual check.

There is also a less comfortable result. All five observed R² values exceed their own permutation 97.5th percentiles. The percentile criterion alone separates nothing in this application.

The practical threshold is what distinguishes the sabotaged model. This is important because the permutation null is reliably negative when the shuffled residuals contain no predictive signal. On a dataset of this size, almost any real structure can clear the percentile criterion. The operating behavior is therefore strongly influenced by τ.

An analyst should choose that threshold with the scale of a meaningful effect in mind rather than accept the default automatically.

Limitations

The procedure detects conditional mean structure only. Power against subtle effects is modest and depends on the number of clusters, the observations within each cluster, and the noise level.

A null result in a small or noisy study is therefore weak evidence that the model is adequate.

The forest can only detect structure in variables it is given. A relevant omitted variable that never enters the random forest is invisible to the diagnostic.

The within-cluster permutation also creates a specific blind spot. A wholly omitted cluster-level effect will not be flagged because the permutation preserves cluster membership. That failure is often visible through simpler diagnostics, but it still has to be checked elsewhere.

RandomForestSpecCheck is best understood as a screening instrument. It is a useful first question about remaining mean structure, not a final verdict on the model.

The function

The procedure is packaged as a single R function over ranger and dplyr. It takes a fitted model, the data, a formula specifying the predictors to search, the number of permutations, and the practical threshold.

RandomForestSpecCheck(
  fitted_model = my_lmm,
  data         = my_data,
  formula_rf   = resid_model ~ time + cluster,
  B            = 500,
  threshold    = 0.05
)

RandomForestSpecCheck: Random Forest Model Specification Diagnostic for LMMs

Practical threshold:                        0.05
Observed R-squared:                         0.191
Permutation mean R-squared and 95% CI:     -0.009 (-0.011, -0.006)

Result: Model likely misspecified

Interpretation: The observed R-squared exceeds both the
upper bound of the null distribution and the practical
threshold, indicating that the model leaves systematic
structure in the residuals. This suggests that the model
may be misspecified.

The output is intentionally descriptive. A bare logical result would encourage the test to be treated as a verdict.

A flag means that the named predictors still contain information about the residuals. The next step is to inspect the forest's variable-importance measures, fit a plausible parametric extension, and see whether the signal disappears.

Contribution

Permutation inference and tree ensembles are established methods. Machine-learning approaches to residual analysis have also been explored for ordinary and generalized linear models.

The contribution here is their combination in a mixed-model setting. The procedure uses a within-cluster permutation scheme, an out-of-bag random-forest statistic with an interpretable scale, and a two-part decision rule that separates statistical evidence from practical magnitude.

The simulation studies also map where the method has power and where it does not. The negative result for pure variance misspecification is part of that scope rather than something to hide.

The work was written for submission to Computational Statistics & Data Analysis.

← All statistical work