% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/performance_reliability.R
\name{performance_reliability}
\alias{performance_reliability}
\alias{performance_dvour}
\title{Random Effects Reliability}
\usage{
performance_reliability(x, ...)

performance_dvour(x, ...)
}
\arguments{
\item{x}{A model object.}

\item{...}{Currently not used.}
}
\description{
These functions provide information about the reliability of
group-level estimates (i.e., random effects) in mixed models. They are useful
to assess whether the predictors yield consistent group-level variability.
"Group-level" can refer, for instance, to different participants in a study,
and the predictors to the effect of some experimental condition.

The conceptually related functions are implemented,
\code{performance_reliability()}, based on Rouder & Mehrvarz (2024) that uses
estimated model variances, and \code{performance_dvour()} (d-vour), which
corresponds to the Variability-Over-Uncertainty Ratio ("vour") between random
effects coefficient variability and their associated uncertainty.

\strong{Note}: \code{performance_reliability()} requires to recompute the model to
estimate some of the variances of interest, which does not make it very
usable with Bayesian models. Please get in touch if you have would like to
help addressing this.
}
\details{
\subsection{Reliability (Signal-to-Noise Ratio)}{

\code{performance_reliability()} estimates the reliability of random effects
(intercepts and slopes) in mixed-effects models using variance decomposition.
This method follows the \strong{hierarchical modeling} framework of Rouder &
Mehrvarz (2024), defining reliability as the \strong{signal-to-noise variance
ratio}:

\deqn{\gamma^2 = \frac{\sigma_B^2}{\sigma_B^2 + \sigma_W^2}}

where:
\itemize{
\item \eqn{\sigma_B^2} is the \strong{between-subject variance} (i.e., variability
across groups).
\item \eqn{\sigma_W^2} is the \strong{within-subject variance} (i.e., trial-level
measurement noise).
}

This metric quantifies \strong{how much observed variability is due to actual
differences between groups}, rather than measurement error or within-group
fluctuations.

To account for \strong{trial count (\eqn{L})}, reliability is adjusted following:

\deqn{E(r) = \frac{\gamma^2}{\gamma^2 + 1/L}}

where \eqn{L} is the number of \strong{observations per random effect level} (note
that Rouder (2024) recommends 2/L to adjust for contrast effects).
}

\subsection{Variability-Over-Uncertainty Ratio (d-vour)}{

\code{performance_dvour()} computes an alternative reliability measure corresponding
to the normalized \strong{ratio of observed variability to uncertainty in random effect
estimates}. This is defined as:

\deqn{\text{D-vour} = \frac{\sigma_B^2}{\sigma_B^2 + \mu_{\text{SE}}^2}}

where:
\itemize{
\item \eqn{\sigma_B^2} is the \strong{between-group variability} (computed as the SD
of the random effect estimates).
\item \eqn{\mu_{\text{SE}}^2} is the \strong{mean squared uncertainty} in random
effect estimates (i.e., the average uncertainty).
}
\subsection{Interpretation:}{
\itemize{
\item \strong{D-vour > 0.75}: Strong group-level effects (between-group variance is at
least 3 times greater than uncertainty).
\item \strong{D-vour ~ 0.5}: Within-group and between-group variability are similar;
random effect estimates should be used with caution.
\item \strong{D-vour < 0.5}: Measurement noise dominates; random effect estimates are
probably unreliable.
}

While d-vour shares some similarity to Rouder's Reliability, it does not
explicitly model within-group trial-level noise and is only based on the
random effect estimates, and can thus be not accurate when there is not
a lot of random factor groups (the reliability of this index -
the meta-reliability - depends on the number of groups).
}

}
}
\examples{
\dontshow{if (all(insight::check_if_installed(c("lme4", "glmmTMB"), quietly = TRUE))) withAutoprint(\{ # examplesIf}
url <- "https://raw.githubusercontent.com/easystats/circus/refs/heads/main/data/illusiongame.csv"
df <- read.csv(url)

m <- lme4::lmer(RT ~ (1 | Participant), data = df)
performance_reliability(m)
performance_dvour(m)

m <- glmmTMB::glmmTMB(RT ~ (1 | Participant), data = df)
performance_reliability(m)
performance_dvour(m)

m <- lme4::lmer(RT ~ (1 | Participant) + (1 | Trial), data = df)
performance_reliability(m)
performance_dvour(m)

m <- glmmTMB::glmmTMB(RT ~ (1 | Participant) + (1 | Trial), data = df)
performance_reliability(m)
performance_dvour(m)

\donttest{
m <- lme4::lmer(
  RT ~ Illusion_Difference + (Illusion_Difference | Participant) + (1 | Trial),
  data = df
)
performance_reliability(m)
performance_dvour(m)

m <- glmmTMB::glmmTMB(
  RT ~ Illusion_Difference + (Illusion_Difference | Participant) + (1 | Trial),
  data = df
)
performance_reliability(m)
performance_dvour(m)
}
\dontshow{\}) # examplesIf}
}
\references{
\itemize{
\item Rouder, J. N., Pena, A. L., Mehrvarz, M., & Vandekerckhove, J. (2024). On
Cronbach’s merger: Why experiments may not be suitable for measuring
individual differences.
\item Rouder, J. N., & Mehrvarz, M. (2024). Hierarchical-model insights for
planning and interpreting individual-difference studies of cognitive
abilities. Current Directions in Psychological Science, 33(2), 128-135.
\item Williams, D. R., Mulder, J., Rouder, J. N., & Rast, P. (2021). Beneath the
surface: Unearthing within-person variability and mean relations with
Bayesian mixed models. Psychological methods, 26(1), 74.
\item Williams, D. R., Martin, S. R., DeBolt, M., Oakes, L., & Rast, P. (2020). A
fine-tooth comb for measurement reliability: Predicting true score and
error variance in hierarchical models.
}
}
