Commingled samples: A neglected source of bias in reliability analysis

Reliability is a property of test scores from individuals who have been sampled from a well-defined population. Reliability indices, such as coefficient α and related formulas for internal consistency reliability (KR-20, Hoyt's reliability), yield lower bound reliability estimates when (a) subjects have been sampled from a single population and when (b) test items are congeneric (i.e., when items are sampled from a single latent dimension). However, when samples are commingled-that is, when they are composed of scores that are drawn from multiple populations- coefficient α and related indices can be severely biased. In most cases the bias inflates α; in other cases α is attenuated. Equations are derived for elucidating this bias in two-group mixture distributions.