Analyzing the results of Monte Carlo studies in item response theory

Monte Carlo studies in item response theory have been used in a number of ways, for example, to evaluate new parameter estimation procedures, to compare item analysis programs, and to study the effects of multidimensional data on parameter estimation. These studies typically rely on simple descriptive methods to analyze Monte Carlo results, implying that complex effects are unlikely to be detected or their magnitudes estimated. These problems are exacerbated when Monte Carlo studies lack an experimental design to guide the data analyses. The results from two Monte Carlo studies in item response theory are analyzed with inferential methods to illustrate the strengths of these procedures. It is recommended that researchers in item response theory employ both descriptive and inferential methods to analyze Monte Carlo results.