Improved test-retest reliability criteria as applied to probability assessments

Test-retest reliability is a common indicator of response consistency. It is argued that using regression coefficients for detecting systematic response error is less appropriate than testing for shifts in the mean (median) and variance. This procedure is exemplified using probability response data. For this data, shifts in centrality were found to be about 2.5 times more likely than shifts in variability. The shifts in centrality did not favor any particular direction; however, variability tended to decrease over time in early sessions.