Robust nonparametric tests of general linear model coefficients: A comparison of permutation methods and test statistics

Statistical inference in neuroimaging research often involves testing the significance of regression coefficients in a general linear model. In many applications, the researcher assumes a model of the form Y=α+Xβ+Zγ+ε, where Y is the observed brain signal, and X and Z contain explanatory variables that are thought to be related to the brain signal. The goal is to test the null hypothesis H₀:β=0 with the nuisance parameters γ included in the model. Several nonparametric (permutation) methods have been proposed for this problem, and each method uses some variant of the F ratio as the test statistic. However, recent research suggests that the F ratio can produce invalid permutation tests of H₀:β=0 when the ε terms are heteroscedastic (i.e., have non-constant variance), which can occur for a variety of reasons. This study compares the classic F test statistic to the robust W (Wald) test statistic using eight different permutation methods. The results reveal that permutation tests using the F ratio can produce accurate results when the errors are homoscedastic, but high false positive rates when the errors are heteroscedastic. In contrast, permutation tests using the W test statistic produced valid results when the errors were homoscedastic, and asymptotically valid results when the errors were heteroscedastic. In the situation with homoscedastic errors, permutation tests using the W statistic showed slightly reduced power compared to the F statistic, but the difference disappeared as the sample size n increased. Consequently, the W test statistic is recommended for robust nonparametric hypothesis tests of regression coefficients in neuroimaging research.