Abstract
There is considerable interest today in the forecasting of conflict dynamics. Commonly, the root mean square error and other point metrics are used to evaluate the forecasts from such models. However, conflict processes are non-linear, so these point metrics often do not produce adequate evaluations of the calibration and sharpness of the forecast models. Forecast density evaluation improves the model evaluation. We review tools for density evaluation, including continuous rank probability scores, verification rank histograms, and sharpness plots. The usefulness of these tools for evaluating conflict forecasting models is explained. We illustrate this, first, in a comparison of several time series models' forecasts of simulated data from a Markov-switching process, and second, in a comparison of several models' abilities to forecast conflict dynamics in the Cross Straits. These applications show the pitfalls of relying on point metrics alone for evaluating the quality of conflict forecasting models. As in other fields, it is more useful to employ a suite of tools. A non-linear vector autoregressive model emerges as the model which is best able to forecast conflict dynamics between China and Taiwan.
| Original language | English (US) |
|---|---|
| Pages (from-to) | 944-962 |
| Number of pages | 19 |
| Journal | International Journal of Forecasting |
| Volume | 30 |
| Issue number | 4 |
| DOIs | |
| State | Published - Oct 2014 |
Bibliographical note
Funding Information:Earlier versions of this paper were presented at the Annual Meeting of the Midwest Political Science Association, Chicago, April 2012, and at the summer meeting of the Society for Political Methodology, Princeton University, July 2011. Drafts of this paper were also presented at colloquia at Princeton University, Texas A & M University, University of Pittsburgh, and University of Rochester. We thank the members of the Society for Political Methodology, participants at the colloquia, and especially Robert Erikson, Eleonora Mattiacci, Xun Pang, and Michael D. Ward for comments. This research is supported by the U.S. National Science Foundation , award numbers SES-0921051 , SES-0921018 , and SES-1004414 . The authors are responsible for the contents.
Funding Information:
Phillip A. Schrodt is Professor of Political Science at Pennsylvania State University. Prior to joining Penn State in 2010, he taught for 21 years at the University of Kansas, and 11 years at Northwestern University. His areas of research are quantitative models of political conflict and computational methodology. His current research focuses on predicting political change using statistical and pattern recognition methods, and has been supported by the National Science Foundation, the Defense Advanced Research Projects Agency, and the U.S. government’s Political Instability Task Force. He is a past president and current fellow of the Society for Political Methodology.
Keywords
- Bayesian
- Conflict dynamics
- Density evaluation
- Scoring rules
- Time series
- Verification rank histogram