Abstract
Topic detection (TD) is a fundamental research issue in the Topic Detection and Tracking (TDT) community with practical implications; TD helps analysts to separate the wheat from the chaff among the thousands of incoming news streams. In this paper, we propose a simple and effective topic detection model called the temporal Discriminative Probabilistic Model (DPM), which is shown to be theoretically equivalent to the classic vector space model with feature selection and temporally discriminative weights. We compare DPM to its various probabilistic cousins, ranging from mixture models like von-Mises Fisher (vMF) to mixed membership models like Latent Dirichlet Allocation (LDA). Benchmark results on the TDT3 data set show that sophisticated models, such as vMF and LDA, do not necessarily lead to better results; in the case of LDA, notably worst performance was obtained under variational inference, which is likely due to the significantly large number of LDA model parameters involved for document-level topic detection. On the contrary, using a relatively simple time-aware probabilistic model such as DPM suffices for both offline and online topic detection tasks, making DPM a theoretically elegant and effective model for practical topic detection.
Original language | English (US) |
---|---|
Article number | 5374412 |
Pages (from-to) | 1795-1808 |
Number of pages | 14 |
Journal | IEEE Transactions on Pattern Analysis and Machine Intelligence |
Volume | 32 |
Issue number | 10 |
DOIs | |
State | Published - 2010 |
Externally published | Yes |
Bibliographical note
Funding Information:The authors thank the anonymous reviewers for their keen insight and valuable feedback. This research was supported in part by US National Science Foundation (NSF) Grant IIS-0812183.
Keywords
- DPM
- TFIDF.
- Topic detection
- bursty feature
- online
- probabilistic model
- time-aware