Mixed-membership naive bayes models

Hanhuai Shan, Arindam Banerjee

Research output: Contribution to journalArticlepeer-review

24 Scopus citations

Abstract

In recent years, mixture models have found widespread usage in discovering latent cluster structure from data. A popular special case of finite mixturemodels is the family of naive Bayes (NB) models, where the probability of a feature vector factorizes over the features for any given component of the mixture. Despite their popularity, naive Bayes models do not allow data points to belong to different component clusters with varying degrees, i.e., mixed memberships, which puts a restriction on their modeling ability. In this paper, we propose mixed-membership naïve Bayes (MMNB) models. On one hand, MMNB can be viewed as a generalization of NB by putting a Dirichlet prior on top to allow mixed memberships. On the other hand, MMNB can also be viewed as a generalization of latent Dirichlet allocation (LDA) with the ability to handle heterogeneous feature vectors with different types of features, e.g., real, categorical, etc.. We propose two variational inference algorithms to learnMMNBmodels. The first one is based on ideas originally used in LDA, and the second one uses substantially fewer variational parameters, leading to a significantly faster algorithm. Further, we extend MMNB/LDA to discriminative mixed-membership models for classification by suitably combining MMNB/LDA with multi-class logistic regression. The efficacy of the proposed mixed-membership models is demonstrated by extensive experiments on several datasets, including UCI benchmarks, recommendation systems, and text datasets.

Original languageEnglish (US)
Pages (from-to)1-62
Number of pages62
JournalData Mining and Knowledge Discovery
Volume23
Issue number1
DOIs
StatePublished - Jul 2011

Bibliographical note

Funding Information:
Acknowledgements We want to warmly thank Nikunj Oza for valuable input on discriminative mixed membership models. The research was supported by National Science Foundation grants IIS-0812183, IIS-0916750, National Science Foundation CAREER grant IIS-0953274, and National Aeronautics and Space Administration grant NNX08AC36A.

Keywords

  • Generative models
  • Latent Dirichlet allocation
  • Logistic regression
  • Mixed-membership
  • Naive Bayes
  • Variational inference

Fingerprint

Dive into the research topics of 'Mixed-membership naive bayes models'. Together they form a unique fingerprint.

Cite this