Using automatic speech recognition to assess spoken responses to cognitive tests of semantic verbal fluency

Serguei V.S. Pakhomov, Susan E. Marino, Sarah Banks, Charles Bernick

Research output: Contribution to journalArticlepeer-review

11 Scopus citations


Cognitive tests of verbal fluency (VF) consist of verbalizing as many words as possible in one minute that either start with a specific letter of the alphabet or belong to a specific semantic category. These tests are widely used in neurological, psychiatric, mental health, and school settings and their validity for clinical applications has been extensively demonstrated. However, VF tests are currently administered and scored manually making them too cumbersome to use, particularly for longitudinal cognitive monitoring in large populations. The objective of the current study was to determine if automatic speech recognition (ASR) could be used for computerized administration and scoring of VF tests. We examined established techniques for constraining language modeling to a predefined vocabulary from a specific semantic category (e.g., animals). We also experimented with post-processing ASR output with confidence scoring, as well as with using speaker adaptation to improve automated VF scoring. Audio responses to a VF task were collected from 38 novice and experienced professional fighters (boxing and mixed martial arts) participating in a longitudinal study of effects of repetitive head trauma on brain function. Word error rate, correlation with manual word count and distance from manual word count were used to compare ASR-based approaches to scoring to each other and to the manually scored reference standard. Our study's results show that responses to the VF task contain a large number of extraneous utterances and noise that lead to relatively poor baseline ASR performance. However, we also found that speaker adaptation combined with confidence scoring significantly improves all three metrics and can enable use of ASR for reliable estimates of the traditional manual VF scores.

Original languageEnglish (US)
Pages (from-to)14-26
Number of pages13
JournalSpeech Communication
StatePublished - Dec 1 2015

Bibliographical note

Funding Information:
Work on the automated verbal fluency assessment system was supported in part by Grants from the National Institutes of Health (NINDS – 5R01NS076665 ) and the Alzheimer’s Association ( DNCFI-12-242985 ). We also would like to thank James Ryan for helping with manual scoring of the audio responses on the VF task.


  • Automatic speech recognition
  • Cognitive testing
  • Confidence scoring
  • Speaker adaptation
  • Speech analysis
  • Verbal fluency


Dive into the research topics of 'Using automatic speech recognition to assess spoken responses to cognitive tests of semantic verbal fluency'. Together they form a unique fingerprint.

Cite this