Creating advice-taking reinforcement learners

Richard Maclin, Jude W. Shavlik

Research output: Contribution to journalArticlepeer-review

109 Scopus citations

Abstract

Learning from reinforcements is a promising approach for creating intelligent agents However, reinforcement learning usually requires a large number of training episodes. We present and evaluate a design that addresses this shortcoming by allowing a connectionist Q-learner to accept advice given, at any time and in a natural manner, by an external observer. In our approach the advice giver watches the learner and occasionally makes suggestions, expressed as instructions in a simple imperative programming language. Based on techniques from knowledge-based neural networks, we insert these programs directly into the agent's utility function. Subsequent reinforcement learning further integrates and refines the advice. We present empirical evidence that investigates several aspects of our approach and shows that, given good advice, a learner can achieve statistically significant gains in expected reward. A second experiment shows that advice improves the expected reward regardless of the stage of training at which it is given, while another study demonstrates that subsequent advice can result in further gains in reward. Finally, we present experimental results that indicate our method is more powerful than a naive technique for making use of advice.

Original languageEnglish (US)
Pages (from-to)251-281
Number of pages31
JournalMachine Learning
Volume22
Issue number1-3
DOIs
StatePublished - 1996
Externally publishedYes

Keywords

  • Adaptive agents
  • Advice-giving
  • Knowledge-based neural networks
  • Learning from instruction
  • Neural networks
  • Q-learning
  • Reinforcement learning
  • Theory refinement

Fingerprint

Dive into the research topics of 'Creating advice-taking reinforcement learners'. Together they form a unique fingerprint.

Cite this