Convergence of simulation-based policy iteration

William L. Cooper, Shane G. Henderson, Mark E. Lewis

Research output: Contribution to journalArticlepeer-review

27 Scopus citations


Simulation-based policy iteration (SBPI) is a modification of the policy iteration algorithm for computing optimal policies for Markov decision processes. At each iteration, rather than solving the average evaluation equations, SBPI employs simulation to estimate a solution to these equations. For recurrent average-reward Markov decision processes with finite state and action spaces, we provide easily verifiable conditions that ensure that simulation-based policy iteration almostsurely eventually never leaves the set of optimal decision rules. We analyze three simulation estimators for solutions to the average evaluation equations. Using our general results, we derive simple conditions on the simulation run lengths that guarantee the almost-sure convergence of the algorithm.

Original languageEnglish (US)
Pages (from-to)213-234
Number of pages22
JournalProbability in the Engineering and Informational Sciences
Issue number2
StatePublished - 2003


Dive into the research topics of 'Convergence of simulation-based policy iteration'. Together they form a unique fingerprint.

Cite this