TY - GEN
T1 - Online control basis selection by a regularized actor critic algorithm
AU - Yuan, Jianjun
AU - Lamperski, Andrew
PY - 2017/6/29
Y1 - 2017/6/29
N2 - Policy gradient algorithms are useful reinforcement learning methods which optimize a control policy by performing stochastic gradient descent with respect to controller parameters. In this paper, we extend actor-critic algorithms by adding an ℓ1 norm regularization on the actor part, which makes our algorithm automatically select and optimize the useful controller basis functions. Our method is closely related to existing approaches to sparse controller design and actuator selection, but in contrast to these, our approach runs online and does not require a plant model. In order to utilize ℓ1 regularization online, the actor updates are extended to include an iterative soft-thresholding step. Convergence of the algorithm is proved using methods from stochastic approximation. The effectiveness of our algorithm for control basis and actuator selection is demonstrated on numerical examples.
AB - Policy gradient algorithms are useful reinforcement learning methods which optimize a control policy by performing stochastic gradient descent with respect to controller parameters. In this paper, we extend actor-critic algorithms by adding an ℓ1 norm regularization on the actor part, which makes our algorithm automatically select and optimize the useful controller basis functions. Our method is closely related to existing approaches to sparse controller design and actuator selection, but in contrast to these, our approach runs online and does not require a plant model. In order to utilize ℓ1 regularization online, the actor updates are extended to include an iterative soft-thresholding step. Convergence of the algorithm is proved using methods from stochastic approximation. The effectiveness of our algorithm for control basis and actuator selection is demonstrated on numerical examples.
UR - http://www.scopus.com/inward/record.url?scp=85027052467&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85027052467&partnerID=8YFLogxK
U2 - 10.23919/ACC.2017.7963640
DO - 10.23919/ACC.2017.7963640
M3 - Conference contribution
AN - SCOPUS:85027052467
T3 - Proceedings of the American Control Conference
SP - 4448
EP - 4453
BT - 2017 American Control Conference, ACC 2017
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2017 American Control Conference, ACC 2017
Y2 - 24 May 2017 through 26 May 2017
ER -