Control, Inference and Statistical Physics
Control, Inference and Statistical Physics
Path Integral Control Theory
Path Integral Control Theory
Path integral control is a method for solving stochastic optimal control problems by using a Monte Carlo (MC) approach to approximate the Hamilton-Jacobi-Bellman (HJB) equation. It transforms the value function into an expectation over all possible trajectories (a path integral) and then uses Monte Carlo simulations to find the optimal control. This approach can be applied to control problems with moderate to large dimensions by avoiding the need for a global grid of the HJB equation.
Presentations given by Dr. Bert Kappen
KL Control Theory and Decision Making under Uncertainty Video
An efficient approach to stochastic optimal control Video
Control as Variational Inference
Control as Variational Inference
Control as Inference by Sergey Levine Part 1, Part 2, Part 3, Part 4, Part 5
Readings
Todorov(2006) Linearly-solvable Markov decision problems, pdfTodorov(2008) General duality between optimal control and estimation pdfKappen(2009), Optimal control as a graphical model inference problem, pdfZiebart(2010), Modeling Interaction via the Principle of Maximum Causal Entropy, pdfRawlik, Toussaint, Vijayakumar (2013), On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference, pdfNachum, Norouzi, Xu, Schuurmans(2017), Bridging the Gap Between Value and Policy Based Reinforcement Learning, pdfSchulman, Chen, Abbeel(2017), Equivalence Between Policy Gradients and Soft Q-Learning, pdfHaarnoja, Zhou, Abbeel, Levine (2018), Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, pdfLevine(2018), Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review, pdfReadings