4篇前沿强化学习论文

HIERARCHICAL VISUOMOTOR CONTROL OF HUMANOIDS

JoshMerel∗, ArunAhuja∗, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess & Greg Wayne DeepMind London, UK{jsmerel,arahuja,vuph,stunya,liusiqi,dhruvat,

heess,gregwayne}@google.com

ABSTRACT

We aim to build complex humanoid agents that integrate perception, motor con- trol, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills in- formed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combin- ing pre-training of low-level motor controllers with a high-level, task-focused con- troller that switches among low-level sub-policies. The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. Supplementary video link1

NEURAL PROBABILISTIC MOTOR PRIMITIVES FOR HUMANOID CONTROL

Josh Merel∗, Leonard Hasenclever∗, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, & Nicolas HeessDeepMind London, UK{jsmerel,leonardh,agalashov,arahuja,vuph,

gregwayne,ywteh,heess}@google.com

ABSTRACT

We focus on the problem of learning a single motor module that can flexibly ex- press a range of behaviors for the control of high-dimensional physically simu- lated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural prob- abilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we calllinear feedback policy cloning. We encourage readers to view a supplementary video summarizing our results.

3

4

UNSUPERVISED CONTROL THROUGHNON-PARAMETRIC DISCRIMINATIVE REWARDS

David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen & Volodymyr Mnih DeepMind{dwf,tomvandewiele,tkulkarni,cdi,stevenhansen,vmnih}@google.com

ABSTRACT

Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually- specified goals using only a stream of observations and actions. Our agent simulta- neously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects simi- larity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains – Atari, the DeepMind Control Suite and DeepMind Lab.