专栏首页CreateAMind强化学习框架 IMPALA 介绍

强化学习框架 IMPALA 介绍



1. Abstract

In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in single-machine training but also scales to thousands of machines without sacrificing data efficiency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction

method called V-trace.

The source code is publicly available at github.com/deepmind/scalable_agent.

2. Introduction

Unlike the popular A3C-based agents, in which workers communicate gradients with respect to the parameters of the policy to a central parameter server, IMPALA actors communicate trajectories of experience (sequences of states, actions, and rewards) to a centralised learner. Since the learner in IMPALA has access to full trajectories of experience we use a GPU to perform updates on mini-batches of trajectories while aggressively parallelising all time independent operations. This type of decoupled architecture can achieve very high throughput.

IMPALA (Figure 1) uses an actor-critic setup to learn a policy π and a baseline function Vπ . The process of generating experiences is decoupled from learning the parameters of π and Vπ . The architecture consists of a set of actors, repeatedly generating trajectories of experience, and one or more learners that use the experiences sent from actors to learn π off-policy.

At the beginning of each trajectory, an actor updates its own local policy μ to the latest learner policy π and runs it for n steps in its environment. After n steps, the actor sends the trajectory of states, actions and rewards x1 , a1 , r1 , . . . , xn , an , rn together with the corresponding policy distributions μ(at |xt ) and initial LSTM state to the learner through a queue. The learner then continuously updates its policy π on batches of trajectories, each collected from many actors. This simple architecture enables the learner(s) to be accelerated using GPUs and actors to be easily distributed across many machines.

However, because the policy used to generate a trajectory can lag behind the policy on the learner by several updates at the time of gradient calculation, learning becomes off-policy. Therefore, we introduce the V-trace off-policy actor-critic algorithm to correct for this harmful discrepancy.

With the scalable architecture and V-trace combined, IMPALA achieves exceptionally high data throughput rates of 250,000 frames per second, making it over 30 times faster than single-machine A3C. Crucially, IMPALA is also more data efficient than A3C based agents and more robust to hyperparameter values and network architectures, allowing it to make better use of deeper neural networks. We demonstrate the effectiveness of IMPALA by training a single agent on multi-task problems using DMLab-30, a new challenge set which consists of 30 diverse cognitive tasks in the 3D DeepMind Lab (Beattie et al., 2016) environment and by training a single agent on all games in the Atari-57 set of tasks.

3. V-trace target

4. V-trace actor-critic algorithm

5. Experimental results

本文分享自微信公众号 - CreateAMind(createamind),作者:kangaroo

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。




0 条评论
登录 后参与评论


  • 开源ALNS 自适应大邻域搜索(Adaptive Large Neighborhood Search)

    This package offers a general, well-documented and tested implementation of the ...

  • deepmind 做通用人工智能的思路

    Automated discovery of early visual concepts from raw image data is a major open...

  • vae 相关论文 表示学习 1

    05 Nov 2016 (modified: 18 Apr 2017)ICLR 2017 conference submissionReaders: Ever...

  • AlexNet论文总结

    Introduction Preconditions ImageNet Objects in realistic settings exhibit cons...

  • 【论文推荐】最新5篇行人再识别(ReID)相关论文—迁移学习、特征集成、重排序、 多通道金字塔、深层生成模型

    【导读】专知内容组整理了最近五篇行人再识别(Person Re-identification)相关文章,为大家进行介绍,欢迎查看! 1.Unsupervised...

  • GAN和微分轨迹栅格化:改进鸟瞰模型中交通参与者的运动预测(CS RO)


  • 部署机器学习以帮助数字人性化:提高OpenStreetMap中的图像注释效率(CS HC)


  • 亚马逊DRKG使用体验

    基于文章:探索「老药新用」最短路径:亚马逊AI Lab开源大规模药物重定位知识图谱DRKG,记录了该项目的实际部署与探索过程,供参考。

  • 【论文推荐】最新6篇图像分割相关论文—隐马尔可夫随机场、级联三维全卷积、信号处理、全卷积网络、多源域适应、循环分割

    【导读】专知内容组整理了最近六篇图像分割(Image Segmentation)相关文章,为大家进行介绍,欢迎查看! 1.Combination of Hidd...

  • 蝙蝠式在结构上栖息动作的仿生研究:惯性动力学(CS RO)