腾讯paper 模仿学习

3. 基于分批历史数据的指数加权模仿学习方法 Exponentially Weighted Imitation Learning for Batched Historical Data 这项研究由腾讯 AI Lab 独立完成，主要研究了仅使用成批量的历史数据的深度策略学习。这篇文章中我们主要考虑的是只使用历史数据的深度策略学习。这个问题的主要挑战在于，与大部分强化学习问题不同，我们不再有一个环境的模拟器来进行学习。为了解决这个问题，研究者提出一个单调优势加权的模仿学习算法来从历史数据中学习，并且可以应用到复杂非线性函数近似以及混合动作空间的问题中。这个方法并不依赖用来生成数据的行为策略的知识，所以可以被用来从一个未知的策略生成的数据中进行学习。在一些条件下，该算法（尽管非常简单）可以证明策略提升的下界，并且在实验中的效果超过了其它方法，并有望为复杂游戏 AI 提供更好的模仿学习。论文中也提供了详尽的数值实验来展示所提出的算法的有效性。

Abstract We consider deep policy learning with only batched historical trajectories. The main challenge of this problem is that the learner no longer has a simulator or “environment oracle” as in most reinforcement learning settings. To solve this problem, we propose a monotonic advantage reweighted imitation learning strategy that is applicable to problems with complex nonlinear function approximation and works well with hybrid (discrete and continuous) action space. The method does not rely on the knowledge of the behavior policy, thus can be used to learn from data generated by an unknown policy. Under mild conditions, our algorithm, though surprisingly simple, has a policy improvement bound and outperforms most competing methods empirically. Thorough numerical results are also provided to demonstrate the efficacy of the proposed methodology.

https://ray.readthedocs.io/en/latest/rllib-algorithms.html#advantage-re-weighted-imitation-learning-marwil