专栏首页CreateAMindA Theory of State Abstraction for Reinforcement Learning

A Theory of State Abstraction for Reinforcement Learning

A Theory of State Abstraction for Reinforcement Learning

David Abel Department of Computer Science Brown University david_abel@brown.edu

Abstract

Reinforcement learning presents a challenging problem: agents must generalize experiences, efficiently explore the world, and learn from feedback that is delayed and often sparse, all while making use of a limited computational budget. Abstraction is essential to all of these endeavors. Through abstraction, agents can form concise models of both their surroundings and behavior, supporting effective decision making in diverse and complex environments. To this end, the goal of my doctoral research is to characterize the role abstraction plays in reinforcement learning, with a focus on state abstraction. I offer three desiderata articulating what it means for a state abstraction to be useful, and introduce classes of state abstractions that provide a partial path toward satisfying these desiderata. Collectively, I develop theory for state abstractions that can 1) preserve near-optimal behavior, 2) be learned and computed efficiently, and 3) can lower the time or data needed to make effective decisions. I close by discussing extensions of these results to an information theoretic paradigm of abstraction, and an extension to hierarchical abstraction that enjoys the same desirable properties.

1 Introduction

The focus of my doctoral research is on clarifying the representational practices that underlie effective Reinforcement Learning (RL), drawing on Information Theory, Computational Complexity, and Computational Learning Theory. The guiding question of my research is: “How do intelligent agents come up with the right abstract understanding of the worlds they inhabit?”, as pictured in Figure 1. I study this question by isolating and addressing its simplest unanswered forms through a mixture of theoretical analysis and experimentation.

My interest in this question stems from its foundational role in many aspects of learning and decision making: agents can’t model everything in their environment, but must necessarily pick up on something about their surroundings in order to explore, plan far into the future, generalize, solve credit assignment, communicate, and efficiently solve problems. Abstraction is essential to all of these endeavors: through abstraction, agents can construct models of both their surroundings and behavior that are compressed and useful. The

本文分享自微信公众号 - CreateAMind(createamind)

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2019-01-12

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • 论文解读:主视觉大脑皮层的深度层级模型:机器视觉可以从中学到些什么?

    、论文:Deep Hierarchies in the Primate Visual Cortex: What Can We Learn for Compute...

    用户1908973
  • State Abstraction as 压缩 in Apprenticeship Learning

    State Abstraction as Compression in Apprenticeship Learning https://github.com/d...

    用户1908973
  • STCN

    https://www.arxiv-vanity.com/papers/1902.06568/

    用户1908973
  • 信息访问悖论:共享时代的日益孤立(CS CAS)

    Twitter,Instagram和YouTube等现代在线媒体使任何人都可以成为信息生产者,并提供在线内容供潜在的全球消费。通过增加全球可访问的实时信息的数量...

    时代在召唤
  • 测量异构信息网络的多样性(CS AI)

    多样性是一个与许多研究领域相关的概念,从生态学到信息论,再到经济学,举几个例子。这个概念在信息检索、网络分析和人工神经网络社区中得到了越来越多的关注。虽然在网络...

    用户6853689
  • 通过多摄像头域自适应进行无监督车辆计数(CS CV)

    监测城市中的车辆流量是改善城市环境和市民生活质量的关键问题。图像是感知和评估大面积车辆流动的最佳传感方式。目前图像中的车辆计数技术依赖于大量的注释数据,随着新的...

    刘持诚
  • Science《科学》| 澳大利亚新感染艾滋病人数继续稳步下降

    Australia continues to see steady drop in new HIV infections

    HuangWeiAI
  • Woocommerce Trends 2020

    Top Woocommerce Trends To Follow In 2020. If you have an online store and missed...

    用户4822892
  • 从本体感知到新型环境中的长距离规划:分层RL模型(CS AI)

    为了使智能代理能够在复杂环境中灵活高效地进行操作,它们必须能够在时间,空间和概念抽象的多个级别进行推理。在较低级别,代理必须解释其本体感受输入并控制其肌肉,而在...

    刘子蔚
  • AllenNLP系列文章之四:指代消解

    指代消解是自然语言处理的一大任务之一,它是信息抽取不可或缺的组成部分。在信息抽取中,由于用户关心的事件和实体间语义关系往往散布于文本的不同位置,其中涉及到的实体...

    sparkexpert

扫码关注云+社区

领取腾讯云代金券