专栏首页CreateAMindLearning to Remember More with Less Memorization

Learning to Remember More with Less Memorization

Learning to Remember More with Less Memorization

Hung Le, Truyen Tran, Svetha Venkatesh

(Submitted on 5 Jan 2019 (v1), last revised 20 Mar 2019 (this version, v2))

Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.

https://github.com/thaihungle/UW-DNC

本文分享自微信公众号 - CreateAMind(createamind)

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2019-05-23

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • A Tutorial on Energy-Based Learning

    Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu Jie Huang T...

    用户1908973
  • Auto-Encoding GAN

    Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, Shakir Mohamed

    用户1908973
  • Model-Based 两篇paper

    Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based C...

    用户1908973
  • 【重磅】DeepMind AlphaGo团队获得Minsky Medal(“人工智能界的诺贝尔奖”)

    【导读】继Google DeepMind在Nature上发表最新论文AlphaGo Zero的重磅消息,今天 AlphaGo 团队被国际人工智能会议(IJCAI...

    WZEARW
  • 工业机器人协同安全保障面临的挑战(cs)

    中文摘要:系统安全和网络安全等相关关键属性的协同保障是关键系统工程中最棘手的挑战之一。在本章中,我们总结了协调保障安全和安全的方法。然后,我们从安全和安全的角度...

    用户7454091
  • 应用于人工智能学习价值的动态模型(CS)

    人工智能 (AI) 开发专家预测,智能系统和代理开发的进步将重塑我们社会的重要领域。然而,如果不谨慎、批判、反省地取得这样的进步,可能会给人类带来负面的结果。因...

    孙孙孙
  • 土耳其命名实体识别中最新神经序列标签模型的评估(CS CL)

    命名实体识别(NER)是一项经过广泛研究的任务,用于提取文本中的命名实体并对其进行分类。NER不仅在下游语言处理应用程序(例如关系提取和问题解答)中至关重要,而...

    刘子蔚
  • 【论文推荐】最新五篇视频分类相关论文—细粒度行人识别、群组归一化、MLtuner、时序特征

    WZEARW
  • 【论文推荐】最新六篇强化学习相关论文—Sublinear、机器阅读理解、加速强化学习、对抗性奖励学习、人机交互

    WZEARW
  • 【NDN基础】Named Data Networking 学习笔记

    版权声明:本文为博主原创文章,转载请注明出处。 https://blog.csdn.net/gongxifacai_believe/artic...

    魏晓蕾

扫码关注云+社区

领取腾讯云代金券