提升100倍速度的切片递归神经网络

AI 研习社论文推荐版块已经正式上线，欢迎大家前往社区论文板块查阅喔~

http://ai.yanxishe.com/page/paper

Sliced Recurrent Neural Networks

（切片递归神经网络）

Yu Zeping/Liu Gongshen

推荐原因

RNN训练慢、训练困难的问题已经是老生常谈了，循环结构带来的跨越多个步骤时的梯度消失和难以并行的特点几乎被认为是不可克服的，人们也已经接受了“RNN就是这样的”。这篇来自上海交通大学刘功申团队的分片RNN论文就提出了一种新的途径，通过分片的方式极大地提升了RNN的并行性，不仅可以只增加很少的参数数量就增加高维信息提取的能力，更在训练速度上相比传统RNN取得了超过100倍的提升。

来自AI研习社用户@杨 晓凡的推荐

摘要

Recurrent neural networks have achieved great success in many NLP tasks. However, they have difficulty in parallelization because of the recurrent structure, so it takes much time to train RNNs. In this paper, we introduce sliced recurrent neural networks (SRNNs), which could be parallelized by slicing the sequences into many subsequences. SRNNs have the ability to obtain high-level information through multiple layers with few extra parameters. We prove that the standard RNN is a special case of the SRNN when we use linear activation functions. Without changing the recurrent units, SRNNs are 136 times as fast as standard RNNs and could be even faster when we train longer sequences. Experiments on six largescale sentiment analysis datasets show that SRNNs achieve better performance than standard RNNs.