seq2seq模型之raw_rnn

本文是seq2seq模型的第二篇,主要是通过raw_rnn来实现seq2seq模型。 github地址是:https://github.com/zhuanxuhit/nd101 原文地址:https://github.com/ematvey/tensorflow-seq2seq-tutorials/blob/master/2-seq2seq-advanced.ipynb

import numpy as np
import tensorflow as tf
import helper

tf.reset_default_graph()
sess = tf.InteractiveSession()
PAD = 0
EOS = 1
# UNK = 2
# GO  = 3

vocab_size = 10
input_embedding_size = 20

encoder_hidden_units = 20
decoder_hidden_units = encoder_hidden_units * 2
encoder_inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name='encoder_inputs')

encoder_inputs_length = tf.placeholder(shape=(None,), dtype=tf.int32, name='encoder_inputs_length')

decoder_targets = tf.placeholder(shape=(None, None), dtype=tf.int32, name='decoder_targets')
embeddings = tf.Variable(tf.truncated_normal([vocab_size, input_embedding_size], mean=0.0, stddev=0.1), dtype=tf.float32)
encoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)

encoder

此处定义的encoder和第一篇中的不同,需要注意

from tensorflow.contrib.rnn import LSTMCell, LSTMStateTuple
encoder_cell = LSTMCell(encoder_hidden_units)

关于LSTMCell和 BasicLSTMCell 的区别,这个可以去tf的官网看说明:

It(BasicLSTMCell) does not allow cell clipping, a projection layer, and does not use peep-hole connections: it is the basic baseline.

这个projection layer就是在输出ouput上加了一层fc,将其作为lstm的输出

cell clipping ?

peep-hole connections?

((encoder_fw_outputs,
  encoder_bw_outputs),
 (encoder_fw_final_state,
  encoder_bw_final_state)) = (
    tf.nn.bidirectional_dynamic_rnn(cell_fw=encoder_cell,
                                    cell_bw=encoder_cell,
                                    inputs=encoder_inputs_embedded,
                                    sequence_length=encoder_inputs_length,
                                    dtype=tf.float32, time_major=True)
    )

此处 encoder_cell 的形状是:[max_time, batch_size, ...],

encoder_fw_outputs 的形状是:[max_time, batch_size, cell_fw.output_size]

sequence_length:如果没有提供,则默认长度就是 [0,max_time-1],如果提供了则取 [0,sequence_length-1]

state 的形状是 : [batch_size, cell_fw.state_size]

encoder_fw_outputs
<tf.Tensor 'bidirectional_rnn/fw/fw/TensorArrayStack/TensorArrayGatherV3:0' shape=(?, ?, 20) dtype=float32>
encoder_bw_outputs
<tf.Tensor 'ReverseSequence:0' shape=(?, ?, 20) dtype=float32>
encoder_fw_final_state
LSTMStateTuple(c=<tf.Tensor 'bidirectional_rnn/fw/fw/while/Exit_2:0' shape=(?, 20) dtype=float32>, h=<tf.Tensor 'bidirectional_rnn/fw/fw/while/Exit_3:0' shape=(?, 20) dtype=float32>)
encoder_cell.state_size
LSTMStateTuple(c=20, h=20)
  • encoder_fw_final_state.h is activations of hidden layer of LSTM cell
  • encoder_fw_final_state.c is final output, which can potentially be transfromed with some wrapper

此处 h 指的是内部的状态,而 c 则是 h 经过 activations 后的状态

encoder_outputs = tf.concat((encoder_fw_outputs, encoder_bw_outputs), 2)

encoder_final_state_c = tf.concat(
    (encoder_fw_final_state.c, encoder_bw_final_state.c), 1)

encoder_final_state_h = tf.concat(
    (encoder_fw_final_state.h, encoder_bw_final_state.h), 1)

encoder_final_state = LSTMStateTuple(
    c=encoder_final_state_c,
    h=encoder_final_state_h
)

在这个例子中,我们不会舍去 encoder_outputs,而是会将其用于attention机制

decoder

decoder_cell = LSTMCell(decoder_hidden_units)
encoder_max_time, batch_size = tf.unstack(tf.shape(encoder_inputs))

下一步我们要要决定 decoder 运行多少步后结束,有两种策略

  • Stop after specified number of unrolling steps
  • Stop after model produced

此处我们选择第一种,固定的步骤,在之前的一些tutorial中一般是len(encoder_input)+10,此处我们测试用则简单点使用len(encoder_input)+2

decoder_lengths = encoder_inputs_length + 3
# +2 additional steps, +1 leading <EOS> token for decoder inputs

Output projection

decoder的预测流程是:

output(t) -> output projection(t) -> prediction(t) (argmax) -> input embedding(t+1) -> input(t+1)

我们先指定输出projection的W和b

W = tf.Variable(tf.truncated_normal([decoder_hidden_units, vocab_size], 0, 0.1), dtype=tf.float32)
b = tf.Variable(tf.zeros([vocab_size]), dtype=tf.float32)

Decoder via tf.nn.raw_rnn

使用dynamic_rnn有一些限制,不能让我们自定义输入,像下图一样

raw_rnn

图片来自: http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/

assert EOS == 1 and PAD == 0

eos_time_slice = tf.ones([batch_size], dtype=tf.int32, name='EOS')
pad_time_slice = tf.zeros([batch_size], dtype=tf.int32, name='PAD')

eos_step_embedded = tf.nn.embedding_lookup(embeddings, eos_time_slice)
pad_step_embedded = tf.nn.embedding_lookup(embeddings, pad_time_slice)

对于标准的 tf.nn.dynamic_rnn ,其输入 (t, ..., t+n) 需要事先作为一个 Tensor 输入,其动态 "Dynamic" 的含义是 n 的大小在每个batch中可以改变

此时如果我们希望一些更加复杂的机制,如每个cell的输出作为下一个的输入,或者实现 soft attention ,就没有办法了,这个时候我们就转向 tf.nn.raw_rnn 函数了

tf.nn.raw_rnn 最重要的就是 loop_fn 函数的编写,loop_fn做了一个映射

(time, previous_cell_output, previous_cell_state, previous_loop_state) -> (elements_finished, input, cell_state, output, loop_state).

上面转换的时机是在调用rnncell之前,准备好输入

loop_fn调用的时机有2个:

  1. Initial call at time=0 to provide initial cell_state and input to RNN.
  2. Transition call for all following timesteps where you define transition between two adjacent steps.

下面分别定义

def loop_fn_initial():
    initial_elements_finished = (0 >= decoder_lengths)  # all False at the initial step
    initial_input = eos_step_embedded
    initial_cell_state = encoder_final_state
    initial_cell_output = None
    initial_loop_state = None  # we don't need to pass any additional information
    return (initial_elements_finished,
            initial_input,
            initial_cell_state,
            initial_cell_output,
            initial_loop_state)
# (time, previous_cell_output, previous_cell_state, previous_loop_state) -> 
#     (elements_finished, input, cell_state, output, loop_state).
def loop_fn_transition(time, previous_output, previous_state, previous_loop_state):

    def get_next_input():
        output_logits = tf.add(tf.matmul(previous_output, W), b) # projection layer
        # [batch_size, vocab_size]
        prediction = tf.argmax(output_logits, axis=1)
        next_input = tf.nn.embedding_lookup(embeddings, prediction)
        # [batch_size, input_embedding_size]
        return next_input
    
    elements_finished = (time >= decoder_lengths) # this operation produces boolean tensor of [batch_size]
                                                  # defining if corresponding sequence has ended

    finished = tf.reduce_all(elements_finished) # -> boolean scalar
    input = tf.cond(finished, lambda: pad_step_embedded, get_next_input)
    # input shape [batch_size,input_embedding_size]
    state = previous_state
    output = previous_output
    loop_state = None

    return (elements_finished, 
            input,
            state,
            output,
            loop_state)

上面我们分别定义了两个loop_fn,下面我们会将其合并为一个

def loop_fn(time, previous_output, previous_state, previous_loop_state):
    if previous_state is None:    # time == 0
        assert previous_output is None and previous_state is None
        return loop_fn_initial()
    else:
        return loop_fn_transition(time, previous_output, previous_state, previous_loop_state)

decoder_outputs_ta, decoder_final_state, _ = tf.nn.raw_rnn(decoder_cell, loop_fn)
decoder_outputs = decoder_outputs_ta.stack()
decoder_outputs # hidden_size = 40
<tf.Tensor 'TensorArrayStack/TensorArrayGatherV3:0' shape=(?, ?, 40) dtype=float32>

为了对输出做最后的projection操作,我们需要reshape操作

[max_steps, batch_size, hidden_dim] to [max_steps*batch_size, hidden_dim]

decoder_max_steps, decoder_batch_size, decoder_dim = tf.unstack(tf.shape(decoder_outputs))
decoder_outputs_flat = tf.reshape(decoder_outputs, (-1, decoder_dim))
decoder_logits_flat = tf.add(tf.matmul(decoder_outputs_flat, W), b)
decoder_logits = tf.reshape(decoder_logits_flat, (decoder_max_steps, decoder_batch_size, vocab_size))
decoder_prediction = tf.argmax(decoder_logits, 2)

Optimizer

rnn的输出shape是:[max_time, batch_size, hidden_units], 通过一个FC(projection layer)变换为[max_time, batch_size, vocab_size],vocab_size是固定的,max_time and batch_size 是动态的

print(decoder_targets)
Tensor("decoder_targets:0", shape=(?, ?), dtype=int32)
print(tf.one_hot(decoder_targets, depth=vocab_size, dtype=tf.float32))
Tensor("one_hot:0", shape=(?, ?, 10), dtype=float32)
stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
    labels=tf.one_hot(decoder_targets, depth=vocab_size, dtype=tf.float32),
    logits=decoder_logits,
)

loss = tf.reduce_mean(stepwise_cross_entropy)
train_op = tf.train.AdamOptimizer().minimize(loss)
sess.run(tf.global_variables_initializer())

模型训练

跟第一篇中的训练一致

batch_size = 100

batches = helper.random_sequences(length_from=3, length_to=8,
                                   vocab_lower=2, vocab_upper=10,
                                   batch_size=batch_size)

print('head of the batch:')
for seq in next(batches)[:10]:
    print(seq)
head of the batch:
[3, 8, 9, 4, 9, 4]
[2, 3, 7, 2, 5, 2]
[4, 8, 8, 2]
[9, 6, 2, 5, 8, 3, 5, 3]
[9, 7, 2, 6]
[7, 6, 2]
[4, 9, 6, 6]
[8, 4, 7, 8, 9, 8, 7]
[4, 3, 6, 7, 4, 2]
[6, 5, 8]
def next_feed():
    batch = next(batches)
    encoder_inputs_, encoder_input_lengths_ = helper.batch(batch)
    decoder_targets_, _ = helper.batch(
        [(sequence) + [EOS] + [PAD] * 2 for sequence in batch]
    )
    return {
        encoder_inputs: encoder_inputs_,
        encoder_inputs_length: encoder_input_lengths_,
        decoder_targets: decoder_targets_,
    }
loss_track = []
max_batches = 3001
batches_in_epoch = 1000

try:
    for batch in range(max_batches):
        fd = next_feed()
        _, l = sess.run([train_op, loss], fd)
        loss_track.append(l)

        if batch == 0 or batch % batches_in_epoch == 0:
            print('batch {}'.format(batch))
            print('  minibatch loss: {}'.format(sess.run(loss, fd)))
            predict_ = sess.run(decoder_prediction, fd)
            for i, (inp, pred) in enumerate(zip(fd[encoder_inputs].T, predict_.T)):
                print('  sample {}:'.format(i + 1))
                print('    input     > {}'.format(inp))
                print('    predicted > {}'.format(pred))
                if i >= 2:
                    break
            print()

except KeyboardInterrupt:
    print('training interrupted')
batch 0
  minibatch loss: 2.3008058071136475
  sample 1:
    input     > [2 7 7 4 0 0 0 0]
    predicted > [8 8 0 0 8 0 0 0 0 0 0]
  sample 2:
    input     > [5 2 6 3 2 4 4 0]
    predicted > [8 9 9 9 9 9 9 9 9 9 0]
  sample 3:
    input     > [7 8 3 6 0 0 0 0]
    predicted > [5 1 8 1 4 0 0 0 0 0 0]

batch 1000
  minibatch loss: 0.7816314697265625
  sample 1:
    input     > [4 7 4 4 0 0 0 0]
    predicted > [4 4 4 4 1 0 0 0 0 0 0]
  sample 2:
    input     > [4 5 4 5 5 3 7 0]
    predicted > [4 5 5 5 5 5 5 1 0 0 0]
  sample 3:
    input     > [6 6 7 7 0 0 0 0]
    predicted > [6 7 7 7 1 0 0 0 0 0 0]

batch 2000
  minibatch loss: 0.37288352847099304
  sample 1:
    input     > [7 3 2 3 0 0 0 0]
    predicted > [7 3 2 3 1 0 0 0 0 0 0]
  sample 2:
    input     > [9 6 3 9 6 0 0 0]
    predicted > [9 9 6 6 6 1 0 0 0 0 0]
  sample 3:
    input     > [8 4 6 6 2 0 0 0]
    predicted > [8 4 6 6 2 1 0 0 0 0 0]

batch 3000
  minibatch loss: 0.22249329090118408
  sample 1:
    input     > [6 3 5 0 0 0 0 0]
    predicted > [6 3 5 1 0 0 0 0 0 0 0]
  sample 2:
    input     > [3 2 5 5 2 4 0 0]
    predicted > [3 2 5 5 2 4 1 0 0 0 0]
  sample 3:
    input     > [2 7 3 0 0 0 0 0]
    predicted > [2 7 3 1 0 0 0 0 0 0 0]
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(loss_track)
print('loss {:.4f} after {} examples (batch_size={})'.format(loss_track[-1], len(loss_track)*batch_size, batch_size))
loss 0.2228 after 300100 examples (batch_size=100)

output_45_1.png

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏青玉伏案

算法与数据结构(五) 普利姆与克鲁斯卡尔的最小生成树(Swift版)

上篇博客我们聊了图的物理存储结构邻接矩阵和邻接链表,然后在此基础上给出了图的深度优先搜索和广度优先搜索。本篇博客就在上一篇博客的基础上进行延伸,也是关于图的。今...

2577
来自专栏我的博客

PHP取余的那些事

1、百分号取余 $val=9.45; $result=$val*100; echo intval($result); //这里输出944 echo $re...

37510
来自专栏小樱的经验随笔

Vijos P1116 一元三次方程求解【多解,暴力,二分】

一元三次方程求解 描述 有形如:ax^3+bx^2+cx+d=0 这样的一个一元三次方程。给出该方程中各项的系数(a,b,c,d 均为实数),并约定该方程存在三...

36512
来自专栏Jack-Cui

Day4、Python

题目     一个数如果恰好等于它的因子之和,这个数就成为“完数”。例如6=1+2+3。编程找出1000以内的所有完数。 程序分析     完全数(...

2470
来自专栏数据结构与算法

cf550D. Regular Bridge(构造)

一篇写的非常好的博客:http://www.cnblogs.com/mangoyang/p/9302269.html

771
来自专栏Python中文社区

实现属于自己的TensorFlow(一) - 计算图与前向传播

前言 前段时间因为课题需要使用了一段时间TensorFlow,感觉这种框架很有意思,除了可以搭建复杂的神经网络,也可以优化其他自己需要的计算模型,所以一直想自...

4287
来自专栏漫漫深度学习路

tensorflow学习笔记(三十三):ExponentialMovingAverage

ExponentialMovingAverage Some training algorithms, such as GradientDescent and M...

4966
来自专栏xingoo, 一个梦想做发明家的程序员

[翻译]CURAND Libaray--Host API--(2)

2.3 返回值 所有的CURAND host端的函数返回值都是curandStatus_t.如果调用没有错误,则返回成功,即返回值为CURAND_STATUS_...

22910
来自专栏Petrichor的专栏

tensorflow编程: Neural Network

对 features 和 tf.negative(features) 分别 Relu 并 concatenate 在一起。

1823
来自专栏漫漫深度学习路

tensorflow自定义op:梯度

tensorflow自定义op,梯度 tensorflow 是 自动微分的,但是如果你不给它定义微分方程的话,它啥也干不了 在使用 tensorflow 的时...

7027

扫码关注云+社区

领取腾讯云代金券