为什么我的Deep Q网络不能掌握一个简单的Gridworld (Tensorflow)？(如何评估Deep-Q-Net)
为什么我的Deep Q网络不能掌握一个简单的Gridworld (Tensorflow)？(如何评估Deep-Q-Net)
提问于 2016-02-15 00:56:47
我尝试熟悉Q学习和深度神经网络，目前正在尝试实现Playing Atari with Deep Reinforcement Learning。
为了测试我的实现并使用它，我想我可以尝试一个简单的网格世界。其中我有一个N x N网格，从左上角开始，在右下角结束。可能的动作有:左、上、右、下。
尽管我的实现已经变得非常类似于this(希望它是一个好的)，但它似乎没有学到任何东西。看看它需要完成的总步数(我猜平均在10x10的网格大小下大约是500步，但也有非常低和很高的值)，对我来说，它看起来比其他任何东西都更随机。
我尝试了使用卷积层和不使用卷积层，并尝试了所有参数，但老实说，我不知道我的实现是否有问题，或者它需要训练更长的时间(我让它训练了相当长的一段时间)，或者其他什么。但至少它看起来是收敛的，这里是一次培训课程的损失值曲线图：
那么这种情况下的问题是什么呢？
而且，也许更重要的是，我如何才能“调试”这个深度Q网络，在有监督的训练中，有训练、测试和验证集，并且例如具有精确度和召回率，可以对它们进行评估。对于Deep-Q-Nets的无监督学习，我有哪些选择，以便下次我可以自己修复它？
最后是代码：
这就是网络：
ACTIONS = 5
# Inputs
x = tf.placeholder('float', shape=[None, 10, 10, 4])
y = tf.placeholder('float', shape=[None])
a = tf.placeholder('float', shape=[None, ACTIONS])
# Layer 1 Conv1 - input
with tf.name_scope('Layer1'):
W_conv1 = weight_variable([8,8,4,8])
b_conv1 = bias_variable([8])
h_conv1 = tf.nn.relu(conv2d(x, W_conv1, 5)+b_conv1)
# Layer 2 Conv2 - hidden1
with tf.name_scope('Layer2'):
W_conv2 = weight_variable([2,2,8,8])
b_conv2 = bias_variable([8])
h_conv2 = tf.nn.relu(conv2d(h_conv1, W_conv2, 1)+b_conv2)
h_conv2_max_pool = max_pool_2x2(h_conv2)
# Layer 3 fc1 - hidden 2
with tf.name_scope('Layer3'):
W_fc1 = weight_variable([8, 32])
b_fc1 = bias_variable([32])
h_conv2_flat = tf.reshape(h_conv2_max_pool, [-1, 8])
h_fc1 = tf.nn.relu(tf.matmul(h_conv2_flat, W_fc1)+b_fc1)
# Layer 4 fc2 - readout
with tf.name_scope('Layer4'):
W_fc2 = weight_variable([32, ACTIONS])
b_fc2 = bias_variable([ACTIONS])
readout = tf.matmul(h_fc1, W_fc2)+ b_fc2
# Training
with tf.name_scope('training'):
readout_action = tf.reduce_sum(tf.mul(readout, a), reduction_indices=1)
loss = tf.reduce_mean(tf.square(y - readout_action))
train = tf.train.AdamOptimizer(1e-6).minimize(loss)
loss_summ = tf.scalar_summary('loss', loss)这里是训练：
# 0 => left
# 1 => up
# 2 => right
# 3 => down
# 4 = noop
ACTIONS = 5
GAMMA = 0.95
BATCH = 50
TRANSITIONS = 2000
OBSERVATIONS = 1000
MAXSTEPS = 1000
D = deque()
epsilon = 1
average = 0
for episode in xrange(1000):
step_count = 0
game_ended = False
state = np.array([0.0]*100, float).reshape(100)
state[0] = 1
rsh_state = state.reshape(10,10)
s = np.stack((rsh_state, rsh_state, rsh_state, rsh_state), axis=2)
while step_count < MAXSTEPS and not game_ended:
reward = 0
step_count += 1
read = readout.eval(feed_dict={x: [s]})[0]
act = np.zeros(ACTIONS)
action = random.randint(0,4)
if len(D) > OBSERVATIONS and random.random() > epsilon:
action = np.argmax(read)
act[action] = 1
# play the game
pos_idx = state.argmax(axis=0)
pos = pos_idx + 1
state[pos_idx] = 0
if action == 0 and pos%10 != 1: #left
state[pos_idx-1] = 1
elif action == 1 and pos > 10: #up
state[pos_idx-10] = 1
elif action == 2 and pos%10 != 0: #right
state[pos_idx+1] = 1
elif action == 3 and pos < 91: #down
state[pos_idx+10] = 1
else: #noop
state[pos_idx] = 1
pass
if state.argmax(axis=0) == pos_idx and reward > 0:
reward -= 0.0001
if step_count == MAXSTEPS:
reward -= 100
elif state[99] == 1: # reward & finished
reward += 100
game_ended = True
else:
reward -= 1
s_old = np.copy(s)
s = np.append(s[:,:,1:], state.reshape(10,10,1), axis=2)
D.append((s_old, act, reward, s))
if len(D) > TRANSITIONS:
D.popleft()
if len(D) > OBSERVATIONS:
minibatch = random.sample(D, BATCH)
s_j_batch = [d[0] for d in minibatch]
a_batch = [d[1] for d in minibatch]
r_batch = [d[2] for d in minibatch]
s_j1_batch = [d[3] for d in minibatch]
readout_j1_batch = readout.eval(feed_dict={x:s_j1_batch})
y_batch = []
for i in xrange(0, len(minibatch)):
y_batch.append(r_batch[i] + GAMMA * np.max(readout_j1_batch[i]))
train.run(feed_dict={x: s_j_batch, y: y_batch, a: a_batch})
if epsilon > 0.05:
epsilon -= 0.01我感谢你的每一个帮助和想法！
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/35394446
复制相关文章
相似问题