# 教程 | 无需反向传播的深度学习：DeepMind的合成梯度

Geoffrey Hinton 相关的工作

```class Layer(object):

def __init__(self,input_dim, output_dim,nonlin,nonlin_deriv):

self.weights = (np.random.randn(input_dim, output_dim) * 0.2) - 0.1
self.nonlin = nonlin
self.nonlin_deriv = nonlin_deriv

def forward(self,input):
self.input = input
self.output = self.nonlin(self.input.dot(self.weights))
return self.output

def backward(self,output_delta):
self.weight_output_delta = output_delta * self.nonlin_deriv(self.output)
return self.weight_output_delta.dot(self.weights.T)

def update(self,alpha=0.1):
self.weights -= self.input.T.dot(self.weight_output_delta) * alpha```

update 可能是最简单的方法。它只需要在权重的输出中使用导数，并使用它来进行权重更新。如果你对这些步骤有任何疑问，再一次地，查看 A Neural Network in 11 Lines of Python 再回来。如果你能全部理解，那么让我们在训练中看看我们的层对象。

```layer_1 = Layer(input_dim,layer_1_dim,sigmoid,sigmoid_out2deriv)
layer_2 = Layer(layer_1_dim,layer_2_dim,sigmoid,sigmoid_out2deriv)
layer_3 = Layer(layer_2_dim, output_dim,sigmoid, sigmoid_out2deriv)

for iter in range(iterations):
error = 0

for batch_i in range(int(len(x) / batch_size)):
batch_x = x[(batch_i * batch_size):(batch_i+1)*batch_size]
batch_y = y[(batch_i * batch_size):(batch_i+1)*batch_size]

layer_1_out = layer_1.forward(batch_x)
layer_2_out = layer_2.forward(layer_1_out)
layer_3_out = layer_3.forward(layer_2_out)

layer_3_delta = layer_3_out - batch_y
layer_2_delta = layer_3.backward(layer_3_delta)
layer_1_delta = layer_2.backward(layer_2_delta)
layer_1.backward(layer_1_delta)

layer_1.update()
layer_2.update()
layer_3.update()```

```class DNI(object):

def __init__(self,input_dim, output_dim,nonlin,nonlin_deriv,alpha = 0.1):

# same as before
self.weights = (np.random.randn(input_dim, output_dim) * 0.2) - 0.1
self.nonlin = nonlin
self.nonlin_deriv = nonlin_deriv

# new stuff
self.weights_synthetic_grads = (np.random.randn(output_dim,output_dim) * 0.2) - 0.1
self.alpha = alpha

# used to be just "forward", but now we update during the forward pass using Synthetic Gradients :)
def forward_and_synthetic_update(self,input):

# cache input
self.input = input

# forward propagate
self.output = self.nonlin(self.input.dot(self.weights))

# generate synthetic gradient via simple linear transformation

# update our regular weights using synthetic gradient
self.weights += self.input.T.dot(self.weight_synthetic_gradient) * self.alpha

# return backpropagated synthetic gradient (this is like the output of "backprop" method from the Layer class)
# also return forward propagated output (feels weird i know... )

# this is just like the "update" method from before... except it operates on the synthetic weights

0 条评论

## 相关文章

33790

31880

340110

### Python 资源大全中文版

GitHub 上有一个 Awesome - XXX 系列的资源整理,资源非常丰富，涉及面非常广。awesome-python 是 vinta 发起维护的 Pyt...

1.4K60

### GitHub 上 10 款免费开源 Windows 工具

GitHub 是如今所有开源事物的中央仓库, 这个网站最近发布了一个叫做《2016 Octoverse 状态报告》，详细列出了从去年起其一系列亮点, 包括总的...

60980

41260

44770

36560

### AI 技术讲座精选：数学不好，也可以学习人工智能（六）——巧用数学符号

【AI100 导读】欢迎阅读《数学不好，也可以学好人工智能》系列的第六篇文章。如果你错过了之前的五部分，一定记得把它们找出来看一下！这篇文章作者会帮你学习数学符...

43380

3.3K90