Image Style Transfer Using Convolutional Neural Network(理论篇)
Image Style Transfer Using Convolutional Neural Network(理论篇)
今天这篇是关于neual art的,也就是style transfer算法;
文章来源:
A Neural Algorithm of Artistic Style, CVPR2015
Image Style Transfer Using Convolutional Neural Networks, CVPR2016
前一段时间有一个比较火的APP叫prisma,可以把自己上传的一张图进行风格化,内部原理就是今天说的neural art或者叫style transfer吧
上面两篇文章的内容差不多,下面一篇可以看作是上面一篇的扩展,增加了一些其他的实验,我们说的就是这篇
介绍
其实没什么说的,就是如何学习到一张图的style或者叫风格吧;比如现在有一张梵高的星空图,一张你自己拍的风景图,如何将梵高星空图的style添加到风景图之中呢???
大概就是像下面这样:
将一张图的style加到另外一张图的content上,这个就是style transfer
结构
其实算法的思想很简单,就是使用CNN进行特征提取,然后使用这些提取到的特征进行reconstruct.我们知道不同的CNN的conv layer提取到特征是不一样的,低层次的偏向于点 线等特征,高层次的更加偏向于texture信息.
所以算法的intuition,就是下图显示的:
所以作者使用VGG19的网络结构来做feature extractor,其最终是将conv2_2作为content layer,将conv1_1,conv2_1,conv3_1,conv4_1和conv5_1作为style layer
具体实现
算法使用随机的一个白噪声图(white noise image)作为输入,定义与内容图的content loss和风格图的`style loss’,之后使用标准的BP算法更新weight,调整输入的图像(白噪声图)
注意这里是调整输入的图,目的就是对于某个特定的输入图像x,其loss(包含content loss和style losss)达到最小
content loss
A layer with Nl distinct filters has Nl feature maps each of size Ml , where Ml is the height times the width of the feature map. So the responses in a layer l can be stored in a matrix
where
is the activation of the ith filter at position j in layer l.
假设p和x分别代表原始的图像和生成的图像,l代表layer,在某层的responce,则他们之间的content loss定义为:
style loss
feature correlations are given by the Gram matrix Gl ∈ RNl ×Nl , where
is the inner product between the vectorised feature maps i and j in layer l:
特征之间的相关性可以表示原始图像在layer中的多尺度表达,也就是表示了纹理信息
Let a and x be the original image and the image that is generated, and Al and Gl their respective style representation in layer l. The contribution of layer l to the total loss is then:
总的loss就是content loss和style loss的线性组合:
架构
总体的结构如下图所示:
补充
style和content的比重
也就是a/b的比重,效果如下图:
可以看出,比值越小合成的图风格化越明显
不同的layer对结果的影响
使用不同的layer作为content feature extraxtor或者style feature extractor效果是不一样的.
We find that matching the style representations up to higher layers in the network preserves local images structures an increasingly large scale, leading to a smoother and more continuous visual experience.
所以据此选择了conv(1-5)_1作为style layer
下图显示不同的conv layer作为content layer的不同影响:
不同的初始化方法
实验中我们是使用random white noise image作为input,但是也可以直接使用content image或者style image作为input,作者的结论是:
the different initialisations do not seem to have a strong effect on the outcome of the synthesis procedure
但是:
only initialising with noise allows to generate an arbitrary number of new images. Initialising with a fixed image always deterministically leads to the same outcome (up to stochasticity in the gradient descent procedure)
后记
我自己做了实验,对一张140*480的图,迭代300次,在titan x上用时30s左右,时间确实非常久
原文也给出了结论:
- The dimensionality of the optimisation problem as well as the number of units in the Convolutional Neural Network grow linearly with the number of pixels.
- The images presented in this paper were synthesised in a resolution of about 512 × 512 pixels and the synthesis procedure could take up to an hour on a Nvidia K40 GPU (depending on the exact image size and the stopping criteria for the gradientdescent).