让φj(x)代表网络φ的第j层,输入是x。特征图谱的形状就是Cj x Hj x Wj、定义矩阵Gj(x)为Cj x Cj矩阵(特征矩阵)其中的元素来自于:
如果我们把φj(x)理解成一个Cj维度的特征,每个特征的尺寸是Hj x Wj,那么上式左边Gj(x)就是与Cj维的非中心的协方差成比例。每一个网格位置都可以当做一个独立的样本。这因此能抓住是哪个特征能带动其他的信息。梯度矩阵可以很搞笑的倍计算,通过调整φj(x)的形状为一个矩阵ψ,形状为Cj x HjWj,然后Gj(x)就是ψψT/CjHjWj。
训练细节:我们的风格转换网络是用COCO数据集训练的,我们调整每一个图像到256x256,共8万张训练图,batch-size=4,迭代40000次,大约跑了两轮。用Adam优化,初始学习速率0.001.输出图被用了全变量正则化(strength 在1e-6到1e-4之间),通过交叉验证集选择。不用权重衰减或者dropout,因为模型在这两轮中没有过拟合。对所有的风格转换实验我们取relu2_2层做内容,relu1_2,relu2_2,relu3_3和relu4_3作为风格。VGG-16网络,我们的实验用了Torch和cuDNN,训练用了大约4个小时,在一个GTX Titan X GPU上。
我们用我们的方法和它们的方法一起训练了五十张图片(从MSCOCO验证集中得到)使用The Muse by Pablo Picasso当作一个风格图。对于基础方法我们记录了函数在每一个迭代过程的值。对我们的方法我们对每一张图片记录了公式5的值。我们还计算了公式5的值,当y和输出图像yc相等时,结果显示在表5,我们看到内容图Yc达到了非常高的损失,和我们的方法在50-100之间差不多。
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convo-lutional networks. (2015)
Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEEInternational Conference on Computer Vision. (2015) 415{423
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semanticsegmentation. CVPR (2015)
Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single imageusing a multi-scale deep network. In: Advances in Neural Information ProcessingSystems. (2014) 2366{2374
Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels witha common multi-scale convolutional architecture. In: Proceedings of the IEEEInternational Conference on Computer Vision. (2015) 2650{2658
Mahendran, A., Vedaldi, A.: Understanding deep image representations by invert-ing them. In: Proceedings of the IEEE Conf. on Computer Vision and PatternRecognition (CVPR). (2015)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional net-works: Visualising image classi_cation models and saliency maps. arXiv preprintarXiv:1312.6034 (2013)
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neuralnetworks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)
Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neuralnetworks. In: Advances in Neural Information Processing Systems 28. (May 2015)
Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. arXivpreprint arXiv:1508.06576 (2015)
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional networkfor image super-resolution. In: Computer Vision{ECCV 2014. Springer (2014)184{199
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang,C., Torr, P.H.: Conditional random _elds as recurrent neural networks. In: Proceed-ings of the IEEE International Conference on Computer Vision. (2015) 1529{1537
Liu, F., Shen, C., Lin, G.: Deep convolutional neural _elds for depth estimationfrom a single image. In: Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition. (2015) 5162{5170
Wang, X., Fouhey, D., Gupta, A.: Designing deep networks for surface normalestimation. In: Proceedings of the IEEE Conference on Computer Vision andPattern Recognition. (2015) 539{547
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fer-gus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199(2013)
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: Highcon_dence predictions for unrecognizable images. In: Computer Vision and PatternRecognition (CVPR), 2015 IEEE Conference on, IEEE (2015) 427-436
d'Angelo, E., Alahi, A., Vandergheynst, P.: Beyond bits: Reconstructing imagesfrom local binary descriptors. In: Pattern Recognition (ICPR), 2012 21st Interna-tional Conference on, IEEE (2012) 935{938
Vondrick, C., Khosla, A., Malisiewicz, T., Torralba, A.: Hoggles: Visualizing ob-ject detection features. In: Proceedings of the IEEE International Conference onComputer Vision. (2013) 1{8
Dosovitskiy, A., Brox, T.: Inverting visual representations with convolutional net-works. arXiv preprint arXiv:1506.02753 (2015)
Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding.In: Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings ofthe 2004 IEEE Computer Society Conference on. Volume 1., IEEE (2004) I{I
Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: Com-puter Vision, 2009 IEEE 12th International Conference on, IEEE (2009) 349{356
Yang, J., Lin, Z., Cohen, S.: Fast image super-resolution based on in-place exam-ple regression. In: Proceedings of the IEEE Conference on Computer Vision andPattern Recognition. (2013) 1059{1066
Sun, J., Zheng, N.N., Tao, H., Shum, H.Y.: Image hallucination with primal sketchpriors. In: Computer Vision and Pattern Recognition, 2003. Proceedings. 2003IEEE Computer Society Conference on. Volume 2., IEEE (2003) II{729
Ni, K.S., Nguyen, T.Q.: Image superresolution using support vector regression.Image Processing, IEEE Transactions on 16(6) (2007) 1596{1610
He, L., Qi, H., Zaretzki, R.: Beta process joint dictionary learning for coupledfeature spaces with application to single image super-resolution. In: Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition. (2013) 345{352
Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution as sparse repre-sentation of raw image patches. In: Computer Vision and Pattern Recognition,2008. CVPR 2008. IEEE Conference on, IEEE (2008) 1{8
Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparserepresentation. Image Processing, IEEE Transactions on 19(11) (2010) 2861{2873
Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhoodregression for fast super-resolution. In: Computer Vision{ACCV 2014. Springer(2014) 111{126
Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling withsuper-resolution forests. In: Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition. (2015) 3791{3799
Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformedself-exemplars. In: Computer Vision and Pattern Recognition (CVPR), 2015 IEEEConference on, IEEE (2015) 5197{5206
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learn-ing with deep convolutional generative adversarial networks. arXiv preprintarXiv:1511.06434 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.arXiv preprint arXiv:1512.03385 (2015)
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll_ar, P.,Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision{ECCV 2014. Springer (2014) 740{755
Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980 (2014)
Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: A matlab-like environmentfor machine learning. In: BigLearn, NIPSWorkshop. Number EPFL-CONF-192376(2011)
Hanhart, P., Korshunov, P., Ebrahimi, T.: Benchmarking of quality metrics onultra-high de_nition video sequences. In: Digital Signal Processing (DSP), 201318th International Conference on, IEEE (2013) 1{8
Wang, Z., Bovik, A.C.: Mean squared error: love it or leave it? a new look at signal_delity measures. Signal Processing Magazine, IEEE 26(1) (2009) 98{117
Huynh-Thu, Q., Ghanbari, M.: Scope of validity of psnr in image/video qualityassessment. Electronics letters 44(13) (2008) 800{801
Sheikh, H.R., Sabir, M.F., Bovik, A.C.: A statistical evaluation of recent full refer-ence image quality assessment algorithms. Image Processing, IEEE Transactionson 15(11) (2006) 3440{3451
Kundu, D., Evans, B.L.: Full-reference visual quality assessment for syntheticimages: A subjective study. Proc. IEEE Int. Conf. on Image Processing (2015)
Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexitysingle-image super-resolution based on nonnegative neighbor embedding. (2012)
Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces. Springer (2010) 711-730