GAN的图像修复：多样化补全

2019 cvpr：Pluralistic Image Completion

https://arxiv.xilesou.top/pdf/1903.04227.pdf

https://github.com/lyndonzheng/Pluralistic-Inpainting

• 对于每个输入（masked input），大多数图像补全方法只能产生一种结果（尽管有许多合理的其它可能结果）。基于学习的方法里，通常每个标签只有一个ground true（目标参照图象GT）。即便从有条件VAE采样、仍然会多样性不足。本文提出了一种多元化图像补全方法。
• 提出了一种具有两个平行路径的概率学习框架。一个是重建路径（网络），它只利用对应一个的GT去获取缺失区域的先验信息，并完成重建；另一个是生成路径（网络），它将条件先验耦合于重建路径所获得的分布。在对抗方式下完成训练，训练完后只用生成路径。
• 提出了一种新的、建模长短区域关系的注意力机制，以提高图像一致性。
• 在建筑、人脸（Celeba-HQ）、ImageNet等数据集上不仅取得了更好的补全效果，在多样性上也合理、令人信服。

方法

• 定义为原始完整图像，是被遮挡的（掩masked）图像，则经典的图像补全方法是去学习映射，它们是确定性的。
• 本文还定义表示的“反”，也就是它仅仅是由被遮挡部分构成，而本文的目标是从采样去恢复。

• 损失函数：

进一步地：

• 分布正则化（参照VAE/CVAE）：

对于重建路径：

对于生成路径：

（这里的KL，个人感觉不应该带负号啊？？

• 图像外表匹配：

对于重建路径，约束重建图像和目标参考GT相似：

对于生成路径，约束生成和GT相似：

• 对抗损失：

在特征层面的约束和LSGAN损失：

• 网络结构之attention设计

在前面网络整体的图所示，decoder之前的红色模块即为本文所提出的attention设计：

部分效果

GAN图像修复/补全生成-相关阅读：

001 (2020-04-8) Attentive Normalization for Conditional Image Generation

https://arxiv.xilesou.top/pdf/2004.03828.pdf

002 (2020-03-27) GAN-based Priors for Quantifying Uncertainty

https://arxiv.xilesou.top/pdf/2003.12597.pdf

003 (2020-02-6) Automatic Segmentation and Visualization of Choroid in OCT with Knowledge Infused Deep Learning

https://arxiv.xilesou.top/pdf/2002.04712.pdf

004 (2020-02-7) Local Facial Attribute Transfer through Inpainting

https://arxiv.xilesou.top/pdf/2002.03040.pdf

005 (2020-02-5) Domain Embedded Multi-model Generative Adversarial Networks for Image-based Face Inpainting

https://arxiv.xilesou.top/pdf/2002.02909.pdf

006 (2020-02-6) Image Fine-grained Inpainting

https://arxiv.xilesou.top/pdf/2002.02609.pdf

007 (2020-02-4) Pixel-wise Conditioned Generative Adversarial Networks for Image Synthesis and Completion

https://arxiv.xilesou.top/pdf/2002.01281.pdf

008 (2020-01-11) Symmetric Skip Connection Wasserstein GAN for High-Resolution Facial Image Inpainting

https://arxiv.xilesou.top/pdf/2001.03725.pdf

009 (2019-12-31) Erase and Restore Simple Accurate and Resilient Detection of $L 2$ Adversarial Examples

https://arxiv.xilesou.top/pdf/2001.00116.pdf

010 (2019-12-18) Unsupervised Adversarial Image Inpainting

https://arxiv.xilesou.top/pdf/1912.12164.pdf

011 (2020-02-15) Image Outpainting and Harmonization using Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1912.10960.pdf

012 (2020-03-31) Image Processing Using Multi-Code GAN Prior

https://arxiv.xilesou.top/pdf/1912.07116.pdf

013 (2019-12-9) Environment reconstruction on depth images using Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1912.03992.pdf

014 (2019-12-5) Blind Inpainting of Large-scale Masks of Thin Structures with Adversarial and Reinforcement Learning

https://arxiv.xilesou.top/pdf/1912.02470.pdf

015 (2020-03-19) PiiGAN Generative Adversarial Networks for Pluralistic Image Inpainting

https://arxiv.xilesou.top/pdf/1912.01834.pdf

016 (2019-11-2) Pixel-wise Conditioning of Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1911.00689.pdf

017 (2019-10-23) Facial Expression Restoration Based on Improved Graph Convolutional Networks

https://arxiv.xilesou.top/pdf/1910.10344.pdf

018 (2019-09-27) Region-wise Generative Adversarial ImageInpainting for Large Missing Areas

https://arxiv.xilesou.top/pdf/1909.12507.pdf

019 (2019-12-23) Physics-informed semantic inpainting Application to geostatistical modeling

https://arxiv.xilesou.top/pdf/1909.09459.pdf

020 (2019-08-19) Boundless Generative Adversarial Networks for Image Extension

https://arxiv.xilesou.top/pdf/1908.07007.pdf

021 (2019-08-19) Fully Automated Image De-fencing using Conditional Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1908.06837.pdf

022 (2019-08-16) The Angel is in the Priors Improving GAN based Image and Sequence Inpainting with Better Noise and Structural Priors

https://arxiv.xilesou.top/pdf/1908.05861.pdf

023 (2019-08-14) Faster Unsupervised Semantic Inpainting A GAN Based Approach

https://arxiv.xilesou.top/pdf/1908.04968.pdf

024 (2019-10-29) Generative Modeling by Estimating Gradients of the Data Distribution

https://arxiv.xilesou.top/pdf/1907.05600.pdf

025 (2019-06-14) Single-Channel Signal Separation and Deconvolution with Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1906.07552.pdf

026 (2019-06-6) How to make a pizza Learning a compositional layer-based GAN model

https://arxiv.xilesou.top/pdf/1906.02839.pdf

027 (2019-09-28) Fashion Editing with Adversarial Parsing Learning

https://arxiv.xilesou.top/pdf/1906.00884.pdf

028 (2019-06-3) GazeCorrection Self-Guided Eye Manipulation in the wild using Self-Supervised Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1906.00805.pdf

029 (2019-05-23) Generative Imaging and Image Processing via Generative Encoder

https://arxiv.xilesou.top/pdf/1905.13300.pdf

030 (2020-01-30) Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators

https://arxiv.xilesou.top/pdf/1905.12660.pdf

031 (2019-09-27) Invertible generative models for inverse problems mitigating representation error and dataset bias

https://arxiv.xilesou.top/pdf/1905.11672.pdf

032 (2020-03-6) PEPSI++ Fast and Lightweight Network for Image Inpainting

https://arxiv.xilesou.top/pdf/1905.09010.pdf

033 (2019-05-6) The Missing Data Encoder Cross-Channel Image Completion\\with Hide-And-Seek Adversarial Network

https://arxiv.xilesou.top/pdf/1905.01861.pdf

034 (2019-07-10) Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

https://arxiv.xilesou.top/pdf/1904.07475.pdf

035 (2019-04-15) A deep learning framework for quality assessment and restoration in video endoscopy

https://arxiv.xilesou.top/pdf/1904.07073.pdf

036 (2019-03-28) GANs-NQM A Generative Adversarial Networks based No Reference Quality Assessment Metric for RGB-D Synthesized Views

https://arxiv.xilesou.top/pdf/1903.12088.pdf

037 (2019-11-6) Implicit Generation and Generalization in Energy-Based Models

https://arxiv.xilesou.top/pdf/1903.08689.pdf

038 (2019-03-10) A Sinogram Inpainting Method based on Generative Adversarial Network for Limited-angle Computed Tomography

https://arxiv.xilesou.top/pdf/1903.03984.pdf

039 (2019-10-23) GAN-based Projector for Faster Recovery with Convergence Guarantees in Linear Inverse Problems

https://arxiv.xilesou.top/pdf/1902.09698.pdf

040 (2019-02-10) Cross-spectral Face Completion for NIR-VIS Heterogeneous Face Recognition

https://arxiv.xilesou.top/pdf/1902.03565.pdf

041 (2019-01-25) Diversity-Sensitive Conditional Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1901.09024.pdf

042 (2019-01-9) Detecting Overfitting of Deep Generative Networks via Latent Recovery

https://arxiv.xilesou.top/pdf/1901.03396.pdf

043 (2019-01-11) EdgeConnect Generative Image Inpainting with Adversarial Edge Learning

https://arxiv.xilesou.top/pdf/1901.00212.pdf

044 (2019-07-30) Exploiting the Inherent Limitation of L0 Adversarial Examples

https://arxiv.xilesou.top/pdf/1812.09638.pdf

045 (2019-11-27) Face Completion with Semantic Knowledge and Collaborative Adversarial Learning

https://arxiv.xilesou.top/pdf/1812.03252.pdf

046 (2018-12-3) Semantic Image Inpainting Through Improved Wasserstein Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1812.01071.pdf

047 (2019-04-9) Context Encoding Chest X-rays

https://arxiv.xilesou.top/pdf/1812.00964.pdf

048 (2019-02-26) Void Filling of Digital Elevation Models with Deep Generative Models

https://arxiv.xilesou.top/pdf/1811.12693.pdf

049 (2019-01-15) SEIGAN Towards Compositional Image Generation by Simultaneously Learning to Segment Enhance and Inpaint

https://arxiv.xilesou.top/pdf/1811.07630.pdf

050 (2020-01-11) On Hallucinating Context and Background Pixels from a Face Mask using Multi-scale GANs

https://arxiv.xilesou.top/pdf/1811.07104.pdf

051 (2018-10-20) Improved Techniques for GAN based Facial Inpainting

https://arxiv.xilesou.top/pdf/1810.08774.pdf

052 (2018-10-15) Adversarial Inpainting of Medical Image Modalities

https://arxiv.xilesou.top/pdf/1810.06621.pdf

053 (2019-02-15) Empty Cities Image Inpainting for a Dynamic-Object-Invariant Space

https://arxiv.xilesou.top/pdf/1809.10239.pdf

054 (2018-09-6) Dense Pose Transfer

https://arxiv.xilesou.top/pdf/1809.01995.pdf

055 (2018-08-29) Chest X-ray Inpainting with Deep Generative Models

https://arxiv.xilesou.top/pdf/1809.01471.pdf

056 (2018-08-25) Painting Outside the Box Image Outpainting with GANs

https://arxiv.xilesou.top/pdf/1808.08483.pdf

057 (2019-07-18) Cross-view image synthesis using geometry-guided conditional GANs

https://arxiv.xilesou.top/pdf/1808.05469.pdf

058 (2018-08-6) X-GANs Image Reconstruction Made Easy for Extreme Cases

https://arxiv.xilesou.top/pdf/1808.04432.pdf

059 (2018-11-2) Latent Convolutional Models

https://arxiv.xilesou.top/pdf/1806.06284.pdf

060 (2018-06-13) Self-Supervised Feature Learning by Learning to Spot Artifacts

https://arxiv.xilesou.top/pdf/1806.05024.pdf

061 (2018-07-17) AVID Adversarial Visual Irregularity Detection

https://arxiv.xilesou.top/pdf/1805.09521.pdf

062 (2018-05-21) Unsupervised Deep Context Prediction for Background Foreground Separation

https://arxiv.xilesou.top/pdf/1805.07903.pdf

063 (2018-06-4) An Unsupervised Approach to Solving Inverse Problems using Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1805.07281.pdf

064 (2019-07-20) Generative Steganography by Sampling

https://arxiv.xilesou.top/pdf/1804.10531.pdf

065 (2018-04-25) Generative Model for Heterogeneous Inference

https://arxiv.xilesou.top/pdf/1804.09858.pdf

066 (2018-05-14) The Reincarnation of Grille Cipher A Generative Approach

https://arxiv.xilesou.top/pdf/1804.06514.pdf

067 (2018-04-3) Correlated discrete data generation using adversarial training

https://arxiv.xilesou.top/pdf/1804.00925.pdf

068 (2018-03-27) Structural inpainting

https://arxiv.xilesou.top/pdf/1803.10348.pdf

069 (2018-05-10) Digital Cardan Grille A Modern Approach for Information Hiding

https://arxiv.xilesou.top/pdf/1803.09219.pdf

070 (2018-03-27) Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart

https://arxiv.xilesou.top/pdf/1803.08943.pdf

071 (2018-03-20) Patch-Based Image Inpainting with Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1803.07422.pdf

072 (2018-07-5) Generating Realistic Geology Conditioned on Physical Measurements with Generative Adversarial Networks

https://arxiv.xilesou.top/pdf/1802.03065.pdf

073 (2018-08-30) Deep Structured Energy-Based Image Inpainting

https://arxiv.xilesou.top/pdf/1801.07939.pdf

074 (2017-12-20) Context-Aware Semantic Inpainting

https://arxiv.xilesou.top/pdf/1712.07778.pdf

075 (2017-12-11) GibbsNet Iterative Adversarial Inference for Deep Graphical Models

https://arxiv.xilesou.top/pdf/1712.04120.pdf

076 (2018-08-6) Discriminative Region Proposal Adversarial Networks for High-Quality Image-to-Image Translation

https://arxiv.xilesou.top/pdf/1711.09554.pdf

077 (2017-11-26) Semantically Consistent Image Completion with Fine-grained Details

https://arxiv.xilesou.top/pdf/1711.09345.pdf

078 (2017-11-17) Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network

https://arxiv.xilesou.top/pdf/1711.06106.pdf

079 (2019-04-2) Perceptual Adversarial Networks for Image-to-Image Transformation

https://arxiv.xilesou.top/pdf/1706.09138.pdf

080 (2017-12-6) r-BTN Cross-domain Face Composite and Synthesis from Limited Facial Patches

https://arxiv.xilesou.top/pdf/1706.00556.pdf

081 (2017-09-1) Adversarial Inverse Graphics Networks Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

https://arxiv.xilesou.top/pdf/1705.11166.pdf

082 (2017-10-12) CVAE-GAN Fine-Grained Image Generation through Asymmetric Training

https://arxiv.xilesou.top/pdf/1703.10155.pdf

083 (2016-11-21) Context Encoders Feature Learning by Inpainting

https://arxiv.xilesou.top/pdf/1604.07379.pdf