stackGAN通过文字描述生成图片的V2项目

https://github.com/hanzhanggit/StackGAN-v2

StackGAN-v2

• StackGAN-v1: Tensorflow implementation
• StackGAN-v1: Pytorch implementation
• Inception score evaluation

Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks by Han Zhang*, Tao Xu*, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas.

Dependencies

python 2.7

Pytorch

In addition, please add the project folder to PYTHONPATH and pip install the following packages:

• tensorboard
• python-dateutil
• easydict
• pandas
• torchfile

Data

1. Download our preprocessed char-CNN-RNN text embeddings for birds and save them to data/

• [Optional] Follow the instructions reedscot/icml2016 to download the pretrained char-CNN-RNN text encoders and extract text embeddings.

1. Download the birds image data. Extract them to data/birds/
2. Download ImageNet dataset and extract the images to data/imagenet/
3. Download LSUN dataset and save the images to data/lsun

Training

• Train a StackGAN-v2 model on the bird (CUB) dataset using our preprocessed embeddings:
  • python main.py --cfg cfg/birds_3stages.yml --gpu 0
• Train a StackGAN-v2 model on the ImageNet dog subset:
  • python main.py --cfg cfg/dog_3stages_color.yml --gpu 0
• Train a StackGAN-v2 model on the ImageNet cat subset:
  • python main.py --cfg cfg/cat_3stages_color.yml --gpu 0
• Train a StackGAN-v2 model on the lsun bedroom subset:
  • python main.py --cfg cfg/bedroom_3stages_color.yml --gpu 0
• Train a StackGAN-v2 model on the lsun church subset:
  • python main.py --cfg cfg/church_3stages_color.yml --gpu 0
• *.yml files are example configuration files for training/evaluation our models.
• If you want to try your own datasets, here are some good tips about how to train GAN. Also, we encourage to try different hyper-parameters and architectures, especially for more complex datasets.

Pretrained Model

• StackGAN-v2 for bird. Download and save it to models/ (The inception score for this Model is 4.04±0.05)
• StackGAN-v2 for dog. Download and save it to models/ (The inception score for this Model is 9.55±0.11)
• StackGAN-v2 for cat. Download and save it to models/
• StackGAN-v2 for bedroom. Download and save it to models/
• StackGAN-v2 for church. Download and save it to models/

Evaluating

• Run python main.py --cfg cfg/eval_birds.yml --gpu 1 to generate samples from captions in birds validation set.
• Change the eval_*.yml files to generate images from other pre-trained models.

Examples generated by StackGAN-v2

Tsne visualization of randomly generated birds, dogs, cats, churchs and bedrooms

Citing StackGAN++

If you find StackGAN useful in your research, please consider citing:

@article{Han17stackgan2,
  author    = {Han Zhang and Tao Xu and Hongsheng Li and Shaoting Zhang and Xiaogang Wang and Xiaolei Huang and Dimitris Metaxas},
  title     = {StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks},
  journal   = {arXiv: 1710.10916},
  year      = {2017},
}

@inproceedings{han2017stackgan,
Author = {Han Zhang and Tao Xu and Hongsheng Li and Shaoting Zhang and Xiaogang Wang and Xiaolei Huang and Dimitris Metaxas},
Title = {StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks},
Year = {2017},
booktitle = {{ICCV}},
}

Our follow-up work

• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks [Supplementary]

References

• Generative Adversarial Text-to-Image Synthesis Paper Code
• Learning Deep Representations of Fine-grained Visual Descriptions Paper Code