一次GAN项目背景下的tensorflow_datasets的mnist数据集的下载笔记

ERROR: matplotlib 2.1.2 requires cycler>=0.10, which is not installed.
ERROR: matplotlib 2.1.2 requires pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1, which is not installed.
ERROR: matplotlib 2.1.2 requires pytz, which is not installed.
Installing collected packages: six, google-pasta, astor, numpy, h5py, keras-applications, markdown, absl-py, werkzeug, protobuf, grpcio, tensorboard, tensorflow-estimator, keras-preprocessing, gast, opt-einsum, termcolor, wrapt, tensorflow-gpu
Successfully installed absl-py-0.8.1 astor-0.8.1 gast-0.2.2 google-pasta-0.1.8 grpcio-1.25.0 h5py-2.10.0 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 numpy-1.17.4 opt-einsum-3.1.0 protobuf-3.11.1 six-1.13.0 tensorboard-1.15.0 tensorflow-estimator-1.15.1 tensorflow-gpu-1.15.0 termcolor-1.1.0 werkzeug-0.16.0 wrapt-1.11.2

(GAN) $ pip install -e .

(GAN) $ python
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> hello = tf.constant('HELLO!')
>>> with tf.Session() as sess:
...     print(sess.run(hello))
b'HELLO!'

>>> import tensorflow_datasets as tfds
>>> print(tfds.list_builders())
['bair_robot_pushing_small', 'cats_vs_dogs', 'celeb_a', 'celeb_a_hq', 'cifar10', 'cifar100', 'coco2014', 'diabetic_retinopathy_detection', 'dummy_dataset_shared_generator', 'dummy_mnist', 'fashion_mnist', 'image_label_folder', 'imagenet2012', 'imdb_reviews', 'lm1b', 'lsun', 'mnist', 'moving_mnist', 'nsynth', 'omniglot', 'open_images_v4', 'quickdraw_bitmap', 'squad', 'starcraft_video', 'svhn_cropped', 'tf_flowers', 'wmt_translate_ende', 'wmt_translate_enfr']

python main.py --model_dir try --gin_config ../example_configs/mnist.gin

requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.228.3', port=8081): Max retries exceeded with url: /IXCa9bee7c9d834e0e4d14e3b433c66e97e/exdb/mnist/train-images-idx3-ubyte.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe51008c2e8>: Failed to establish a new connection: [Errno 113] No route to host',))

>>> ds_train = tfds.load(name="mnist", split="train")

requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.228.3', port=8081): Max retries exceeded with url: /IXCa9bee7c9d834e0e4d14e3b433c66e97e/exdb/mnist/train-images-idx3-ubyte.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe51008c2e8>: Failed to establish a new connection: [Errno 113] No route to host',))

# Fetch the dataset directly
mnist = tfds.image.MNIST()
# or by string name
mnist = tfds.builder('mnist')
mnist.download_and_prepare()

requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.17.228.3', port=8081): Max retries exceeded with url: /IXCa9bee7c9d834e0e4d14e3b433c66e97e/exdb/mnist/train-images-idx3-ubyte.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe510110d68>: Failed to establish a new connection: [Errno 113] No route to host',))

yann.lecu.com_exdb_mnis_t10k-labe-idx1-ubytlX-SfSvqLNNrV7AqoACgdSrEYP_4wsDz4T_rhacyC0o.gz       
yann.lecu.com_exdb_mnis_trai-labe-idx1-ubyt7cc_IeM51G_ngIY2ORleKjMjLVCXd-TCUHlYvEiRiKI.gz
yann.lecu.com_exdb_mnis_t10k-labe-idx1-ubytlX-SfSvqLNNrV7AqoACgdSrEYP_4wsDz4T_rhacyC0o.gz.INFO  
yann.lecu.com_exdb_mnis_trai-labe-idx1-ubyt7cc_IeM51G_ngIY2ORleKjMjLVCXd-TCUHlYvEiRiKI.gz.INFO

https://blog.csdn.net/i8088/article/details/79126150
https://blog.csdn.net/abc13526222160/article/details/85109498
https://blog.csdn.net/jiaoyangwm/article/details/79254667

from tensorflow.examples.tutorials.mnist import input_data #这是TensorFlow 为了教学Mnist而提前设计好的程序
mnist = input_data.read_data_sets('MNIST_data', one_hot=True) #TensorFlow 会检测数据是否存在。当数据不存在时，系统会自动，在当前代码py文件位置，自动创建MNIST_data文件夹，并将数据下载到该件夹内。当执行完语句后，读者可以自行前往MNIST_data／文件夹下查看上述4 个文件是否已经被正确地下载
#若因网络问题无法正常下载，可以前往MNIST官网http://yann.lecun.com/exdb/mnist/使用下载工具下载上述4 个文件， 并将它们复制到MNIST_data／文件夹中。

from tensorflow.examples.tutorials.mnist import input_data #这是TensorFlow 为了教学Mnist而提前设计好的程序
mnist = input_data.read_data_sets('MNIST_data', one_hot=True) #TensorFlow 会检测数据是否存在。当数据不存在时，系统会自动，在当前代码py文件位置，自动创建MNIST_data文件夹，并将数据下载到该件夹内。当执行完语句后，读者可以自行前往MNIST_data／文件夹下查看上述4 个文件是否已经被正确地下载
#若因网络问题无法正常下载，可以前往MNIST官网http://yann.lecun.com/exdb/mnist/使用下载工具下载上述4 个文件， 并将它们复制到MNIST_data／文件夹中。

tensorflow.python.framework.errors_impl.NotFoundError: /users4/zsun/tensorflow_datasets/mnist/1.0.0/dataset_info.json; No such file or directory

https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/dataset_info.py
https://github.com/tensorflow/datasets/blob/v1.3.0/tensorflow_datasets/core/dataset_builder.py#L236-L308

https://tensorflow.google.cn/datasets/api_docs/python/tfds/core/DatasetInfo

>>> mnist = tfds.image.MNIST()
>>> mnist.info
<tensorflow_datasets.core.dataset_info.DatasetInfo object at 0x7fd6945811d0>
>>> print(mnist.info)
tfds.core.DatasetInfo(
    name='mnist',
    version=1.0.0,
    description='The MNIST database of handwritten digits.',
    urls=['http://yann.lecun.com/exdb/mnist/'],
    features=FeaturesDict({
        'image': Image(shape=(28, 28, 1), dtype=tf.uint8),
        'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=10)
    },
    total_num_examples=0,
    splits={

    },
    supervised_keys=('image', 'label'),
    citation='"""
        @article{lecun2010mnist,
          title={MNIST handwritten digit database},
          author={LeCun, Yann and Cortes, Corinna and Burges, CJ},
          journal={ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist},
          volume={2},
          year={2010}
        }
    """',
)

>>> print(mnist.info.as_json)
{
  "citation": "@article{lecun2010mnist,\n  title={MNIST handwritten digit database},\n  author={LeCun, Yann and Cortes, Corinna and Burges, CJ},\n  journal={ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist},\n  volume={2},\n  year={2010}\n}\n",
  "description": "The MNIST database of handwritten digits.",
  "location": {
    "urls": [
      "http://yann.lecun.com/exdb/mnist/"
    ]
  },
  "name": "mnist",
  "supervisedKeys": {
    "input": "image",
    "output": "label"
  },
  "version": "1.0.0"
}
>>>
>>> import json
>>> with open('/users4/zsun/tensorflow_datasets/mnist/1.0.0/dataset_info.json', 'w') as f:
>>>     json.dump(mnist.info.as_json, f)


  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_info.py", line 546, in read_from_json
    dataset_info_pb2.DatasetInfo())
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 430, in Parse
    return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 450, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/users4/zsun/anaconda3/envs/GAN/lib/python3.6/site-packages/google/protobuf/json_format.py", line 592, in _ConvertFieldValuePair
    raise ParseError(str(e))
google.protobuf.json_format.ParseError: Message type "tensorflow_datasets.DatasetInfo" has no field named "{".
 Available Fields(except extensions): ['name', 'description', 'version', 'citation', 'sizeInBytes', 'location', 'downloadChecksums', 'schema', 'splits', 'supervisedKeys']

mnist = tfds.builder('mnist')
mnist.download_and_prepare()