我想对keras.datasets库中存在的CIFAR-10数据集的60000个观察值进行随机化。我知道对于构建神经网络来说,它可能不是那么重要,但我是Python新手,我想学习用这种编程语言处理数据。
因此,要导入数据集,我运行以下命令
from keras.datasets import cifar10
(X_train, Y_train), (X_test, Y_test) = cifar10.load_data()
这会自动给出训练集和测试集的默认细分;但我想混合它们。我想到的步骤是:
但也许有一种更快的方法可以做到这一点。
我尝试了几个小时的不同方法,但我什么也没能做到。有人能帮帮我吗?我真的很感激,谢谢。
发布于 2018-12-12 05:31:01
您可以使用sklearn.model_selection.train_test_split
拆分数据。如果您想在每次运行代码时使用相同的随机化索引选择,则可以设置random_state
值,这样每次都会有相同的测试/训练拆分。
from keras.datasets import cifar10
(X_train, Y_train), (X_test, Y_test) = cifar10.load_data()
# View first image
import matplotlib.pyplot as plt
plt.imshow(X_train[0])
plt.show()
import numpy as np
from sklearn.model_selection import train_test_split
# Concatenate train and test images
X = np.concatenate((X_train,X_test))
y = np.concatenate((Y_train,Y_test))
# Check shape
print(X.shape) # (60000, 32, 32, 3)
# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=10000, random_state=1234)
# Check shape
print(X_train.shape) # (50000, 32, 32, 3)
# View first image
plt.imshow(X_train[0])
plt.show()
发布于 2018-12-12 05:34:02
这是你要求的一个完整的演示。首先,我们下载数据并将其随机化一次,然后将第一个50K用于训练,其余10K用于验证目的。
In [21]: import tensorflow
In [22]: import tensorflow.keras.datasets as datasets
In [23]: cifar10 = datasets.cifar10.load_data()
In [24]: (X_train, Y_train), (X_test, Y_test) = datasets.cifar10.load_data()
In [25]: X_train.shape, Y_train.shape
Out[25]: ((50000, 32, 32, 3), (50000, 1))
In [26]: X_test.shape, Y_test.shape
Out[26]: ((10000, 32, 32, 3), (10000, 1))
In [27]: import numpy as np
In [28]: X, Y = np.vstack((X_train, X_test)), np.vstack((Y_train, Y_test))
In [29]: X.shape, Y.shape
Out[29]: ((60000, 32, 32, 3), (60000, 1))
In [30]: # Shuffle only the training data along axis 0
...: def shuffle_train_data(X_train, Y_train):
...: """called after each epoch"""
...: perm = np.random.permutation(len(Y_train))
...: Xtr_shuf = X_train[perm]
...: Ytr_shuf = Y_train[perm]
...:
...: return Xtr_shuf, Ytr_shuf
In [31]: X_shuffled, Y_shuffled = shuffle_train_data(X, Y)
In [32]: (X_train_new, Y_train_new) = X_shuffled[:50000, ...], Y_shuffled[:50000, ...]
In [33]: (X_test_new, Y_test_new) = X_shuffled[50000:, ...], Y_shuffled[50000:, ...]
In [34]: X_train_new.shape, Y_train_new.shape
Out[34]: ((50000, 32, 32, 3), (50000, 1))
In [35]: X_test_new.shape, Y_test_new.shape
Out[35]: ((10000, 32, 32, 3), (10000, 1))
我们有一个函数shuffle_train_data
,它一致地打乱数据,使示例及其标签保持相同的顺序。
https://stackoverflow.com/questions/53731141
复制相似问题