# MNIST 数据集载入线性模型

Brief 概述

.jpeg: height, width, channels

.png : height, width, channels, alpha

p.s. 注意 .png 储存格式的图片含有透明度的信息，在处理图片的时候可以舍弃。

Binary Data 二进制数据

Reasons for using binary data 使用二进制数据的理由

The approach to load images 读取数据的方法

Code[1]

import gzip, os

import numpyasnp

location =input('The directory of MNIST dataset: ')

path = os.path.join(location,'train-images-idx3-ubyte.gz')

try:

with gzip.open(path,'rb')asfi:

data_i = np.frombuffer(fi.read(), dtype=np.int8, offset=16)

images_flat_all = data_i.reshape(-1,784)

print(images_flat_all)

print('----- Separation -----')

print('Size of images_flat: ',len(images_flat_all))

except:

print("The file directory doesn't exist!")

The directory of MNIST dataset: /Users/kcl/Documents/Python_Projects/01_AI_Zero_to_Master/_2_Image_Datasets/MNIST_data

[[0 0 0 ... 0 0 0]

[0 0 0 ... 0 0 0]

[0 0 0 ... 0 0 0]

...

[0 0 0 ... 0 0 0]

[0 0 0 ... 0 0 0]

[0 0 0 ... 0 0 0]]

----- Separation -----

Size of images_flat:60000

Code[2]

[54...568]

----- Separation -----

Sizeofimages_labels:60000

p.s. 数据储存格式同理测试集与其他种类数据库

Explanation to the code 代码说明

MNIST 原始数据中直到第十六位数才开始描述图像信息，而数据标签则是第八位就开始描述信息，因此 offset 设置从第十六或是八位开始读取

Linear Model 线性模型

Code[3]

#import numpy as np

#import tensorflow as tf

#x_data = np.random.rand(100).astype(np.float32)

#y_data = x_data * 0.1 + 0.3

#weight = tf.Variable(tf.random_uniform(shape=[1], minval=-1.0, maxval=1.0))

#bias = tf.Variable(tf.zeros(shape=[1]))

#y = weight * x_data + bias

#loss = tf.reduce_mean(tf.square(y - y_data))

#training = optimizer.minimize(loss)

#sess = tf.Session()

#init = tf.global_variables_initializer()

#sess.run(init)

#forstepinrange(101):

#sess.run(training)

#ifstep % 10 == 0:

#print('Round {}, weight: {}, bias: {}'

#.format(step, sess.run(weight[0]), sess.run(bias[0])))

MNIST in Linear Model

batch size： 每一批次训练图片的数量需要调控以免内存不够

loss function: 损失函数的原理是计算预测和实际答案之间的差距

Code[4]

import gzip, os

import numpy as np

################ Step No.1 to well manage the dataset. ################

classMNIST:

# Images size is told in the official website 28*28 px.

image_size =28

image_size_flat = image_size * image_size

# Let the validation set flexible when making an instance.

def__init__(self, val_ratio=.1, data_dir='MNIST_data'):

self.val_ratio = val_ratio

self.data_dir = data_dir

# Load 4 files to individual lists with one string pixels.

# Determine the actual number of training / validation sets.

self.val_train_num = round(len(img_train) *self.val_ratio)

self.main_train_num = len(img_train) -self.val_train_num

# The normalized image pixels value can be more convenient when training.

# dtype=np.int64 would be more general when applying to Tensorflow.

self.img_train = img_train[:self.main_train_num] /255.0

self.lab_train = lab_train[:self.main_train_num].astype(np.int)

self.img_train_val = img_train[self.main_train_num:] /255.0

self.lab_train_val = lab_train[self.main_train_num:].astype(np.int)

# Also convert the format of testing set.

self.img_test = img_test /255.0

self.lab_test = lab_test.astype(np.int)

# Extract the same codes from "load_flat_images" and "load_labels".

# This method won't be called during training procedure.

path = os.path.join(self.data_dir, dataset_name)

with gzip.open(path,'rb') asbinary_file:

# The datasets files are stored in 8 bites, mind the format.

data = np.frombuffer(binary_file.read(), np.uint8, offset=offset)

returndata

# This method won't be called during training procedure.

# Images offset position is 16 by default format

images_flat_all = data.reshape(-1,self.image_size_flat)

returnimages_flat_all

# This method won't be called during training procedure.

# Labels offset position is 8 by default format.

returnlabels_all

# This method would be called for training usage.

defone_hot(self, labels):

# Properly use numpy module to mimic the one hot effect.

class_num = np.max(self.lab_test) +1

convert = np.eye(class_num, dtype=float)[labels]

returnconvert

#---------------------------------------------------------------------#

path = input("The directory of MNIST dataset: ")

data = MNIST(val_ratio=.1, data_dir=path)

The directoryofMNIST dataset:/Users/kcl/Documents/Python_Projects/01_AI_Zero_to_Master/_2_Image_Datasets/MNIST_data

Code[5]

importtensorflowastf

fromtqdmimporttqdm

flat_size = data.image_size_flat

label_num = np.max(data.lab_test) +1

################ Step No.2 to construct tensor graph. ################

x_train= tf.placeholder(dtype=tf.float32, shape=[None, flat_size])

t_label_oh = tf.placeholder(dtype=tf.float32, shape=[None, label_num])

t_label = tf.placeholder(dtype=tf.int64, shape=[None])

################ These are the values ################

# Initialize the beginning weights and biases by random_normal method.

weights = tf.Variable(tf.random_normal([flat_size, label_num],

mean=10.0, stddev=1.0,

dtype=tf.float32))

biases = tf.Variable(tf.random_normal([label_num], mean=0.0, stddev=1.0,

dtype=tf.float32))

########### that we wish to get by training ##########

logits = tf.matmul(x_train, weights) + biases#

# Shrink the distances between values into 0 to 1 by softmax formula.

p_label_soh = tf.nn.softmax(logits)

# Pick the position of largest value along y axis.

p_label = tf.argmax(p_label_soh, axis=1)

#---------------------------------------------------------------------#

####### Step No.3 to get a loss value by certain loss function. #######

# This softmax function can not accept input being "softmaxed" before.

CE = tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels=t_label_oh)

# Shrink all loss values in a matrix to only one averaged loss.

loss = tf.reduce_mean(CE)

#---------------------------------------------------------------------#

#### Step No.4 get a minimized loss value using gradient descent. ####

# Decrease this only averaged loss to a minimum value by using gradient descent.

#---------------------------------------------------------------------#

# First return a boolean list values by tf.equal function

correct_predict = tf.equal(p_label, t_label)

# And cast them into 0 and 1 values so that its average value would be accuracy.

accuracy = tf.reduce_mean(tf.cast(correct_predict, dtype=tf.float32))

sess = tf.Session()

sess.run(tf.global_variables_initializer())

###### Step No.5 iterate the training set and check the accuracy. #####

# The trigger to train the linear model with a defined cycles.

defoptimize(iteration, batch_size=32):

foriintqdm(range(iteration)):

total = len(data.lab_train)

random = np.random.randint(, total, size=batch_size)

# Randomly pick training images / labels with a defined batch size.

x_train_batch = data.img_train[random]

t_label_batch_oh = data.one_hot(data.lab_train[random])

batch_dict = {

x_train: x_train_batch,

t_label_oh: t_label_batch_oh

}

sess.run(optimizer, feed_dict=batch_dict)

# The trigger to check the current accuracy value

defAccuracy():

# Use the totally separate dataset to test the trained model

test_dict = {

x_train: data.img_test,

t_label_oh: data.one_hot(data.lab_test),

t_label: data.lab_test

}

Acc = sess.run(accuracy, feed_dict=test_dict)

print('Accuracy on Test Set: '.format(Acc))

#---------------------------------------------------------------------#

### Step No.6 plot wrong predicted pictures with its predicted label.##

importmatplotlib.pyplotasplt

# We can decide how many wrong predicted images are going to be shown up.

# We can focus on the specific wrong predicted labels

defwrong_predicted_images(pic_num=[3,4], label_number=None):

test_dict = {

x_train: data.img_test,

t_label_oh: data.one_hot(data.lab_test),

t_label: data.lab_test

}

correct_pred, p_lab = sess.run([correct_predict, p_label],

feed_dict=test_dict)

# To reverse the boolean value in order to pick up wrong labels

wrong_pred = (correct_pred ==False)

# Pick up the wrong doing elements from the corresponding places

wrong_img_test = data.img_test[wrong_pred]

wrong_t_label = data.lab_test[wrong_pred]

wrong_p_label = p_lab[wrong_pred]

fig, axes = plt.subplots(pic_num[], pic_num[1])

edge = data.image_size

foraxinaxes.flat:

# If we were not interested in certain label number,

# pick up the wrong predicted images randomly.

iflabel_numberisNone:

i = np.random.randint(, len(wrong_t_label),

size=None, dtype=np.int)

pic = wrong_img_test[i].reshape(edge, edge)

ax.imshow(pic, cmap='binary')

xlabel ="True: , Pred: ".format(wrong_t_label[i],

wrong_p_label[i])

# If we are interested in certain label number,

# pick up the specific wrong images number randomly.

else:

# Mind that np.where return a "tuple" that should be indexing.

specific_idx = np.where(wrong_t_label==label_number)[]

i = np.random.randint(, len(specific_idx),

size=None, dtype=np.int)

pic = wrong_img_test[specific_idx[i]].reshape(edge, edge)

ax.imshow(pic, cmap='binary')

xlabel ="True: , Pred: ".format(wrong_t_label[specific_idx[i]],

wrong_p_label[specific_idx[i]])

ax.set_xlabel(xlabel)

# Pictures don't need any ticks, so we remove them in both dimensions

ax.set_xticks([])

ax.set_yticks([])

plt.show()

#---------------------------------------------------------------------#

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning:compiletimeversion3.5of module'tensorflow.python.framework.fast_tensor_util'does notmatchruntimeversion3.6

returnf(*args, **kwds)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning:Conversion of the secondargumentof issubdtype from `float`to`np.floating`isdeprecated. In future, it willbetreatedas`np.float64 == np.dtype(float).type`.

from ._conv import register_convertersas_register_converters

Code[6]

print(x_train.shape)

Accuracy() #Accuracybeforedoinganything

optimize(10)

Accuracy() #Iterate10times

optimize(1000)

Accuracy() #Iterate10+1000times

optimize(10000)

Accuracy() #Iterate10+1000+10000times

(?, 784)

Accuracy on TestSet:10.58%

AccuracyonTestSet:48.68%

100%|██████████|1000/1000[00:00

2%|▏ |152/10000[00:00

AccuracyonTestSet:86.83%

100%|██████████|10000/10000[00:06

AccuracyonTestSet:88.52%

Annotation No.1 tf.matmul(x_train, weights)

Reason of using one_hot()

Code[7]

wrong_predicted_images(pic_num=[3,3], label_number=5)

Code[8]

wrong_predicted_images(pic_num=[3,3], label_number=4)

Code[9]

# When everythingisdone above, mindtoexecutethelinebelow.

# It willhelpustorelease occupied memory in our computer.

sess.close()

• 发表于:
• 原文链接https://kuaibao.qq.com/s/20181010A1MVL300?refer=cp_1026
• 腾讯「云+社区」是腾讯内容开放平台帐号（企鹅号）传播渠道之一，根据《腾讯内容开放平台服务协议》转载发布内容。
• 如有侵权，请联系 yunjia_community@tencent.com 删除。

2021-12-07

2021-12-07

2021-12-07

2021-12-07

2021-12-07

2021-12-07

2018-05-23

2018-04-25

2018-04-18

2018-04-10

2021-12-07

2021-12-07