TVP

# 深度学习中的正则化技术

1. 什么是正则化？

2. 正则化如何减少过拟合？

3. 深度学习中的各种正则化技术：

L2和L1正则化

Dropout

4. 案例：在MNIST数据集上使用Keras的案例研究

1. 什么是正则化?

Source: Slideplayer

2. 正则化如何减少过拟合?

3. 深度学习中的各种正则化技术

L1和L2正则化

L1和L2是最常见的正则化类型。它们通过增加一个被称为正则项的额外项来更新成本函数：

Cost function = Loss (say, binary cross entropy) +Regularization term

L2中，我们有：

from keras import regularizers

kernel_regularizer=regularizers.l2(0.01)

Dropout

dropout做什么呢？每次迭代，随机选择一些节点，将它们连同相应的输入和输出一起删掉，如下图：

model = Sequential([

Dense(output_dim=hidden1_num_units, input_dim=input_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=output_num_units, input_dim=hidden5_num_units, activation='softmax'),

])

datagen = ImageDataGenerator(horizontal flip=True)

datagen.fit(train)

from keras.callbacks import EarlyStopping

EarlyStopping(monitor='val_err', patience=5)

Patience表示在该数量的epochs内没有进一步的性能改进后，就停止训练。为了更好地理解，我们再看看上面的图。在虚线之后，每个epoch都会导致一个更高的验证集错误。因此，在虚线之后的5个epoch（因为我们设置patience等于5），由于没有进一步的改善，模型将停止训练。

%pylab inline

import numpy as np

import pandas as pd

from sklearn.metrics import accuracy_score

from matplotlib import pyplot

import tensorflow as tf

import keras

# To stop potential randomness

seed = 128

rng = np.random.RandomState(seed)

img_name = rng.choice(train.filename)

pylab.imshow(img, cmap='gray')

pylab.axis('off')

pylab.show()

#storing images in numpy arrays

temp = []

for img_name in train.filename:

img = img.astype('float32')

temp.append(img)

x_train = np.stack(temp)

x_train /= 255.0

x_train = x_train.reshape(-1, 784).astype('float32')

split_size = int(x_train.shape[0]*0.7)

x_train, x_test = x_train[:split_size], x_train[split_size:]

y_train, y_test = y_train[:split_size], y_train[split_size:]

# import keras modules

from keras.models import Sequential

from keras.layers import Dense

# define vars

input_num_units = 784

hidden1_num_units = 500

hidden2_num_units = 500

hidden3_num_units = 500

hidden4_num_units = 500

hidden5_num_units = 500

output_num_units = 10

epochs = 10

batch_size = 128

model = Sequential([

Dense(output_dim=hidden1_num_units, input_dim=input_num_units, activation='relu'),

Dense(output_dim=hidden2_num_units, input_dim=hidden1_num_units, activation='relu'),

Dense(output_dim=hidden3_num_units, input_dim=hidden2_num_units, activation='relu'),

Dense(output_dim=hidden4_num_units, input_dim=hidden3_num_units, activation='relu'),

Dense(output_dim=hidden5_num_units, input_dim=hidden4_num_units, activation='relu'),

Dense(output_dim=output_num_units, input_dim=hidden5_num_units, activation='softmax'),

])

trained_model_5d = model.fit(x_train, y_train, nb_epoch=epochs, batch_size=batch_size, validation_data=(x_test, y_test))

from keras import regularizers

model = Sequential([

Dense(output_dim=hidden1_num_units, input_dim=input_num_units, activation='relu',

kernel_regularizer=regularizers.l2(0.0001)),

Dense(output_dim=hidden2_num_units, input_dim=hidden1_num_units, activation='relu',

kernel_regularizer=regularizers.l2(0.0001)),

Dense(output_dim=hidden3_num_units, input_dim=hidden2_num_units, activation='relu',

kernel_regularizer=regularizers.l2(0.0001)),

Dense(output_dim=hidden4_num_units, input_dim=hidden3_num_units, activation='relu',

kernel_regularizer=regularizers.l2(0.0001)),

Dense(output_dim=hidden5_num_units, input_dim=hidden4_num_units, activation='relu',

kernel_regularizer=regularizers.l2(0.0001)),

Dense(output_dim=output_num_units, input_dim=hidden5_num_units, activation='softmax'),

])

trained_model_5d = model.fit(x_train, y_train, nb_epoch=epochs, batch_size=batch_size, validation_data=(x_test, y_test))

## l1

model = Sequential([

Dense(output_dim=hidden1_num_units, input_dim=input_num_units, activation='relu',

kernel_regularizer=regularizers.l1(0.0001)),

Dense(output_dim=hidden2_num_units, input_dim=hidden1_num_units, activation='relu',

kernel_regularizer=regularizers.l1(0.0001)),

Dense(output_dim=hidden3_num_units, input_dim=hidden2_num_units, activation='relu',

kernel_regularizer=regularizers.l1(0.0001)),

Dense(output_dim=hidden4_num_units, input_dim=hidden3_num_units, activation='relu',

kernel_regularizer=regularizers.l1(0.0001)),

Dense(output_dim=hidden5_num_units, input_dim=hidden4_num_units, activation='relu',

kernel_regularizer=regularizers.l1(0.0001)),

Dense(output_dim=output_num_units, input_dim=hidden5_num_units, activation='softmax'),

])

trained_model_5d = model.fit(x_train, y_train, nb_epoch=epochs, batch_size=batch_size, validation_data=(x_test, y_test))

## dropout

model = Sequential([

Dense(output_dim=hidden1_num_units, input_dim=input_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden2_num_units, input_dim=hidden1_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden3_num_units, input_dim=hidden2_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden4_num_units, input_dim=hidden3_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden5_num_units, input_dim=hidden4_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=output_num_units, input_dim=hidden5_num_units, activation='softmax'),

])

trained_model_5d = model.fit(x_train, y_train, nb_epoch=epochs, batch_size=batch_size, validation_data=(x_test, y_test))

datagen = ImageDataGenerator(zca_whitening=True)

temp = []

for img_name in train.filename:

img = img.astype('float32')

temp.append(img)

x_train = np.stack(temp)

X_train = x_train.reshape(x_train.shape[0], 1, 28, 28)

X_train = X_train.astype('float32')

# fit parameters from data

datagen.fit(X_train)

## splitting

split_size = int(x_train.shape[0]*0.7)

x_train, x_test = X_train[:split_size], X_train[split_size:]

y_train, y_test = y_train[:split_size], y_train[split_size:]

## reshaping

x_train=np.reshape(x_train,(x_train.shape[0],-1))/255

x_test=np.reshape(x_test,(x_test.shape[0],-1))/255

## structure using dropout

model = Sequential([

Dense(output_dim=hidden1_num_units, input_dim=input_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden2_num_units, input_dim=hidden1_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden3_num_units, input_dim=hidden2_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden4_num_units, input_dim=hidden3_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=hidden5_num_units, input_dim=hidden4_num_units, activation='relu'),

Dropout(0.25),

Dense(output_dim=output_num_units, input_dim=hidden5_num_units, activation='softmax'),

])

trained_model_5d = model.fit(x_train, y_train, nb_epoch=epochs, batch_size=batch_size, validation_data=(x_test, y_test))

from keras.callbacks import EarlyStopping

trained_model_5d = model.fit(x_train, y_train, nb_epoch=epochs, batch_size=batch_size, validation_data=(x_test, y_test)

, callbacks = [EarlyStopping(monitor='val_acc', patience=2)])

https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques/

• 发表于:
• 原文链接https://kuaibao.qq.com/s/20180528A07BHQ00?refer=cp_1026
• 腾讯「腾讯云开发者社区」是腾讯内容开放平台帐号（企鹅号）传播渠道之一，根据《腾讯内容开放平台服务协议》转载发布内容。
• 如有侵权，请联系 cloudcommunity@tencent.com 删除。

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25

2024-06-25