从keras dropout层提取dropout蒙版？
我想在训练时从Sequential Keras模型中的dropout层提取并存储1/0的dropout掩码数组。我想知道在Keras中是否有一种直接的方法来做到这一点，或者我是否需要切换到tensorflow (How to get the dropout mask in Tensorflow)。
将非常感谢您的帮助！我对TensorFlow和Keras还很陌生。
dropout层有几个函数(dropout_layer.get_output_mask()，dropout_layer.get_input_mask())，我试着使用它，但在调用上一层之后得到了None。
model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten(name="flat", input_shape=(28, 28, 1)))
model.add(tf.keras.layers.Dense(
512,
activation='relu',
name = 'dense_1',
kernel_initializer=tf.keras.initializers.GlorotUniform(seed=123),
bias_initializer='zeros'))
dropout = tf.keras.layers.Dropout(0.2, name = 'dropout') #want this layer's mask
model.add(dropout)
x = dropout.output_mask
y = dropout.input_mask
model.add(tf.keras.layers.Dense(
10,
activation='softmax',
name='dense_2',
kernel_initializer=tf.keras.initializers.GlorotUniform(seed=123),
bias_initializer='zeros'))
model.compile(...)
model.fit(...)回答 2
Stack Overflow用户
发布于 2019-09-21 04:03:28
它在Keras中不容易暴露。它深入到调用Tensorflow dropout为止。
因此，尽管您使用的是Keras，但它也将是图中的一个张量，可以通过名称获取(找到它的名称：In Tensorflow, get the names of all the Tensors in a graph)。
当然，这个选项缺少一些keras信息，您可能必须在Lambda层中执行此操作，以便Keras向张量添加某些信息。你必须格外小心，因为张量即使在没有训练的情况下也会存在(跳过掩模)。
现在，您也可以使用一种不那么复杂的方法，这可能会消耗一些处理：
def getMask(x):
boolMask = tf.not_equal(x, 0)
floatMask = tf.cast(boolMask, tf.float32) #or tf.float64
return floatMask使用Lambda(getMasc)(output_of_dropout_layer)
但是，您将需要一个函数式API Model，而不是使用Sequential模型。
inputs = tf.keras.layers.Input((28, 28, 1))
outputs = tf.keras.layers.Flatten(name="flat")(inputs)
outputs = tf.keras.layers.Dense(
512,
# activation='relu', #relu will be a problem here
name = 'dense_1',
kernel_initializer=tf.keras.initializers.GlorotUniform(seed=123),
bias_initializer='zeros')(outputs)
outputs = tf.keras.layers.Dropout(0.2, name = 'dropout')(outputs)
mask = Lambda(getMask)(outputs)
#there isn't "input_mask"
#add the missing relu:
outputs = tf.keras.layers.Activation('relu')(outputs)
outputs = tf.keras.layers.Dense(
10,
activation='softmax',
name='dense_2',
kernel_initializer=tf.keras.initializers.GlorotUniform(seed=123),
bias_initializer='zeros')(outputs)
model = Model(inputs, outputs)
model.compile(...)
model.fit(...)训练和预测
由于您不能训练面具(这没有任何意义)，因此它不应该是用于训练的模型的输出。
现在，我们可以试试这个：
trainingModel = Model(inputs, outputs)
predictingModel = Model(inputs, [output, mask]) 但是在预测中不存在掩模，因为dropout只应用于训练。所以这最终并没有给我们带来任何好处。
训练的唯一方法是使用虚拟损失和虚拟目标：
def dummyLoss(y_true, y_pred):
return y_true #but this might evoke a "None" gradient problem since it's not trainable, there is no connection to any weights, etc.
model.compile(loss=[loss_for_main_output, dummyLoss], ....)
model.fit(x_train, [y_train, np.zeros((len(y_Train),) + mask_shape), ...)不能保证这些方法会起作用。
Stack Overflow用户
发布于 2019-11-01 02:27:42
我发现了一种非常老套的方法，通过简单地扩展所提供的dropout层来实现这一点。(几乎所有代码都来自TF。)
class MyDR(tf.keras.layers.Layer):
def __init__(self,rate,**kwargs):
super(MyDR, self).__init__(**kwargs)
self.noise_shape = None
self.rate = rate
def _get_noise_shape(self,x, noise_shape=None):
# If noise_shape is none return immediately.
if noise_shape is None:
return array_ops.shape(x)
try:
# Best effort to figure out the intended shape.
# If not possible, let the op to handle it.
# In eager mode exception will show up.
noise_shape_ = tensor_shape.as_shape(noise_shape)
except (TypeError, ValueError):
return noise_shape
if x.shape.dims is not None and len(x.shape.dims) == len(noise_shape_.dims):
new_dims = []
for i, dim in enumerate(x.shape.dims):
if noise_shape_.dims[i].value is None and dim.value is not None:
new_dims.append(dim.value)
else:
new_dims.append(noise_shape_.dims[i].value)
return tensor_shape.TensorShape(new_dims)
return noise_shape
def build(self, input_shape):
self.noise_shape = input_shape
print(self.noise_shape)
super(MyDR,self).build(input_shape)
@tf.function
def call(self,input):
self.noise_shape = self._get_noise_shape(input)
random_tensor = tf.random.uniform(self.noise_shape, seed=1235, dtype=input.dtype)
keep_prob = 1 - self.rate
scale = 1 / keep_prob
# NOTE: if (1.0 + rate) - 1 is equal to rate, then we want to consider that
# float to be selected, hence we use a >= comparison.
self.keep_mask = random_tensor >= self.rate
#NOTE: here is where I save the binary masks.
#the file grows quite big!
tf.print(self.keep_mask,output_stream="file://temp/droput_mask.txt")
ret = input * scale * math_ops.cast(self.keep_mask, input.dtype)
return rethttps://stackoverflow.com/questions/58033895
复制相似问题
腾讯云开发者
