我想在我的TensorFlow图中为几个权重矩阵添加一个最大范数约束,ala Torch的
renorm方法.
如果任何神经元的权重矩阵的L2范数超过max_norm,我想缩减其权重,使其L2范数正好是max_norm.
使用TensorFlow表达这一点的最佳方式是什么？
解决方法
这是一个可能的实现：
import tensorflow as tf
def maxnorm_regularizer(threshold,axes=1,name="maxnorm",collection="maxnorm"):
def maxnorm(weights):
clipped = tf.clip_by_norm(weights,clip_norm=threshold,axes=axes)
clip_weights = tf.assign(weights,clipped,name=name)
tf.add_to_collection(collection,clip_weights)
return None # there is no regularization loss term
return maxnorm
以下是您将如何使用它：
from tensorflow.contrib.layers import fully_connected
from tensorflow.contrib.framework import arg_scope
with arg_scope(
[fully_connected],weights_regularizer=max_norm_regularizer(1.5)):
hidden1 = fully_connected(X,200,scope="hidden1")
hidden2 = fully_connected(hidden1,100,scope="hidden2")
outputs = fully_connected(hidden2,5,activation_fn=None,scope="outs")
max_norm_ops = tf.get_collection("max_norm")
[...]
with tf.Session() as sess:
sess.run(init)
for epoch in range(n_epochs):
for X_batch,y_batch in load_next_batch():
sess.run(training_op,Feed_dict={X: X_batch,y: y_batch})
sess.run(max_norm_ops)
这将创建一个3层神经网络,并在每一层(阈值为1.5)对其进行最大范数正则化训练.我只是尝试过,似乎工作.希望这可以帮助！欢迎提出改进建议. 原文链接：https://www.f2er.com/python/241899.html