python - Tensorflow - adding L2 regularization loss simple example -
i familiar machine learning, learning tensorflow on own reading slides universities. below i'm setting loss function linear regression 1 feature. i'm adding l2 loss total loss, not sure if i'm doing correctly:
# regularization reg_strength = 0.01 # create loss function. tf.variable_scope("linear-regression"): w = tf.get_variable("w", shape=(1, 1), initializer=tf.contrib.layers.xavier_initializer()) b = tf.get_variable("b", shape=(1,), initializer=tf.constant_initializer(0.0)) yhat = tf.matmul(x, w) + b error_loss = tf.reduce_sum(((y - yhat)**2)/number_of_examples) #reg_loss = reg_strength * tf.nn.l2_loss(w) # reg 1 reg_loss = reg_strength * tf.reduce_sum(w**2) # reg 2 loss = error_loss + reg_loss # set optimizer. opt_operation = tf.train.gradientdescentoptimizer(0.001).minimize(loss)
my specific questions are:
i have 2 lines (commented
reg 1
,reg 2
) compute l2 loss of weightw
. line markedreg 1
uses tensorflow built-in function. these 2 l2 implementations equivalent?am adding regularization loss
reg_loss
correctly final loss function?
almost
according l2loss operation code
output.device(d) = (input.square() * static_cast<t>(0.5)).sum();
it multiplies 0.5 (or in other words divides 2
)
Comments
Post a Comment