Lab09-3. Sigmoid Backpropagation

Backpropagation

1. Backpropagation

• 네트워크 전체에 대해 반복적인 연쇄법칙(Chain rule)을 적용하여 그라디언트(Grient)를 계산하는 방법중 하나

2. Sigmoid 미분

$$\begin{align*} sigmoid(x) &= \frac{1}{1+e^{-wx}}\\\\ \frac{\partial}{\partial x}sigmoid(x) &= \frac{\partial}{\partial x}\left(\frac{1}{1+e^{-wx}} \right)\\\\ &= -\frac{\frac{\partial}{\partial x}(1+e^{-wx})}{(1+e^{^-wx})^2} \\\\ &= -\frac{\frac{\partial}{\partial x}1+\frac{\partial}{\partial X}(e^{-wx})}{(1+e^{^-wx})^2} \\\\ &= -\frac{0+-(e^{-wx})}{(1+e^{^-wx})^2} \\\\ &= -\left(\frac{-1*(e^{-wx})}{(1+e^{^-wx})^2}\right) \\\\ &= \frac{(e^{-wx})}{(1+e^{^-wx})^2} \\\\ &= \frac{(1+e^{-wx}-1)}{(1+e^{^-wx})^2} \\\\ &= \frac{(1+e^{-wx})}{(1+e^{^-wx})^2} - \frac{1}{(1+e^{^-wx})^2} \\\\ &= \frac{1}{1+e^{^-wx}} - \frac{1}{(1+e^{^-wx})^2} \\\\ &= \frac{1}{1+e^{^-wx}}\left(1 - \frac{1}{1+e^{^-wx}} \right) \\\\ &= sigmoid(x)(1-sigmoid(x)) \end{align*}$$

 

2. Cost(Loss) 함수 미분

$$\begin{align*} a &= sigmoid(x) = \frac{1}{1+e^{-wx}}\\\\ E &= loss(a, t) \\ &= -t\log{}{a}+(1-t)\log(1-a) \\\\ \frac{\partial E}{\partial a} &= \frac{\partial E}{\partial a} \left( -t\log{}{a}+(1-t)\log(1-a) \right) \\\\ &= \frac{\partial E}{\partial a}\left(-t\log{}{a} \right) + \frac{\partial E}{\partial a}\left((1-t)\log{}{(1-a)} \right) \\\\ &= \frac{-t}{a} + \frac{1-t}{1-a} \\\\ &= \frac{-t(1-a)+a(1-t))}{a(1-a)} \\\\ &= \frac{-t+at+a-at}{a(1-a)} \\\\ &= \frac{a-t}{a(1-a)} \end{align*}$$

 

3. Sigmoid Backpropagation

 

1) Chain rule에 따라 계산

$$\begin{align*} (1) \frac{\partial E}{\partial a_1} &= \frac{a_1-t}{a_1(1-a_1)} \\\\ (2) \frac{\partial E}{\partial l} &= \frac{\partial E}{\partial a_1}*\frac{\partial a_1}{\partial l} \\\\ &= \frac{a_1-t}{a_1(1-a_1)} *a_1(1-a_1) \\\\ &= a_1-t \\\\ (3) \frac{\partial E}{\partial b} &= \frac{\partial E}{\partial l}*\frac{\partial l}{\partial b} \\\\ &= (a_1-t)*1 \\\\ &= a_1-t\\\\ (4) \frac{\partial E}{\partial o} &=\frac{\partial E}{\partial l}*\frac{\partial l}{\partial o} \\\\ &= (a_1-t)*1 \\\\ &= a_1-t \\\\ (5) \frac{\partial E}{\partial w} &=\frac{\partial E}{\partial o}*\frac{\partial o}{\partial w} \\\\ &= (a_1-t)*a_0 \\\\ &=(a_1-t)*a_0^T \end{align*}$$

 

2) Sigmoid Backpropagation

• 소스코드

  • 행렬을 transpose(전치)하는 것의 이유는 모르겠음...

     

  import tensorflow as tf
  import numpy as np
  
  xy = np.loadtxt('Data/data-04-zoo.csv', delimiter=',', dtype=np.float32)
  X_data = xy[:, 0:-1]
  Y_data = xy[:, [-1]]
  
  print("Shape of X data: ", X_data.shape)
  print("Shape of Y data: ", Y_data.shape)
  
  print("Y data unique values: ", np.unique(Y_data))
  
  nb_classes = 7 # 0~6
  
  X = tf.placeholder(tf.float32, [None, 16])
  Y = tf.placeholder(tf.int32, [None, 1])
  
  target = tf.one_hot(Y, nb_classes)
  print("Before reshape: ", target.shape)
  target = tf.reshape(target, [-1, nb_classes])
  print("After reshape: ", target.shape)
  target = tf.cast(target, tf.float32)
  
  W = tf.Variable(tf.random_normal([16, nb_classes]), name='weight')
  b = tf.Variable(tf.random_normal([nb_classes]), name='bias')
  
  def sigmoid_func(x):
      return 1. / (1. + tf.exp(-x))
  
  def sigmoid_func_prime(x):
      return sigmoid_func(x) * (1. - sigmoid_func(x))
  
  layer1 = tf.matmul(X, W) + b
  y_hat = sigmoid_func(layer1)
  
  loss_i = -target*tf.log(y_hat) + (1-target)*tf.log(1-y_hat)
  loss = tf.reduce_sum(loss_i)
  
  d_loss = (y_hat - target) / ((1. - y_hat) + 1e-7)
  d_sigmoid = sigmoid_func_prime(layer1)
  d_layer = d_loss * d_sigmoid
  d_b = d_layer
  d_W = tf.matmul(tf.transpose(X), d_layer)
  
  learning_rate = 0.01
  train_step = [
      tf.assign(W, W - learning_rate * d_W),
      tf.assign(b, b - learning_rate * tf.reduce_sum(d_b)),
  ]
  
  prediction = tf.argmax(y_hat, 1)
  acct_mat = tf.equal(tf.argmax(y_hat, 1), tf.argmax(target, 1))
  acct_res = tf.reduce_mean(tf.cast(acct_mat, tf.float32))
  
  with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
  
      for step in range(500):
          sess.run(train_step, feed_dict={X: X_data, Y: Y_data})
  
          if step % 10 == 0:
              step_loss, acc = sess.run([loss, acct_res],
                                        feed_dict={X: X_data, Y: Y_data})
              print("Step: {:5} \t Loss: {:10.5f}\t Acc: {:.2%}".format(step, step_loss, acc))
  
      pred = sess.run(prediction, feed_dict={X: X_data})
      for p, y in zip(pred, Y_data):
          msg = "[{}]\tPrediction: {:d} \tY: {:d}"
          print(msg.format(p == int(y[0]), p, int(y[0])))