weight initialization techniques in nural network

 weight :

      Weight is the parameter within a neural network that transforms input data within the network's hidden layers. A neural network is a series of nodes, or neurons. Within each node is a set of inputs, weight, and a bias value.A weight represent the strength of connections.

you can find
the number of weights by counting the edges in that network.

Why weight is initialized in neural network?

The weights of artificial neural networks must be initialized to small random numbers. This is because this is an expectation of the stochastic optimization algorithm used to train the model, called stochastic gradient descent

Why is weight initialization important?

Why Initialize Weights. The aim of weight initialization is to prevent layer activation outputs from exploding or vanishing during the course of a forward pass through a deep neural network

weight initialization types :

1.zeros class

2.ones class

3.glortnormal class

4.orthoganal class

5.identity class

6.Henormal class

zeros class:

If all the weights are initialized to zeros, the derivatives will remain same for every w in W[l]. As a result, neurons will learn same features in each iterations.

python:

#syntax

tf.keras.initializers.Zeros()

         

>>> # Usage in a Keras layer:

>>> initializer = tf.keras.initializers.Zeros()

>>> layer = tf.keras.layers.Dense(3, kernel_initializer=initializer)

ones class:

 This technique tries to address the problems of zero initialization since it prevents neurons from learning the same features of their inputs since our goal is to make each neuron learn different functions of its input and this technique gives much better accuracy than zero initialization.

python:

tf.keras.initializers.Ones()

glortnormal class:

initializers. glorot_normal . Draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / (fan_in + fan_out)) where fan_in is the number of input units in the weight tensor and fan_out is the number of output units in the weight tensor.

python:

tf.keras.initializers.GlorotNormal(seed=None)

orthoganal class:

Orthogonal initialization is a simple yet relatively effective way of combatting exploding and vanishing gradients, especially when paired with other methods such as gradient clipping and more advanced architectures

python:

>>> # Standalone usage:

>>> initializer = tf.keras.initializers.Orthogonal()

>>> values = initializer(shape=(2, 2))

identity class:

Initializer that generates the identity matrix.

Also available via the shortcut function tf.keras.initializers.identity.

Only usable for generating 2D matrices.

python:

>>> # Standalone usage:

>>> initializer = tf.keras.initializers.Identity()

>>> values = initializer(shape=(2, 2))

Henormal class:

He normal initializer.

Also available via the shortcut function tf.keras.initializers.he_normal.

It draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / fan_in) where fan_in is the number of input units in the weight tensor.

python:

>>> # Standalone usage:

>>> initializer = tf.keras.initializers.HeNormal()

>>> values = initializer(shape=(2, 2))