Introduction to NN -Part 2

1 / 43
Outline

• Basic models of ANN

• Learning
• Activation Functions
• Important terminologies of ANNs

2 / 43
Basic models of ANN

3 / 43
Basic models of ANN

Figure 1: Basic models of ANN 4 / 43

Basic models of ANN

• Models are based on three entities

• The model’s synaptic interconnections.
• The training or learning rules adopted for updating and adjusting the connection weights.
• Their activation functions

5 / 43
Interconnections

6 / 43
Interconnections

• The arrangement of neurons to form layers and the connection pattern formed within
and between layers is called the network architecture.
• Five types:
1. Single layer feed forward network
2. Multilayer feed-forward network
3. Single node with its own feedback
4. Single-layer recurrent network
5. Multilayer recurrent network

7 / 43
Single layer Feed Forward Network
• Layer is formed by taking processing elements and combining it with other
processing elements.
• Input and output are linked with each other
• Inputs are connected to the processing nodes with various weights, resulting in series of
outputs one per node.

8 / 43
Multi-layer Feed forward Network

• Formed by the interconnection of several layers.

• Input layer receives input and buffers input signal.
• Output layer generated output.
• Layer between input and output is called hidden layer.
• Hidden layer is internal to the network.
• Zero to several hidden layers in a network.
• More the hidden layer, more is the complexity of network, but efficient output is
produced.

9 / 43
Feed Back Network

• If no neuron in the output layer is an input to a node in the same layer / proceeding layer –
feed forward network.
• If outputs are directed back as input to the processing elements in the same
layer/proceeding layer – feedback network.
• If the output are directed back to the input of the same layer then it is lateral feedback.

10 / 43
Recurrent Network

• Recurrent networks are networks with feedback networks with closed loop.
• Fig 2.8 -simple recurrent neural network having a single neuron with feedback to itself.
• Fig 2.9 single layer network with feedback from output can be directed to processing
element itself or to other processing element/both.

11 / 43
Single layer network with feedback
• It has single neuron with feedback to itself.
• In this a processing elements output can be directed back to the processing element itself
or to the other processing element or to both.

12 / 43
Multilayer network with feedback
• Processing element output can be directed back to the nodes in the preceding layer,
forming a multilayer recurrent network.
• Processing element output can be directed to processing element itself or to other
processing element in the same layer.

13 / 43
Learning Rule

14 / 43
Learning Rule

• Two broad kinds of learning in ANNs is:

i) parameter learning - updates connecting weights in a neural net.
ii) Structure learning - focus on change in the network.
• Apart from these, learning in ANN is classified into three categories as
i) supervised learning
ii) unsupervised learning
iii) reinforcement learning

15 / 43
Supervised Learning

• If there is a difference means an

• Learning with the help of a teacher.
error signal is generated by the
• Example learning process of a small child.
network.
• Child doesn’t know read/write.
• Their each & every action is supervised by a • It is used for adjustment of
teacher weights until actual output
• In ANN, each input vector requires a matches desired output.
corresponding target vector, which represents the
desired output.
• The input vector along with target vector is called
training pair.
• The input vector results in output vector.
• The actual output vector is compared with desired
output vector.
16 / 43
Unsupervised Learning

• Learning is performed without the help of a

teacher.
• Exact clusters are formed by
• Example: tadpole learn to swim by itself.
discovering similarities &
• In ANN, during training process, network receives dissimilarities so called as
input patterns and organize it to form clusters. self-organizing.
• From the Fig. it is observed that no feedback is
applied from environment to inform what output
should be or whether they are correct.
• The network itself discover
patterns, regularities, features/ categories from the input data
and relations for the input data over the output.

17 / 43
Reinforcement Learning

• Similar to supervised learning.

• Learning based on information is called
reinforcement learning & the feedback
sent is called reinforcement signal.
• The network receives some feedback
from the environment.
• Feedback is only evaluative.
• The external reinforcement signals are
processed in the critic signal generator,
and the obtained critic signals are sent to
the ANN for adjustment of weights
properly to get critic feedback in future.

18 / 43
Activation functions

19 / 43
Activation Functions

20 / 43
Activation Functions

• To make work more efficient and for exact output, some force or activation is given.
• Like that, activation function is applied over the net input to calculate the output of an
ANN.
• Information processing of processing element has two major parts: input and output.
• An integration function (f) is associated with input of processing element.

21 / 43
Activation Functions - Identity function

It is a linear function which is defined as f(x)=x for all x.

The output is same as the input.

22 / 43
Activation Functions - Binary step function

It is defined as

where 𝜃 represents thresh hold value.

It is used in single layer nets to convert the net input to an output that is binary (0 or 1).

23 / 43
Activation Functions - Bipolar step function

It is defined as

where 𝜃 represents threshold value.

used in single layer nets to convert the net input to an output that is bipolar (+1 or -1).

24 / 43
Activation Functions - Sigmoid function

Used in Back propagation nets.

Two types:
a) binary sigmoid function
• logistic sigmoid function or unipolar sigmoid
function.
• it is defined as
1
𝑓 (𝑥) = 1+𝑒−𝜆𝑥
where 𝜆-steepness parameter.
• The derivative of this function is
𝑓 0 (𝑥) = 𝜆 𝑓 (𝑥) [1 − 𝑓 (𝑥)].
• The range of sigmoid function is 0 to 1.

25 / 43
 Activation Functions

b)Bipolar sigmoid function

−𝜆𝑥
𝑓 (𝑥) = 1+𝑒2−𝜆𝑥 − 1 = 1−𝑒
1+𝑒−𝜆𝑥
where 𝜆-steepness parameter.
The range of sigmoid function is -1
to +1.

26 / 43
Activation Functions - Ramp function

27 / 43
Important terminologies of ANN

28 / 43
Important terminologies of ANNs

• Weights
• Bias
• Threshold
• Learning rate
• Momentum factor
• Vigilance parameter

29 / 43
Weight

• The weight contain information about the input signal.

• It is used by the net to solve the problem.
• It is represented in terms of matrix & called as connection matrix.
• If weight matrix W contains all the elements of an ANN, then the set of all W matrices
will determine the set of all possible information processing configuration.
• The ANN can be realized by finding an appropriate matrix W.
• Weight encode long-term memory (LTM) and the activation states of network encode
short-term memory (STM) in a neural network.

30 / 43
31 / 43
Bias

• It has an impact in calculating net input.

• It is included by adding a component
𝑥 0 = 1 to the input vector X.
• Thus input vector becomes
X=(1, 𝑋1 ,....𝑋𝑖 , .....𝑋𝑛 )
• The bias is considered like another
weight that is 𝑤 𝑜 𝑗 = 𝑏 𝑗
Í𝑛
𝑦 𝑖𝑛 𝑗 = 𝑖=0 𝑥𝑖 𝑤 𝑖 𝑗
𝑦 𝑖𝑛 𝑗 = 𝑥 0 𝑤 0 𝑗 + 𝑥 1 𝑤 1 𝑗 .... + 𝑥 𝑛 𝑤 𝑛 𝑗
Í𝑛
𝑦 𝑖𝑛 𝑗 = 𝑏 𝑗 + 𝑖=1 𝑥𝑖 𝑤 𝑖 𝑗
𝑤 𝑖 𝑗 is the weight from processing element
"i" (source node) to processing element
"j" (destination node)
32 / 43
The bias is of two types

• Positive bias - Increase the net input

• Negative bias - Decrease the net input
• The relationship between input and output given by the equation of straight line y=mx+c.

33 / 43
Threshold

• It is a set value based upon which the final output is calculated.

• Calculated net input and threshold is compared to get the network output.

34 / 43
Learning rate
• Denoted by 𝛼.
• Control the amount of weight adjustment at each step of training.
• The learning rate range from 0 to 1.
• Determine the rate of learning at each step
Momentum Factor
• Convergence is made faster if a momentum factor is added to the weight updation
process.
• Done in back propagation network.
Vigilance parameter
• Denoted by 𝜌.
• Used in Adaptive Resonance Theory (ART) network.
• Used to control the degree of similarity.
• Ranges from 0.7 to 1 to perform useful work in controlling the number of clusters.

35 / 43
McCulloch Pitts Neuron

• Discovered in 1943.
• Usually called as M-P neuron.
• M-P neurons are connected by directed weighted paths.
• Activation of M-P neurons is binary (i.e) at any time step the neuron may fire or may not
fire.
• Weights associated with communication links may be excitatory(wgts are
positive)/inhibitory(wgts are negative).
• Threshold plays major role here. There is a fixed threshold for each neuron and if the net
input to the neuron is greater than the threshold then the neuron fires.
• They are widely used in logic functions.

36 / 43
• A simple M-P neuron is shown in the figure.
• It is excitatory with weight (w>0) / inhibitory with weight -p (p<0).
• In the Fig., inputs from 𝑥 1 , to 𝑥 𝑛 possess excitatory weighted connection and 𝑥 𝑛+1 to 𝑥 𝑛+𝑚
has inhibitory interconnections.
• Since the firing of neuron is on threshold, based activation function is defined as

37 / 43
Example - 1

For the given Figure calculate the net input

38 / 43
Example - 2

Calculate the net input of the shown network with bias

39 / 43
Example - 3

Obtain the output of the neuron Y for the network shown in Figure using activation function
as: (1) binary sigmoid and (ii) bipolar sigmoidal.

Solution: We know the formula:

1
binary 𝑓 (𝑥) = 1+𝑒−𝜆𝑥
sigmoid
2
bipolar 𝑓 (𝑥) = 1+𝑒−𝜆𝑥
−1
sigmoid
40 / 43
41 / 43
Summary

Basic Models of ANN

• Synaptic interconnection links,
• learning rule - supervised,
unsupervised, reinforcement
• activation function- identity, binary
step function, bipolar step function,
sigmoid function, ramp

42 / 43
Summary

Important terminologies of ANNs

• Weights,
• Bias,
• Threshold,
• Learning rate,
• Momentum factor,
• Vigilance parameter

McCulloch Pitts Neuron Inhibitory, Excitatory

43 / 43