MULTI

- LAYER
PERCEPTRON
 SYLLABUS
UNIT 3

 Multi-Layer Perceptron: Preliminaries, Back-Propagation

Algorithm, XOR problem.

 Textbook #1: 4.1-4.5

 Multi - Layer Perceptron

Architectural graph of Multilayer perceptron with two hidden layers

 Networks consists of input, hidden, and output layers.
 Referred to as Multilayer perceptrons.
 Learning algorithm used is the error backpropagation
algorithm (backpropagation or backprop), which is a
generalization of the LMS rule.
 Multi - Layer Perceptron
 Backpropagation Algorithm
 Forward pass: Activate the network, layer by layer –
Backward pass: Error signal backpropagates from output to
hidden and hidden to input, based on which weights are
updated.

Illustration of the directions of two

basic signal flows in a multilayer
perceptron: forward propagation of
function signals and backward
propagation of error signals.
 Multilayer Perceptrons:
Characteristics
1. Nonlinear Activation Function:
Each model neuron employs a nonlinear activation function,
typically a logistic function:

Signal flow graph highlighting

the details of output neuron k
connected to hidden neuron
 Multilayer Perceptrons:
Characteristics
2. Hidden Layers:
The network includes one or more hidden layers. These are
layers that are neither input nor output layers, providing the
capacity for the network to learn complex representations.

3. High Degree of Connectivity:

The network exhibits a high degree of connectivity, with
neurons in one layer being connected to neurons in the
subsequent layer, facilitating complex interactions and
learning within the network.
 Multi - Layer Networks
 Sigmoid threshold unit
 Differentiable threshold unit:
 Output:
 Other functions:

Signal flow graph highlighting the details

of output neuron k
Multilayer Network & Backpropagation
 Depicts a neural network with multiple layers.
 The input layer consists of two nodes labeled F1 and F2.
 The output layer includes nodes corresponding to different
words like "head," "hid," "...," "who'd," and "hood."
 There are hidden layers
with interconnected nodes
between the input and
output layers, representing
a typical structure of a
feedforward neural network.
Multilayer Network & Backpropagation
Scatter Plot:
 Shows a plot with axes labeled F1 (kHz) and F2 (kHz),
representing formant frequencies commonly used in
speech processing.
 The plot includes various data points and boundary lines,
with labels corresponding to words such as "head," "hid,"
"who'd," "hood," etc.
 The boundary lines likely
represent decision boundaries
formed by the neural network
to classify different words
based on the input features
(F1 and F2).
 Multi - Layer Perceptron
 Nonlinear decision surfaces:

 Error Gradient for a Single Sigmoid Unit

 Multi - Layer Perceptron
 Error Gradient for a Single Sigmoid Unit

 But

 So
Backpropagation Algorithm
 Multi - Layer Perceptron
The δ Term:
 For output unit:

 For hidden unit:

 In sum, δ is the derivative times the error.

Derivation of ∆w:
 To update weight as:

where error is defined as

 Multi - Layer Perceptron
Derivation of ∆w:
 Given

Derivation of ∆w: Output Unit Weights

 Consider
 Multi - Layer Perceptron
Derivation of ∆w: Output Unit Weights
 Thus

 Since

 Consider

 But
 Multi - Layer Perceptron
Derivation of ∆w: Hidden Unit Weights
 Multi - Layer Perceptron
Derivation of ∆w: Hidden Unit Weights

Extension to Different Network Topologies

 Multi - Layer Perceptron
 Arbitrary number of layers: for neurons in layer m:

 Arbitrary acyclic graph:

Backpropagation: Properties
 Gradient descent over entire network weight vector.
 Easily generalized to arbitrary directed graphs.
 Will find a local, not necessarily global error minimum.
 Minimizes error over training examples.
 Training can take thousands of iterations → slow!
 Using the network after training is very fast.
 Multi - Layer Perceptron
 Backpropagation is a fundamental algorithm for training
neural networks, particularly in the context of supervised
learning. Here is a concise summary of its key properties:
 Gradient Descent Over Entire Network Weight Vector:
 Backpropagation employs gradient descent to adjust the
weights of the entire network. The weights are updated to
minimize the error between the predicted and actual
outputs.
 Generalization to Arbitrary Directed Graphs:
 The algorithm can be generalized to arbitrary directed
graphs, making it versatile and applicable to various neural
network architectures, including feedforward networks,
convolutional networks, and recurrent networks.
 Local Error Minimum:
 Backpropagation optimizes the network weights to find a
 Backpropagation: Properties
1. Gradient Descent Over Entire Network Weight Vector:
 Backpropagation employs gradient descent to adjust the
weights of the entire network. The weights are updated to
minimize the error between the predicted and actual
outputs.
2. Generalization to Arbitrary Directed Graphs:
 The algorithm can be generalized to arbitrary directed
graphs, making it versatile and applicable to various neural
network architectures, including feedforward networks,
convolutional networks, and recurrent networks.
3. Local Error Minimum:
 Backpropagation optimizes the network weights to find a
local minimum of the error function. There is no guarantee
that it will find the global minimum, which is the absolute
lowest possible error.
 Backpropagation: Properties
4. Error Minimization Over Training Examples:
 The primary objective of backpropagation is to minimize
the error over a given set of training examples. This
process involves iteratively adjusting the weights to reduce
the overall error.
5. Training Iterations and Speed:
 Training a neural network using backpropagation can be
slow, often requiring thousands of iterations (epochs) to
converge to a satisfactory solution. The computational
complexity increases with the size and depth of the
network.
6. Fast Inference After Training:
 Once the network is trained, using it to make predictions
(inference) is very fast. The computational effort required
for inference is much lower compared to the training phase.
Backpropagation Problem 1
Backpropagation Problem 1
Backpropagation Problem 2
Backpropagation Problem 2

Weight updates of all nodes

Weight updates of bias nodes

Backpropagation Problem 2
 Backpropagation Problem 3
• Optimize the weights for the following Neural Network
Backpropagation Problem 3
Backpropagation Problem 3
Backpropagation Problem 3
Backpropagation Problem 4
Backpropagation Problem 4
Backpropagation Problem 4
Backpropagation Problem 4
Backpropagation Problem 4
Key Features of Backpropagation
1. Gradient Descent with Differentiable Units:
Backpropagation uses gradient descent to minimize the
error function. This method requires that all units (neurons)
in the network have differentiable activation functions,
allowing for the calculation of gradients.
2. Unique Weight Calculation Process:
Calculates the weights by propagating the error backward
through the network. This adjusts the weights based on the
error gradient, effectively learning from the mistakes made
during prediction.
3. Three Stages of Training:
a. Feed-Forward of Input Training Pattern: The
input data is fed forward through the network to
generate an output.
Key Features of Backpropagation
b. Calculation and Backpropagation of Error: The
error is calculated and then propagated backward
through the network to determine how much each
weight contributed to the error.
c. Updation of Weights: The weights are updated to
reduce the error, typically using gradient descent. This
process involves adjusting each weight in proportion to
its contribution to the error.
 Multi - Layer Perceptron
Learning Rate and Momentum
 Tradeoffs regarding learning rate:
i. Smaller learning rate: Smoother trajectory but slower
convergence.
ii. Larger learning rate: Fast convergence, but can become
unstable.
 Momentum can help overcome the issues above.

 The update rule can be written as:

Learning Rate and Momentum
 The learning rate and momentum are crucial
hyperparameters that influence the optimization process.
 Weight Vector as an Exponentially Weighted Time
Series:
The weight vector in neural network training can be viewed as
the sum of an exponentially weighted time series. This
perspective helps understand how the weights change over
time during the training process.
 Behavior Based on Successive Gradients:
1. Successive Gradients with the Same Sign:
 When successive gradients have the same sign (e.g., both
positive or both negative), it indicates consistent movement
in the same direction.
Learning Rate and Momentum
 In this case, the weight update is accelerated, leading to
faster convergence, especially downhill (towards the
minimum of the error function). This acceleration speeds
up the learning process.
2. Successive Gradients with Different Signs:
 If successive gradients have different signs (one positive,
one negative), it suggests oscillation or fluctuation in the
optimization path.
 In such scenarios, the weight update is damped, helping to
stabilize oscillations and prevent large swings in weight
values. This damping effect promotes smoother
convergence and reduces the risk of overshooting the
optimal solution.
 Multi - Layer Perceptron
Sequential (online) mode:
 Update rule applied after each input-target presentation.
 Order of presentation should be randomized.
 Benefits: less storage, stochastic search through weight
space helps avoid local minima.
 Disadvantages: hard to establish theoretical convergence
conditions.

Batch mode:
 Update rule applied after all input-target pairs are seen.
 Benefits: accurate estimate of the gradient, convergence to
local minimum is guaranteed under simpler conditions.
 Multi - Layer Perceptron
Sequential (online) mode Batch mode

The update rule is applied after each input- The update rule is applied after all
target presentation. input-target pairs have been seen.

The order of input presentations should be The order of presentation is not a

randomized to ensure better learning and primary concern since all inputs are
to avoid cycles. processed together.

Requires less storage as it processes one Requires more memory as the entire
example at a time. dataset needs to be loaded into memory
for each update.

The stochastic nature helps in exploring the Convergence to a local minimum is more
weight space, which can help in avoiding theoretically guaranteed and usually
local minima. Harder to establish happens under simpler conditions.
theoretical convergence conditions.

Useful when the dataset is too large to fit Preferred when you need more stable
into memory. and accurate gradient estimates.

When you need a quick and memory- Suitable when you have enough memory
efficient training process. to handle the entire dataset at once.
 Multi - Layer Perceptron
Representational Power of Feedforward Networks
 Boolean functions: Every boolean function representable
with two layers.
 Continous functions: Every bounded continuous function
can be approximated with an arbitrarily small error (output
units are linear).
 Arbitrary functions: With three layers (output units are
linear).
 Multi - Layer Perceptron
What the Hidden Layer Does:
 Hidden layers are the intermediary stages between input and
output in a neural network.
 They are responsible for learning the intricate structures in
data and making neural networks a powerful tool for a wide
range of applications, from image and speech recognition to
natural language processing and beyond.
 Able to capture very complex relationships and achieve
exciting performance in many tasks.
 The output is equal to a linear combination
along with a non-linearity. - The model is similar
to a linear regression model.
 A linear regression attempts to fit a linear equation to the
observed data.
 Multi - Layer Perceptron
 A linear relationship is not enough to capture the
complexity of the task and the linear regression model fails.
 Predict the output of an XOR logical gate.

 The two classes cannot be separated using a single line and

the XOR problem is not linearly separable.
 Multi - Layer Perceptron
 The solution to this problem is to learn a non-linear function
by adding a hidden layer with two neurons to our neural
network.

 So, the hidden layer manages to transform the input features

into processed features that can then be correctly classified
in the output layer.
 Multi - Layer Perceptron
Consider computer vision domain
 Deep neural networks that consist of many hidden layers
have achieved impressive results in face recognition by
learning features in a hierarchical way.
 First hidden layers - detect short pieces of corners and
edges in the image. These features are easy to detect given
the raw image but are not very useful by themselves to
recognize the identity of the person in the image.
 Middle hidden layers - combine the detected edges from the
previous layers to detect parts of the face like the eyes, the
nose and the ears.
 Final layers - combine the detectors of the nose, eyes, etc to
detect the overall face of the person.
 Multi - Layer Perceptron
 Each layer from the raw pixels to final goal:

 Can activate one hidden unit and see what output pattern it
produces.
 Multi - Layer Perceptron
 What the Hidden Layer Does
 A smooth ramped output, monotonically increasing.
 Ramp can be oriented in different angles.
 This kind of visualization is only possible with low-
dimensional input

 What the Hidden Layer Does

 Can look at the hidden layer weight as a pattern or feature.

Learning Hidden Layer Representations
 Each input vector is mapped directly to the same output
vector. Figure demonstrates the ability of a neural network
to maintain and replicate binary patterns through its layers.
Learning Hidden Layer Representations
 Neural network is designed to perform an identity
transformation, where each binary input vector is replicated
exactly at the output layer. This demonstrates the network's
ability to learn and reproduce simple binary patterns.
Learning Hidden Layer Representations
 It refer to the intermediate feature representations that a
neural network learns in its hidden layers during the
training process.
 These representations are not directly interpretable but
capture essential patterns and structures in the input data.
 Feature Extraction: The hidden layers act as feature
extractors. Each layer transforms the input data into a more
abstract and informative representation, making it easier for
subsequent layers to perform the final task, such as
classification or regression.
 Hierarchical Learning: Multiple hidden layers are
stacked, allowing the network to learn hierarchical
representations. Lower layers capture low-level features
while higher layers capture more complex patterns.
Learning Hidden Layer Representations
 Transfer Learning: Learned hidden layer representations
transferred to new, related tasks. A pre-trained model (on a
large dataset) is fine-tuned on a smaller, task-specific
dataset, leveraging the learned representations to improve
performance and reduce training time.
 Dimensionality Reduction: Techniques like autoencoders
explicitly learn compact representations that capture the
most important information in the input.
 Interpretability: Learned representations are powerful,
they are not directly interpretable by humans. Techniques
such as activation maximization, saliency maps, and
attention mechanisms are used to visualize and understand
what these representations capture.
Learning Hidden Layer Representations
 Regularization: Techniques like dropout, weight decay,
and batch normalization are used during training to ensure
that the learned representations are robust and generalize
well to new data. These techniques help prevent overfitting.
 Activation Functions: Non-linear activation functions
(e.g., ReLU, sigmoid, tanh) are applied to the hidden layer
outputs to introduce non-linearity, enabling the network to
learn complex mappings from inputs to outputs.
 Backpropagation: The process of learning these
representations is driven by backpropagation, where
gradients of the loss function are computed with respect to
the weights and biases in the network. These gradients are
then used to update the weights, gradually refining the
hidden representations.
Learning Hidden Layer Representations
 Applications: Learned hidden layer representations are crucial
in various applications, including computer vision, natural
language processing, speech recognition, and reinforcement
learning. They enable neural networks to perform tasks such as
image classification, language translation, and game playing
with high accuracy.
Learned encoding is similar to standard 3-bit binary code: The
encoding produced by the neural network resembles the binary
encoding, it is learned by the model during training.
 Binary Encoding: In digital systems, a 3-bit binary code can
represent 2^3 = 8 distinct values.
 Learned Encoding: Representations created in the hidden
layers. These are not predefined but are adjusted during the
training process to optimize the network’s performance on a
specific task.
Learning Hidden Layer Representations
 Similarity to 3-bit Binary Code:
1. Dimensionality: If the learned encoding is said to be like a
3-bit binary code, it has three dimensions or features.
2. Discreteness: The learned representations might exhibit
patterns where certain features are active or inactive in a
manner like binary codes (e.g., features being close to 0 or 1).
Example in Neural Networks:
 Suppose a neural network is trained to classify 8 different
classes. A hidden layer with 3 neurons could learn to
represent each class with a unique combination of
activations.
 The activations of these 3 neurons might end up resembling
binary codes. One class might activate the neurons in a
pattern like (1, 0, 0), another class in (0, 1, 1), and so on.
Learning Hidden Layer Representations
 Interpretability: This make the learned representations more
interpretable. Each neuron activation can correspond to a
specific feature or characteristic learned by the network.

Automatic discovery of useful hidden layer representations is

a key feature of artificial neural networks (ANNs), and this
capability allows ANNs to excel in a wide range of complex
tasks.
 Layered Architecture:
1. Input Layer: The input layer receives raw data.
2. Hidden Layers: These intermediate layers process the data
and extract features.
3. Output Layer: The output layer produces the final
prediction or classification.
Learning Hidden Layer Representations
 Non-Linear Transformations: Each hidden layer applies a
linear transformation followed by a non-linear activation
function to the input data. This allows the network to model
complex, non-linear relationships in the data.
 Learning Process (Backpropagation):
1. Initialization: Network weights are typically initialized
randomly.
2. Forward Pass: Data passes through the network, with each
layer transforming the input data.
3. Loss Calculation: The output is compared to the ground
truth using a loss function.
4. Backward Pass: Backpropagation computes the gradient of
the loss with respect to each weight in the network.
5. Weight Update: Gradients are used to update the weights.
Learning Hidden Layer Representations
 Feature Extraction:
1. Lower Layers: Learn to capture low-level features such
as edges in images.
2. Intermediate Layers: Combine low-level features into
more complex patterns, like shapes in images.
3. Higher Layers: Capture high-level abstractions, such as
object categories in images.
 Regularization Techniques:
To prevent overfitting and ensure the network generalizes
well to new data, regularization techniques like dropout, L2
regularization, and batch normalization are applied. These
techniques help in learning more robust and generalizable
features.
Learning Hidden Layer Representations
 Automatic Feature Discovery:
The network automatically adjusts its weights to minimize the
loss function. This helps the network discover useful features
without manual intervention.
This is done through iterative adjustments based on the error
feedback from the loss function.
 End-to-End Learning:
The entire process from raw input to final output is optimized
simultaneously. This allows the network to learn features that
are directly relevant to the task
 Scalability and Depth:
Multiple hidden layers enables to learn hierarchical
representations. Each layer builds on the previous layer’s
representations, allowing the network to learn increasingly
abstract and complex features.
Learning Hidden Layer Representations
Summary:
1. Learned Encoding: The learned encoding is compared to
a standard 3-bit binary code, suggesting that the hidden
layer has captured information in a compressed, binary-
like format.
2. Useful Hidden Layer Representations: The automatic
discovery of useful hidden layer representations is
highlighted as a crucial feature of ANNs. This means that
the neural network has effectively identified and utilized
significant patterns or features in the data.
3. Compression: The hidden layer representation is noted to
be compressed, indicating that the network has reduced the
dimensionality of the input data while preserving
important information.
Learning Hidden Layer Representations
• Thus, the image emphasizes the efficiency and capability of
neural networks in learning and representing data in a
compressed, binary-like format, which is a fundamental
characteristic that enhances their performance and utility.
 Multi - Layer Perceptron
Overfitting:
 If the regression algorithm works well with the training
dataset but not well with the test data set.
Underfitting:
 If the regression algorithm does not perform well with the
training dataset.
 Overfitting
Error in Robot Perception Tasks:
 Errors can occur in both the training set and the validation
set in different robot perception tasks.
Early Stopping:
 Early stopping is a technique to ensure good performance
on unseen samples. However, it must be applied carefully
to avoid premature halting of the training process.
 Overfitting
Techniques to Overcome Errors:
 Weight Decay: A regularization technique to prevent
overfitting by adding a penalty for larger weights.
 Validation Sets: Used to tune hyperparameters and
prevent overfitting by monitoring performance on unseen
data.
 K-Fold Cross-Validation: A method to improve the
robustness of the model by training and validating on
different subsets of the data multiple times.
 These techniques collectively help improve the
generalization of models in robot perception tasks by
mitigating errors and preventing overfitting.
Recurrent Neural Networks (RNNs)
 Designed for sequential data.
 They are effective for tasks where the order of data is
crucial, such as time series prediction, natural language
processing (NLP), and speech recognition.

Key Features of RNNs for Sequence Recognition:

1. Sequential Data Processing:
 Capable of processing input sequences of variable length
one element at a time.
 They maintain a hidden state that is updated at each time
step, capturing information from previous elements in the
sequence.
 Recurrent Neural Networks (RNNs)
2. Hidden State Memory:
 The hidden state of an RNN acts as a memory that retains
information from previous time steps.
 This allows the network to understand the context and
dependencies within the sequence.
3. Parameter Sharing:
 The same set of weights is applied at each time step, which
makes RNNs computationally efficient and helps them
generalize well across different parts of the sequence.
4. Backpropagation Through Time (BPTT):
 Training RNNs involves backpropagation through time,
where the gradients are calculated by unrolling the network
through the entire sequence and updating the weights based
on the loss.
 Recurrent Neural Networks (RNNs)
Sequence Recognition Using RNNs
 Sequence Recognition: It involves identifying patterns or
structures within a sequence of data. Well-suited due to their
ability to handle sequential information.
 Key Points:
1. Delay:
 RNNs process sequences element by element, maintaining a
hidden state that captures information over time. This can
introduce a delay as the network updates its state with each new
element.
2. Store Tree Structure:
 When dealing with hierarchical or structured data, such as
sentences or parse trees, RNNs can store and utilize tree structures
to capture more complex dependencies within the sequence.
Which helps in understanding nested or recursive patterns in data.
 Recurrent Neural Networks (RNNs)
3. Training with Plain Backpropagation:
 RNNs can be trained using standard backpropagation techniques.
Specifically, backpropagation through time (BPTT) is used to
handle the temporal dependencies by unrolling the RNN over time
and applying backpropagation to the unfolded network.
 This involves computing gradients for each time step and updating
the weights accordingly.
4. Generalization:
 Ability to generalize may not be perfect, especially when dealing
with very long sequences or complex dependencies. Issues such as
overfitting, vanishing gradients, and the need for large amounts of
data can impact their performance.
 Advanced variants like Long Short-Term Memory (LSTM) and
Gated Recurrent Unit (GRU) networks are used to improve
generalization by better handling long-term dependencies.
 Recurrent Neural Networks (RNNs)
 RNNs are a foundational tool for sequence recognition,
capable of processing and learning from sequential data.
 Despite their limitations in generalization, especially in
standard forms, they remain effective when augmented
with advanced architectures and training techniques.
 Recurrent Neural Networks (RNNs)
Recurrent Neural Networks (RNNs) are
 Autoassociation:
1. Definition: This refers to a scenario where the input is
equal to the output. The network is designed to
reproduce its input at the output layer.
2. Purpose: Helps the network learn to recognize patterns
and maintain consistency in representations.
 Accuracy Depends on Numerical Precision:
1. Numerical Precision: The precision of the numerical
computations affects the accuracy of the network. In
tasks involving recurrent computations and deep
structures, small numerical errors can accumulate,
impacting the overall performance.
 Recurrent Neural Networks (RNNs)
Components: Input, stack, and delay
 The input and stack are processed, with the output being fed
back into the network after a delay, indicating a time step.
Components: Elements A and B, and their combination
(A, B)
 Elements A and B are combined into a single representation
(A, B), demonstrating how the network can integrate
multiple inputs into a consolidated hidden state.
Components: Elements C, and the combination (A, B)
 Element C and the previously combined elements (A, B)
are processed to form a new combined state (C, A, B),
showing the recursive nature of stack representation in the
hidden layer.
 Recurrent Neural Networks (RNNs)
 Recurrent Networks, with their ability to autoassociate,
represent stacks, and maintain sequence information through
hidden states, are powerful tools for sequence recognition
and processing hierarchical data.
 Their performance, however, is sensitive to numerical
precision, highlighting the importance of careful numerical
management in practical implementations.
 Recurrent Neural Networks (RNNs)
 NETtalk: Sejnowski and Rosenberg (1987): The application
was developed by Sejnowski and Rosenberg in 1987.
 Purpose: NETtalk is designed to learn how to pronounce
English text.
 The diagram illustrates a neural network model with:
 Input layer: Represented by the text "This is the input."
 Hidden units: Intermediate processing units.
 Output units: Represent phoneme codes.
 Recurrent Neural Networks (RNNs)
NETtalk Data:
 Format: Each entry in the data consists of a word, its
pronunciation, and stress/syllable information.
Recurrent Neural Networks (RNNs)
 Recurrent Neural Networks (RNNs)
Applications: Data Compression
 Concept: Construct an autoassociative memory where the
input is equal to the output.
 Training: Train the network with a small hidden layer.
 Encoding: Use input-to-hidden weights to encode the data.
 Transmission/Storage: Send or store the activation of the
hidden layer.
 Decoding: Decode the received or stored hidden layer
activation using the hidden-to-output weights.
 The diagrams illustrate:
 The structure of an autoassociative network with input,
hidden, and output layers.
 The encoding and decoding process where the input signal is
compressed into the hidden layer and then reconstructed.
 Recurrent Neural Networks (RNNs)
Backpropagation: Example Results
 Graph: Displays error reduction over 10,000 epochs for
different logic functions (OR, AND, XOR).
 Y-axis: Error
 X-axis: Number of epochs (up to 40)
 Recurrent Neural Networks (RNNs)
Backpropagation: Example Results
 Observations:
 OR function was the easiest to learn.
 AND function was next in difficulty.
 XOR function was the most difficult to learn.
 Training Details:
 An epoch refers to one full cycle of training through all
training input patterns.
 The network used had 2 input units, 2 hidden units, and
1 output unit.
 The learning rate was set to 0.001.
 Recurrent Neural Networks (RNNs)
Backpropagation: Example Results

 Each grayscale image corresponds to the neural network's output for

the OR, AND, and XOR functions.
 The rows in the grayscale images represent the network's output for
the input pairs (0,0), (0,1), (1,0), and (1,1).
 The color intensity in the images shows the output value, with white
indicating a value closer to 1 and black indicating a value closer to 0.
 Successful application of backpropagation in training neural networks
to approximate basic logic functions, evidenced by the reduction in
error and the appropriate grayscale output patterns for each function.
 Recurrent Neural Networks (RNNs)
Impact of different parameters and settings on the
training process and performance of neural networks
using backpropagation.
1. Increasing the Number of Hidden Layer Units:
 Time of Training: Increasing the number of hidden units
typically increases the computational complexity, leading
to longer training times per epoch.
 Number of Epochs: With more hidden units, the network
might learn more complex patterns, potentially requiring
more epochs to converge, but this can also vary depending
on the problem and network architecture.
 Recurrent Neural Networks (RNNs)
2. Adjusting the Learning Rate:
Rate of Convergence:
 Increasing Learning Rate: May speed up convergence
initially but can cause instability and overshooting, leading to
possible divergence.
 Decreasing Learning Rate: May slow down convergence
but can provide more stable and accurate learning, avoiding
overshooting.
3. Changing the Slope of the Sigmoid:
 Rate of Convergence: The slope of the sigmoid function
affects the gradient values:
 Steeper Slope: Can lead to faster learning but may also
cause gradient saturation, where gradients become very small
and slow down learning (vanishing gradient problem).
 Recurrent Neural Networks (RNNs)
Backpropagation: Example Results
 Shallower Slope: Can slow down learning due to smaller
gradient values but provides more stability in training.
4. Different Problem Domains:
 Applying backpropagation to various domains like
handwriting recognition demonstrates the versatility of
neural networks. Each domain may require different network
architectures, preprocessing techniques, and parameter
tuning to achieve optimal performance.

 Summary: Experimenting with these factors helps in

understanding and optimizing the training process of neural
networks for different tasks, highlighting the trade-offs
between training time, convergence rate, and stability.
MLP as a General Function Approximator
MLP as a General Function Approximator
 Nonlinear Mapping: MLPs are capable of capturing
complex, nonlinear relationships between input and output.
 Function Approximation: MLPs can approximate any
continuous function on a compact subset of , given
sufficient network complexity.
 Parameters: The theorem establishes the existence of
appropriate weights and biases that enable this
approximation to any desired degree of accuracy.

This highlights the power of MLPs in modeling a wide variety

of functions, making them versatile tools in machine learning.
MLP as a General Function Approximator
Existence Theorem:
 The Universal Approximation Theorem is an existence
theorem, meaning it guarantees that under certain conditions,
a neural network can approximate any continuous function to
any desired degree of accuracy.
 It generalizes approximations similar to those done by finite
Fourier series.

Applicability to Neural Networks:

 The theorem is directly applicable to MLPs, indicating that
these networks can approximate any continuous function.
 It implies that a single hidden layer is theoretically sufficient
to achieve this approximation.
MLP as a General Function Approximator
Limitations of the Theorem:
 The theorem does not claim that a single hidden layer is
optimal regarding learning time, generalization ability, or
other practical considerations.
 Deeper networks (more hidden layers) or other architectures
might be more efficient, faster to train, or better at
generalizing from the training data to unseen data.

Universal Approximation Theorem provides a theoretical

foundation for the capability of neural networks, it does not
address practical aspects such as the efficiency or optimal
structure of the network for various tasks.
 Generalization

 Generalization: A network generalizes well when it

accurately maps input to output for test data not seen during
training.
 Two graphs depict input-output mappings.
 The left graph shows a simpler, smoother mapping,
indicating good generalization.
 The right graph shows a more complex, overfitted mapping,
indicating poor generalization.
 Generalization
Curve-Fitting View:
 This perspective is helpful for understanding generalization.

Issues: Overfitting or Overtraining:

 Overfitting occurs when a network memorizes training data,
leading to poor performance on new, unseen data.
 Smoothness in the mapping is desired to avoid overfitting,
aligning with principles like Occam's razor, which favors
simpler models that generalize better.
 Thus achieving a model that generalizes well without
overfitting, ensuring good performance on both training and
unseen data.
 Generalization
Generalization and Training Set Size:
Factors Influencing Generalization:
 Size of the Training Set: Larger and more representative
training sets generally improve generalization because they
provide more comprehensive information about the problem
domain.
 Architecture of the Network: The design of the network,
including the number of layers and units, affects its ability to
learn and generalize. A well-chosen architecture can help
capture the underlying patterns without overfitting.
 Physical Complexity of the Problem: More complex
problems may require more sophisticated networks and
larger training sets to achieve good generalization.
 Generalization
Sample Complexity and VC Dimension:
 Sample Complexity: Refers to the number of training
samples required for the network to generalize well.
 VC Dimension: A measure of the capacity of a model,
representing its ability to fit a variety of functions.
 Practical Rule of Thumb: The sample complexity is
expressed as
where N is the number of training samples, W is the total
number of free parameters (weights) in the network, and ϵ is
the error tolerance.
 This indicates that the number of required samples grows
with the number of parameters and desired accuracy.
 Training Set Size and Curse of
Dimensionality
 Figure show how the number of required inputs increases
exponentially with dimensionality to maintain the same
sampling density
 This indicates that the number of required samples grows
with the number of parameters and desired accuracy.
Training Set Size and Curse of Dimensionality
 Exponential Growth in Inputs: As the dimensionality of
the input grows, exponentially more inputs are needed to
maintain the same density in unit space.
 Sampling Density: The sampling density of N inputs in
m-dimensional space is proportional to
 Overcoming the Curse: Use prior knowledge about the
function, which can help reduce the dimensionality or
improve the efficiency of the training process.
Thus Higher input dimensionality significantly increases the
required size of the training set to ensure adequate sampling
density, a phenomenon known as the "curse of
dimensionality." Utilizing prior knowledge about the problem
can help mitigate these challenges.
 Cross-Validation

 Use a validation set (which is not used during training) for

measuring the generalizability of the model.
Components and Methods:
Validation Set:
 Used for model selection and early stopping.
 Helps in measuring how well the model generalizes to new,
unseen data.
Limitations of Backpropagation
Connectionism:
 Biological Metaphor: Neural networks are inspired by the
brain's structure, emphasizing local computation, parallelism,
and graceful degradation (the system's ability to maintain
functionality even when parts are damaged).
 Limitations: While backpropagation is effective, it has
limitations in biological plausibility, meaning it doesn't
perfectly mimic how biological brains learn.

 Feature Detection:
 Hidden Neurons: Hidden neurons in neural networks
perform feature detection by identifying and responding to
specific patterns in the input data.
Limitations of Backpropagation
Function Approximation:
 Nested Sigmoid: Neural networks can approximate complex
functions by using layers of nested sigmoid (or similar)
activation functions.

Computational Complexity:
 Efficiency: The computation involved in training neural
networks is polynomial in the number of adjustable
parameters, making it computationally efficient.

Sensitivity Analysis:
 Efficiency: The sensitivity of the network’s output to its
parameters can be estimated efficiently, aiding in
understanding and improving model performance.
Limitations of Backpropagation
Robustness:
 Disturbances: Neural networks are robust, meaning they can
handle small disturbances or noise in the input without
significant errors in the output.

Convergence:
 Stochastic Approximation: Training involves stochastic
approximation, where the network parameters are updated
based on random samples of data.
 Speed: Convergence can be slow, especially for complex
networks or challenging problems.
Limitations of Backpropagation
Local Minima and Scaling Issues:
 Local Minima: Neural networks can get stuck in local
minima during training, where the solution is suboptimal.
 Scaling Issues: Large networks or datasets can present
scaling challenges, requiring careful design and optimization
to manage effectively.

 Thus, neural networks, inspired by biological systems,

perform feature detection and function approximation
efficiently. They are robust and can handle disturbances,
though training can be slow and prone to issues like local
minima. Sensitivity analysis helps improve understanding
and optimization, and scalability remains a key
consideration.
Heuristic for Accelerating Convergence
 Optimizing neural networks or other machine learning
models, a heuristic approach can be employed to adapt the
learning rate dynamically, which accelerates the convergence
of the training process.
 The learning rate adaptation involves adjusting the learning
rate for each tunable weight based on the behavior of the cost
function's derivative.
 Separate Learning Rate for Each Tunable Weight:
Each weight in the model has its own learning rate, allowing for
fine-grained control over the optimization process.
 Dynamic Adjustment of Learning Rates:
The learning rate for each weight is adjusted after every
iteration, responding to the observed changes in the cost
function's derivative.
Heuristic for Accelerating Convergence
 Increasing the Learning Rate:
If the derivative of the cost function with respect to a weight has
maintained the same sign over several iterations (indicating
consistent progress in the same direction), the learning rate for
that weight is increased. This helps in accelerating the
convergence when the direction of optimization is stable.
 Decreasing the Learning Rate:
If the derivative of the cost function alternates in sign over several
iterations (indicating oscillations or potential overshooting), the
learning rate for that weight is decreased. This helps in stabilizing
the convergence when the optimization direction is unstable.
 This heuristic approach enables a more responsive and
efficient optimization process by adapting to the characteristics
of the cost function's landscape during training.