DEEP LEARNING

Unit I
Introduction to Neural Network
Introduction to Neural Networks
Neural networks form the foundation of deep learning, a subfield of
machine learning that focuses on models with multiple layers capable of
learning hierarchical representations of data. Deep learning aims to
replicate the human brain's ability to process information, recognize
patterns, and make decisions. Neural networks achieve this by
leveraging large datasets, computational power, and sophisticated
architectures.
What is a Neural Network?
A neural network is a computational model inspired by the structure and
function of biological neural networks. It consists of interconnected layers
of nodes (neurons) that process data and extract features. Neural
networks learn by adjusting their parameters (weights and biases) to
minimize errors in predictions or outputs.
Why Deep Learning?
Deep learning excels in handling unstructured data,
such as images, audio, and text, where traditional
machine learning approaches struggle. Neural
networks in deep learning are designed to
automatically extract features from raw data,
eliminating the need for manual feature engineering.
Key Concepts in Deep Learning Neural Networks:
• Deep Architectures:
• Neural networks in deep learning often have many layers (hence "deep").
• These layers learn hierarchical representations, with lower layers capturing simple
features (e.g., edges in an image) and higher layers capturing more abstract features (e.g.,
shapes or objects).
• Forward Propagation:
• The process of passing data through the network from input to output.
• Each layer transforms the data and passes it to the next layer.
• Loss Function:
• Quantifies the difference between predicted and actual values.
• Examples: Mean Squared Error (MSE) for regression and Cross-Entropy Loss for
classification.
• Backpropagation and Optimization:
• Backpropagation computes gradients of the loss function with respect to weights and
biases.
• Optimization algorithms like Stochastic Gradient Descent (SGD) or Adam adjust these
parameters to minimize loss.
Types of Neural Networks in Deep Learning:
• Convolutional Neural Networks (CNNs):
• Used for image-related tasks.
• Employ convolutional layers to detect spatial patterns.
• Recurrent Neural Networks (RNNs):
• Process sequential data like time series and text.
• Variants include LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent
Units).
• Transformer Networks:
• Revolutionized natural language processing (e.g., GPT, BERT).
• Use self-attention mechanisms to process sequences efficiently.
• Autoencoders:
• Used for unsupervised learning and data compression.
• Learn to encode data into a compressed representation and decode it back.
• Generative Adversarial Networks (GANs):
• Generate realistic synthetic data.
• Involve two networks: a generator and a discriminator.
Applications of Neural Networks in Deep Learning:
• Computer Vision:
• Image classification, object detection, and medical imaging.
• Natural Language Processing (NLP):
• Machine translation, sentiment analysis, and text summarization.
• Speech and Audio Processing:
• Voice recognition, music generation, and audio analysis.
• Autonomous Systems:
• Self-driving cars and robotics.
• Healthcare:
• Disease diagnosis, drug discovery, and personalized treatment plans.
• Gaming:
• AI for game-playing and procedural content generation.
Advantages of Neural Networks in Deep Learning:
⮚Feature Extraction: Automatically learns features from raw
data.
⮚Scalability: Handles large datasets and complex problems.
⮚Accuracy: Achieves state-of-the-art results in many domains.
Neural networks in deep learning represent a paradigm shift in
artificial intelligence, enabling machines to solve problems
previously considered unsolvable. By understanding their
structure and operation, we can leverage their power for
innovation across diverse fields.
Neural Networks and types

Neural Networks:
• With the advancements of artificial intelligence and machine
learning, neural networks are becoming more widely
discussed thanks to their role in deep learning.
• Neural networks learn continuously and, as a result, can
improve over time, making intelligent decisions based on the
insights identified within the data. Many industries benefit
from using neural networks with applications, including
medical diagnostics, energy demand forecasting, targeted
marketing, and financial prediction.
An introduction to artificial intelligence, machine learning, and deep
learning:

Before exploring neural networks, it’s important to understand artificial

intelligence, machine learning, and deep learning and how they are
related.
• Artificial intelligence describes the process of computers being trained to
mimic the human brain in how it learns and solves problems. Computers
can do this through different types of learning: machine learning and
deep learning.
• The term “artificial intelligence” can be traced back to 1956 when
computer scientist John McCarthy coined it. However, in 1950, British
mathematician and computer scientist Alan Turing discussed the concept
of machines being able to think in a groundbreaking paper that played a
significant role in the development of artificial intelligence [1].
An introduction to artificial intelligence, machine
learning, and deep learning:

• Machine learning is a series of algorithms, each taking in

information, analysing it, and using that insight to make an
informed decision. As machine learning algorithms are given
more data, they can become increasingly intelligent and
make better, more informed decisions.
• Deep learning is a subset of machine learning. Neural
networks play a role in deep learning, as they allow data to
be processed without a human pre-determining the program.
Instead, neural networks communicate data with one another
similarly to how the brain functions, creating a more
autonomous process.
Overview of neural networks

• The basic structure of a neural network consists of three main

components: the input layer, the hidden layer, and the output layer. A
neural network can have one or multiple input, hidden, or output layers
depending on complexity.
• Information is received in the input layer, and the input node processes
it,decides how to categorise it, and transfers it to the next layer: the
hidden layer.
• Information is received from both the input layer and other hidden
layers in the hidden layer. There are various hidden layers based on
the type of neural network being used. At this point in the process,
hidden layers take the input, process the information from the previous
layer, and then move it on to the next layer, either another hidden layer
or the output layer.
• The output layer is the final layer in a neural network. After receiving
the data from the hidden layer (or layers), the output layer processes it
and produces the output value.
Types of Neural Networks
8 types of neural networks:
Various neural networks exist, each with a unique structure and function. This list will
discuss eight commonly used neural networks in today’s technology.
1. Convolutional neural networks:
Convolutional neural networks (CNNs) can input images, identify the objects in a
picture, and differentiate them from one another. Their real-world applications
include pattern recognition, image recognition, and object detection. A CNN’s
structure consists of three main layers. First is the convolutional layer, where most of
the computation occurs. Second is the pooling layer, where the number of parameters
in the input is reduced. Lastly, the fully connected layer classifies the features
extracted from the previous layers.

2. Recurrent neural networks:

Recurrent neural networks (RNNs) can translate language, speech recognition,
natural language processing, and image captioning. Examples of products using
RNNs include smart home technologies and voice command features on mobile
phones. Feedback loops in the structure of RNNs allow information to be stored
similarly to how your memory works.
Types of Neural Networks

3. Radial basis functions networks:

Radial basis function (RBF) networks differ from other neural networks because the input
layer performs no computations. Instead, it passes the data directly to the hidden layer.
As a result, RBFs have a faster learning speed. Applications of RBF networks include time
series prediction and function approximation.
4. Long short-term memory networks:
Long short-term memory (LSTM) networks are unique and can sort data into short-term
and long-term memory cells depending on whether or not the data needs to be looped
back into the network as data points or entire sequences. LSTM can also be used in
handwriting recognition and video-to-text conversion.
5. Multilayer perceptrons:
Multilayer perceptrons (MLPs) are a neural network capable of learning the relationship
between linear and non-linear data. Through backpropagation, MLPs can reduce error
rates. Applications that benefit from MLPs include face recognition and computer vision.
Types of Neural Networks
6. Generative adversarial networks:
Generative adversarial networks (GANs) can generate new data sets that
share the same statistics as the training set and often pass as actual data.
An example of this you’ve likely seen is art created with AI. GANs can
replicate popular art forms based on patterns in the training set, creating
pieces often indistinguishable from human artwork.
7. Deep belief networks:
Deep belief networks (DBNs) are unique because they stack individual
networks that can use each other's hidden network layers as the input for
the next layer. This allows for the neural networks to be trained faster.
They are used to generate images and motion-capture data.
8. Self-organising maps:
Self-organising maps (SOMs), or Kohonen maps, can transform extensive
complex data sets into understandable two-dimensional maps where
geometric relationships can be visualised. This can happen because
SOMs use competitive learning algorithms in which neurons must
compete to be represented in the output. This is decided by which
neurons best represent the input. Practical applications of SOMs include
displaying voting trends for analysis and organising complex data
collected by astronomers so it can be interpreted.
Gradient Descent
Gradient Descent is a fundamental optimization algorithm widely used in deep learning
to minimize the error (or cost) of a model by updating its parameters (weights and
biases). It works by iteratively adjusting the parameters in the opposite direction of the
gradient of the cost function with respect to the parameters. This adjustment aims to find
the optimal set of parameters that minimize the cost function.
Key Concepts of Gradient Descent:
⮚Cost Function:
• Measures the error between the predicted output and the actual target.
• Examples include Mean Squared Error (MSE) for regression and Cross-Entropy
Loss for classification.
⮚Gradient:
• The derivative of the cost function with respect to the model parameters.
• Indicates the direction and magnitude of the steepest ascent. To minimize the cost,
we move in the opposite direction.
⮚Learning Rate (η\etaη):
• A hyperparameter that controls the size of the steps taken toward the minimum.
• If too large, it might overshoot the minimum; if too small, convergence can be slow.
Gradient Descent
Types of Gradient Descent:
⮚Batch Gradient Descent:
• Uses the entire training dataset to compute the gradient.
• Pros: Stable convergence.
• Cons: Computationally expensive for large datasets.
⮚Stochastic Gradient Descent (SGD):
• Uses a single training example to compute the gradient for each
update.
• Pros: Faster updates, good for online learning.
• Cons: Noisy updates can make convergence less stable.
⮚Mini-Batch Gradient Descent:
• Uses a subset (mini-batch) of the training data to compute the
gradient.
• Pros: Balances efficiency and stability.
• Cons: Requires tuning of batch size.
Batch Gradient Descent
Batch Gradient Descent entails computation (involved in each step of gradient descent) over the
entire training set at each step and hence it is highly slow on very big training sets. As a result,
Batch Gradient Descent becomes extremely computationally expensive. This is ideal for error
manifolds that are convex or somewhat smooth. Batch Gradient Descent also scales nicely as the
number of features grows. Batch gradient descent has several advantages and disadvantages,
including:
Advantages:
⮚ Convergence stability: Batch gradient descent provides a smoother path to convergence,
which can help with reliable model training.
⮚ Global minimum: Batch gradient descent can converge to the global minimum cost function
under certain conditions.
⮚ Large datasets: Batch gradient descent can work well on large datasets.
⮚ Convex functions: Batch gradient descent is excellent for convex functions.
⮚ Easy implementation: Batch gradient descent is easy to implement.
Disadvantages:
⮚ Local minima: Batch gradient descent can be difficult to escape local minima.
⮚ Larger datasets: Batch gradient descent can be impractical for larger datasets.
⮚ Slow convergence: Batch gradient descent can be slow to converge.
⮚ Noisy: Batch gradient descent can be noisy, which can lead to less stability.
Stochastic Gradient Descent
Stochastic Gradient Descent (SGD) is a variant of the Gradient Descent
algorithm used for optimizing machine learning models. In this variant,
only one random training example is used to calculate the gradient and
update the parameters at each iteration. Here are some of the advantages
and disadvantages of using SGD:
Advantages of Stochastic Gradient Descent:
⮚Speed: SGD is faster than other variants of Gradient Descent such as
Batch Gradient Descent and Mini-Batch Gradient Descent since it uses
only one example to update the parameters.
⮚Memory Efficiency: Since SGD updates the parameters for each
training example one at a time, it is memory-efficient and can handle
large datasets that cannot fit into memory.
⮚Avoidance of Local Minima: Due to the noisy updates in SGD, it has
the ability to escape from local minima and converges to a global
minimum.
Stochastic Gradient Descent
Disadvantages of Gradient Descent:
⮚Noisy updates: The updates in SGD are noisy and have a high
variance, which can make the optimization process less stable and lead
to oscillations around the minimum.
⮚Slow Convergence: SGD may require more iterations to converge to
the minimum since it updates the parameters for each training example
one at a time.
⮚Sensitivity to Learning Rate: The choice of learning rate can be
critical in SGD since using a high learning rate can cause the algorithm
to overshoot the minimum, while a low learning rate can make the
algorithm converge slowly.
⮚Less Accurate: Due to the noisy updates, SGD may not converge to the
exact global minimum and can result in a suboptimal solution. This can
be mitigated by using techniques such as learning rate scheduling and
momentum-based updates.
Mini-Batch Gradient Descent
Mini-batch gradient descent is an optimization method that updates neural network
weights using a subset of training data. It has several advantages, including faster
convergence and memory efficiency, but it also has some disadvantages.
Advantages:
⮚Faster convergence:
• Mini-batch gradient descent updates parameters more frequently than batch
gradient descent, which can lead to faster convergence.
⮚Memory efficiency:
• Mini-batches allow training on large datasets without loading the entire dataset
into memory at once.
⮚Smoother gradient estimation:
• Mini-batch gradient descent offers a smoother path toward convergence than
stochastic gradient descent.
⮚Efficient parallel processing:
• Mini-batch gradient descent can be efficiently processed in parallel.
Mini-Batch Gradient Descent
Disadvantages:
⮚Learning rate and batch size selection:
• The choice of learning rate and mini-batch size can impact the performance of mini-
batch gradient descent.
⮚Additional hyperparameter tuning:
• Mini-batch gradient descent requires additional hyperparameter tuning.
⮚Increased complexity:
• Mini-batch gradient descent can be more complex to implement than other methods.
⮚May not converge to a global minimum:
• If the batch size is not well-tuned, mini-batch gradient descent may not converge to a
global minimum.
• Mini-batch gradient descent is a widely used method for training deep learning
models.
Gradient Descent
Variants of Gradient Descent:
⮚Momentum:
Adds a fraction of the previous update to the current update to
accelerate convergence and smooth oscillations.
⮚Adagrad:
Adapts the learning rate for each parameter based on past
gradients, making larger updates for infrequently updated
parameters.
⮚RMSProp:
Uses a moving average of squared gradients to normalize updates,
preventing large oscillations.
⮚Adam (Adaptive Moment Estimation):
Combines Momentum and RMSProp to adapt learning rates for each
parameter, resulting in faster and more stable convergence.
Gradient Descent
Gradient Descent in Deep Learning:
1. Backpropagation:
• Gradient descent is used alongside backpropagation to calculate
gradients of the cost function with respect to each parameter in the
network.
• Gradients are computed using the chain rule across multiple layers.
2. Challenges:
• Vanishing/Exploding Gradients: Gradients can become very
small or very large in deep networks, making learning difficult.
• Local Minima and Saddle Points: May trap the optimization
process.
3. Solutions:
• Proper initialization techniques (e.g., Xavier, He initialization).
• Advanced optimizers (e.g., Adam).
• Batch normalization to stabilize training.
Gradient Descent
Practical Tips:
⮚ Learning Rate Tuning:
• Use a learning rate schedule (e.g., reduce learning rate as
training progresses).
• Use learning rate warm-up in the initial stages.
⮚ Regularization:
• Techniques like dropout and L2 regularization help prevent
overfitting.
⮚ Early Stopping:
• Monitor validation loss to terminate training when the model
starts to overfit.
Gradient Descent remains a cornerstone of deep learning, driving
the optimization of neural networks efficiently when paired with
computational advancements and algorithmic innovations.
Sentiment Analysis
Sentiment Analysis is a significant application of Natural Language Processing (NLP) that involves determining the
sentiment or emotion expressed in a piece of text. It is widely used in domains like social media monitoring, customer
feedback analysis, and product reviews. Deep learning has revolutionized sentiment analysis by enabling models to
capture complex patterns and nuances in text data. Below is an overview of sentiment analysis in the context of deep
learning:
1. Overview of Sentiment Analysis:
Definition: Sentiment analysis identifies the sentiment of a text (positive, negative, neutral, or more granular emotional
categories like joy, anger, sadness, etc.).
Applications:
• Customer feedback analysis
• Brand monitoring
• Movie/product review classification
• Political sentiment tracking
2. Role of Deep Learning in Sentiment Analysis:
• Deep learning has significantly improved sentiment analysis by providing models capable of understanding context,
semantics, and syntactic structure.
⮚ Why Deep Learning?
• Traditional machine learning models (e.g., SVM, Naïve Bayes) rely heavily on manual feature extraction and may fail to
capture complex dependencies.
• Deep learning models like Recurrent Neural Networks (RNNs) and Transformers automatically learn feature
representations, improving performance.
Sentiment Analysis
• 3. Deep Learning Models for Sentiment Analysis:
• a. Feedforward Neural Networks (FNNs):
• Suitable for simple sentiment analysis tasks.
• Requires manually engineered features like bag-of-words or TF-IDF.
• b. Recurrent Neural Networks (RNNs):
• Captures sequential information in text.
• Popular variants include:
• LSTM (Long Short-Term Memory): Overcomes vanishing gradient problems in traditional RNNs.
• GRU (Gated Recurrent Units): Similar to LSTMs but computationally less expensive.
• c. Convolutional Neural Networks (CNNs):
• Extracts local features from text (e.g., phrases).
• Often combined with word embeddings like Word2Vec or GloVe for better representations.
• d. Hybrid Models:
• Combines CNNs and RNNs to leverage both local features (CNNs) and sequential dependencies (RNNs).
• e. Transformers:
• Dominates modern sentiment analysis.
• BERT (Bidirectional Encoder Representations from Transformers):
• Captures bidirectional context in text.
• Fine-tuned for specific sentiment analysis tasks.
• GPT (Generative Pre-trained Transformer):
• Excels in generating human-like text but can also be fine-tuned for sentiment analysis.
• Other Models: RoBERTa, DistilBERT, and ALBERT
Sentiment Analysis
4. Preprocessing Steps:
• Text Cleaning: Remove unnecessary characters, HTML tags, etc.
• Tokenization: Split text into words, subwords, or characters.
• Stop Word Removal: Optional step depending on the model.
• Word Embedding: Convert tokens into dense vector representations (e.g., Word2Vec, GloVe,
FastText, or Transformer embeddings).
5. Datasets for Sentiment Analysis:
• IMDB Movie Reviews: Contains positive and negative movie reviews.
• Twitter Sentiment Analysis Dataset: Focuses on tweets with sentiments.
• Amazon Reviews: Annotated reviews for various products.
• Stanford Sentiment Treebank (SST): Includes fine-grained sentiment labels for phrases.
6. Challenges:
• Context Understanding: Sarcasm and idiomatic expressions are hard to detect.
• Domain Adaptation: Models trained on one dataset may not generalize well to another
domain.
• Multilingual Sentiment Analysis: Requires cross-lingual capabilities.
• Imbalanced Data: Sentiment datasets often have skewed distributions.
Sentiment Analysis
7. Tools and Libraries:
• TensorFlow and Keras: Frameworks for building deep learning models.
• PyTorch: Another popular framework with extensive NLP support.
• Hugging Face Transformers: Pre-trained transformer models like BERT, GPT, and RoBERTa.
• NLTK and SpaCy: Useful for preprocessing.
8. Evaluation Metrics:
• Accuracy: Proportion of correct predictions.
• Precision, Recall, F1-Score: Especially important for imbalanced datasets.
• ROC-AUC: For binary sentiment classification.
9. Future Trends:
• Zero-shot Learning: Models like GPT-3 and ChatGPT can classify sentiments without task-
specific fine-tuning.
• Multimodal Sentiment Analysis: Combines text with images or audio for better sentiment
understanding.
• Explainability: Focus on making model predictions interpretable.