What is Data Science?

Data science is basically a field in which information and knowledge are

extracted from the data by using various scientific methods, algorithms, and
processes. It can thus be defined as a combination of various mathematical
tools, algorithms, statistics, and machine learning techniques which are thus
used to find the hidden patterns and insights from the data which help in the
decision-making process. Data science deals with both structured as well as
unstructured data. It is related to both data mining and big data. Data
science involves studying the historic trends and thus using its conclusions
to redefine present trends and also predict future trends. Technologies

Data science involves these key steps:

 Data Collection: Gathering raw data from various sources, such as
databases, sensors, or user interactions.
 Data Cleaning: Ensuring the data is accurate, complete, and ready for
analysis.
 Data Analysis: Applying statistical and computational methods to identify
patterns, trends, or relationships.
 Data Visualization: Creating charts, graphs, and dashboards to present
findings clearly.
 Decision-Making: Using insights to inform strategies, create solutions, or
predict outcomes.

Steps for Data Science Processes:

Step 1: Define the Problem and Create a Project Charter
Clearly defining the research goals is the first step in the Data Science
Process. A project charter outlines the objectives, resources, deliverables,
and timeline, ensuring that all stakeholders are aligned.
Step 2: Retrieve Data
Data can be stored in databases, data warehouses, or data lakes within an
organization. Accessing this data often involves navigating company policies
and requesting permissions.
Step 3: Data Cleansing, Integration, and Transformation
Data cleaning ensures that errors, inconsistencies, and outliers are
removed. Data integration combines datasets from different sources,
while data transformation prepares the data for modeling by reshaping
variables or creating new features.
Step 4: Exploratory Data Analysis (EDA)
During EDA, various graphical techniques like scatter plots, histograms,
and box plots are used to visualize data and identify trends. This phase
helps in selecting the right modeling techniques.
Step 5: Build Models
In this step, machine learning or deep learning models are built to make
predictions or classifications based on the data. The choice of algorithm
depends on the complexity of the problem and the type of data.
Step 6: Present Findings and Deploy Models
Once the analysis is complete, results are presented to stakeholders. Models
are deployed into production systems to automate decision-making or
support ongoing analysis.

What is Business Intelligence?

Business intelligence(BI) is a set of technologies, applications, and
processes that are used by enterprises for business data analysis. It is used
for the conversion of raw data into meaningful information which is thus used
for business decision-making and profitable actions. It deals with the analysis
of structured and sometimes unstructured data which paves the way for new
and profitable business opportunities. It supports decision-making based on
facts rather than assumption-based decision-making. Thus it has a direct
impact on the business decisions of an enterprise. Business intelligence
tools enhance the chances of an enterprise to enter a new market as well as
help in studying the impact of marketing efforts.

Below is a table of differences between Data Science and Business

Intelligence:
S.
No. Factor Data Science Business Intelligence

It is basically a set of
It is a field that uses technologies, applications
mathematics, statistics and and processes that are
1. Concept
various other tools to discover used by the enterprises
the hidden patterns in the data. for business data
analysis.
S.
No. Factor Data Science Business Intelligence

It focuses on the past and

2. Focus It focuses on the future.
present.

It deals with both structured as It mainly deals only with

3. Data
well as unstructured data. structured data.

It is less flexible as in
Data science is much more
case of business
4. Flexibility flexible as data sources can be
intelligence data sources
added as per requirement.
need to be pre-planned.

It makes use of the scientific It makes use of the

5. Method
method. analytic method.

It has a higher complexity in It is much simpler when

6. Complexity comparison to business compared to data
intelligence. science.

It’s expertise is the

7. Expertise It’s expertise is data scientist.
business user.

It deals with the questions of It deals with the question

8. Questions
what will happen and what if. of what happened.

The data to be used is

Data warehouse is
9. Storage disseminated in real-time
utilized to hold data.
clusters.

The ETL (Extract-

The ELT (Extract-Load-
Transform-Load) process
Transform) process is
Integration is generally used for the
10. generally used for the
of data integration of data for
integration of data for data
business intelligence
science applications.
applications.

11. Tools It’s tools are SAS, BigML, It’s tools are
S.
No. Factor Data Science Business Intelligence

InsightSquared Sales
Analytics, Klipfolio,
MATLAB, Excel, etc.
ThoughtSpot, Cyfe,
TIBCO Spotfire, etc.

Companies can harness their Business Intelligence

potential by anticipating the helps in performing root
12. Usage future scenario using data cause analysis on a
science in order to reduce risk failure or to understand
and increase income. the current status.

Business Intelligence has

Greater business value is lesser business value as
achieved with data science in the extraction process of
Business
13. comparison to business business value carries
Value
intelligence as it anticipates out statically by plotting
future events. charts and KPIs (Key
Performance Indicator).

The technologies such as

The sufficient tools and
Hadoop are available and
Handling technologies are not
14. others are evolving for handling
data sets available for handling
understandingItsItsarge data
large data sets.
sets.

Why Is Data Science Important?

In a world flooded with user-data, data science is crucial for driving progress
and innovation in every industry. Here are some key reasons why it is so
important:
 Helps Business in Decision-Making: By analyzing data, businesses can
understand trends and make informed choices that reduce risks and
maximize profits.
 Improves Efficiency: Organizations can use data science to identify
areas where they can save time and resources.
 Personalizes Experiences: Data science helps create customized
recommendations and offers that improve customer satisfaction.
 Predicts the Future: Businesses can use data to forecast trends,
demand, and other important factors.
 Drives Innovation: New ideas and products often come from insights
discovered through data science.
 Benefits Society: Data science improves public services like healthcare,
education, and transportation by helping allocate resources more
effectively.
Real Life Example of Data Science
There are lot of examples you can observe around yourself, where data
science is being used. For Example – Social Media, Medical, Preparing
strategy for Cricket or FIFA by analyzing past matches. Here are some more
real life examples:
Social Media Recommendation:
Have you ever wondered why you always get Instagram Reels aligned
towards your interest? These platforms uses data-science to Analyze your
past interest/data (Like, Comments, watch etc) and create personalized
recommendation to serve content that matches your interests.
Early Diagnosis of Disease:
Data Science can predicts the risk of conditions like diabetes or heart
disease, by analyzing a patient’s medical records and lifestyle habits. This
allows doctors to act early and improve lives. In Future, it can help doctors
detect diseases before symptoms even start to appear. For example,
predicting a Tumor or Cancer at a very early stage. Data Science uses
medical history and Image-data for such prediction.
E-commerce recommendation and Demand Forecast:
E-commerce platforms like Amazon or Flipkart use data science to enhance
the shopping experience. By analyzing your browsing history, purchase
behavior, and search patterns, they recommend products based on your
preferences. It can also help in predicting demand for products by studying
past sales trends, seasonal patterns etc.
Applications of Data Science
Data science has a wide range of applications across various industries, by
transforming how they operate and deliver results. Here are some examples:
 Data science is used to analyze patient data, predict diseases, develop
personalized treatments, and optimize hospital operations.
 It helps detect fraudulent transactions, manage risks, and provide
personalized financial advice.
 Businesses use data science to understand customer behavior,
recommend products, optimize inventory, and improve supply chains.
 Data science powers innovations like search engines, virtual assistants,
and recommendation systems.
 It enables route optimization, traffic management, and predictive
maintenance for vehicles.
 Data science helps in designing personalized learning experiences,
tracking student performance, and improving administrative efficiency.
 Streaming platforms and content creators use data science to recommend
shows, analyze viewer preferences, and optimize content delivery.
 Companies leverage data science to segment audiences, predict
campaign outcomes, and personalize advertisements.
Industry where data science is used
Data science is transforming every industry by unlocking the power of data.
Here are some key sectors where data science plays a vital role:
 Healthcare: Data science improves patient outcomes by using predictive
analytics to detect diseases early, creating personalized treatment plans
and optimizing hospital operations for efficiency.
 Finance: Data science helps detect fraudulent activities, assess and
manage financial risks, and provide tailored financial solutions to
customers.
 Retail: Data science enhances customer experiences by delivering
targeted marketing campaigns, optimizing inventory management, and
forecasting sales trends accurately.
 Technology: Data science powers cutting-edge AI applications such as
voice assistants, intelligent search engines, and smart home devices.
 Transportation: Data science optimizes travel routes, manages vehicle
fleets effectively, and enhances traffic management systems for smoother
journeys.
 Manufacturing: Data science predicts potential equipment failures,
streamlines supply chain processes, and improves production efficiency
through data-driven decisions.
 Energy: Data science forecasts energy demand, optimizes energy
consumption, and facilitates the integration of renewable energy
resources.
 Agriculture: Data science drives precision farming practices by
monitoring crop health, managing resources efficiently, and boosting
agricultural yields.
Data Scientist Roles and Responsibilities
A data scientist is a tech professional that collects, analyzes, and interprets vast
amounts of data using analytical, statistical, and programming skills.

They are responsible for mining valuable information from various sources and
transforming it into actionable insights that can drive business growth.

The specific roles and responsibilities of a data scientist may differ based on the
organization and industry. However, there are some common tasks that most data
scientists are expected to perform. Let’s take a closer look at these roles and
responsibilities:

1. Data Mining and Extraction

One of the primary tasks of a data scientist is to extract valuable data from various
sources. This includes collecting structured and unstructured data from databases,
websites, APIs, social media platforms, and other relevant sources.

They must have a deep understanding of data collection techniques and tools to ensure
the data is accurate and reliable.

2. Data Cleaning and Preprocessing

Data scientists usually spend a large amount of time cleaning and preprocessing data.
This involves removing duplicate records, handling missing values, dealing with outliers,
and transforming the data into a suitable format for analysis.

Data cleaning is crucial to ensure the accuracy and integrity of the data.

3. Exploratory Data Analysis

Before diving into complex modeling and analysis, data scientists perform exploratory
data analysis (EDA) to gain insights into the data. They use statistical techniques
and data visualization tools to identify patterns, trends, and relationships within the data.

EDA helps them understand the data and uncover potential opportunities or challenges.
4. Machine Learning and Model Development
Machine learning is a core component of a data scientist’s work. They develop and train
machine learning models to solve specific business problems. This involves selecting
appropriate algorithms, preprocessing the data, and evaluating the performance of the
models.

Data scientists continuously iterate and refine their models to improve accuracy and
predictive power.

5. Data Visualization and Reporting

Data scientists need to be able to effectively communicate their findings to both
technical as well as non-technical stakeholders. They use data visualization tools to
create charts, graphs, and dashboards that convey complex information in a clear and
concise manner.

Visualization helps stakeholders understand the insights derived from the data and
make informed decisions.

6. Collaboration and Communication

Data scientists collaborate with various teams within the organization, including
business, IT, and executive teams. They work closely with domain experts to
understand business requirements and translate them into data-driven solutions.

Effective communication skills are essential to convey technical concepts to non-

technical stakeholders and build strong working relationships.

7. Continuous Learning and Professional Development

The field of data science is rapidly evolving, and data scientists must stay updated with
the latest tools, techniques, and industry trends. They actively engage in continuous
learning through online courses, workshops, conferences, and networking events.

They need to be passionate about expanding their knowledge and skills to deliver
innovative solutions.
 Data Processing and Visualization: Data Formatting
Data formatting involves converting raw data into a structured format that
aligns with specific requirements or standards. This process includes defining
data types, setting display formats, and applying consistent styles to ensure
data is readable and usable across various platforms and systems. In financial
contexts, data formatting might involve standardizing numerical values, dates,
and textual information to maintain consistency in reports and analyses.

What Are Data Formatting

Techniques?
Several techniques are employed to format data effectively:

 Data Standardization: Transforming data into a common format to ensure

uniformity across different datasets.
 Data Cleansing: Identifying and correcting errors or inconsistencies in the
data to improve quality.
 Data Transformation: Converting data from one format or structure to
another to make it compatible with target systems.
 Conditional Formatting: Applying specific formatting to data that meets
certain conditions, enhancing data analysis.

What are the Types of Data

Formatting?
Data formatting can be categorized based on the type of data:

 Numerical Formatting: Setting the display of numbers, including decimal

places, currency symbols, and thousand separators.
 Textual Formatting: Standardizing text data, such as names and addresses,
to ensure consistency.
 Date and Time Formatting: Ensuring dates and times are presented in a
consistent format, crucial for financial transactions and reporting.
EDA
Exploratory Data Analysis (EDA) is a crucial initial step in data science
projects. It involves analyzing and visualizing data to understand its key
characteristics, uncover patterns, and identify relationships between
variables refers to the method of studying and exploring record sets to
apprehend their predominant traits, discover patterns, locate outliers, and
identify relationships between variables. EDA is normally carried out as a
preliminary step before undertaking extra formal statistical analyses or
modeling.
Key aspects of EDA include:
 Distribution of Data: Examining the distribution of data points to
understand their range, central tendencies (mean, median), and
dispersion (variance, standard deviation).
 Graphical Representations: Utilizing charts such as histograms, box
plots, scatter plots, and bar charts to visualize relationships within the
data and distributions of variables.
 Outlier Detection: Identifying unusual values that deviate from other data
points. Outliers can influence statistical analyses and might indicate data
entry errors or unique cases.
 Correlation Analysis: Checking the relationships between variables to
understand how they might affect each other. This includes computing
correlation coefficients and creating correlation matrices.
 Handling Missing Values: Detecting and deciding how to address
missing data points, whether by imputation or removal, depending on their
impact and the amount of missing data.
 Summary Statistics: Calculating key statistics that provide insight into
data trends and nuances.
 Testing Assumptions: Many statistical tests and models assume the
data meet certain conditions (like normality or homoscedasticity). EDA
helps verify these assumptions.
Why Exploratory Data Analysis is Important?
Exploratory Data Analysis (EDA) is important for several reasons, especially
in the context of data science and statistical modeling. Here are some of the
key reasons why EDA is a critical step in the data analysis process:
1. Understanding Data Structures: EDA helps in getting familiar with the
dataset, understanding the number of features, the type of data in each
feature, and the distribution of data points. This understanding is crucial
for selecting appropriate analysis or prediction techniques.
2. Identifying Patterns and Relationships: Through visualizations and
statistical summaries, EDA can reveal hidden patterns and intrinsic
 relationships between variables. These insights can guide further analysis
and enable more effective feature engineering and model building.
3. Detecting Anomalies and Outliers: EDA is essential for identifying
errors or unusual data points that may adversely affect the results of your
analysis. Detecting these early can prevent costly mistakes in predictive
modeling and analysis.
4. Testing Assumptions: Many statistical models assume that data follow a
certain distribution or that variables are independent. EDA involves
checking these assumptions. If the assumptions do not hold, the
conclusions drawn from the model could be invalid.
5. Informing Feature Selection and Engineering: Insights gained from
EDA can inform which features are most relevant to include in a model
and how to transform them (scaling, encoding) to improve model
performance.
6. Optimizing Model Design: By understanding the data’s characteristics,
analysts can choose appropriate modeling techniques, decide on the
complexity of the model, and better tune model parameters.
7. Facilitating Data Cleaning: EDA helps in spotting missing values and
errors in the data, which are critical to address before further analysis to
improve data quality and integrity.
8. Enhancing Communication: Visual and statistical summaries from EDA
can make it easier to communicate findings and convince others of the
validity of your conclusions, particularly when explaining data-driven
insights to stakeholders without technical backgrounds.

Types of Exploratory Data Analysis

EDA, or Exploratory Data Analysis, refers back to the method of analyzing
and analyzing information units to uncover styles, pick out relationships, and
gain insights. There are various sorts of EDA strategies that can be hired
relying on the nature of the records and the desires of the evaluation.
Depending on the number of columns we are analyzing we can divide EDA
into three types: Univariate, bivariate and multivariate .
1. Univariate Analysis
Univariate analysis focuses on a single variable to understand its internal
structure. It is primarily concerned with describing the data and finding
patterns existing in a single feature. This sort of evaluation makes a
speciality of analyzing character variables inside the records set. It involves
summarizing and visualizing a unmarried variable at a time to understand its
distribution, relevant tendency, unfold, and different applicable records.
Common techniques include:
 Histograms: Used to visualize the distribution of a variable.
 Box plots: Useful for detecting outliers and understanding the spread and
skewness of the data.
 Bar charts: Employed for categorical data to show the frequency of each
category.
 Summary statistics: Calculations like mean, median, mode, variance,
and standard deviation that describe the central tendency and dispersion
of the data.
2. Bivariate Analysis
Bivariate evaluation involves exploring the connection between variables. It
enables find associations, correlations, and dependencies between pairs of
variables. Bivariate analysis is a crucial form of exploratory data analysis that
examines the relationship between two variables. Some key techniques used
in bivariate analysis:
 Scatter Plots: These are one of the most common tools used in bivariate
analysis. A scatter plot helps visualize the relationship between two
continuous variables.
 Correlation Coefficient: This statistical measure (often Pearson’s
correlation coefficient for linear relationships) quantifies the degree to
which two variables are related.
 Cross-tabulation: Also known as contingency tables, cross-tabulation is
used to analyze the relationship between two categorical variables. It
shows the frequency distribution of categories of one variable in rows and
the other in columns, which helps in understanding the relationship
between the two variables.
 Line Graphs: In the context of time series data, line graphs can be used
to compare two variables over time. This helps in identifying trends,
cycles, or patterns that emerge in the interaction of the variables over the
specified period.
 Covariance: Covariance is a measure used to determine how much two
random variables change together. However, it is sensitive to the scale of
the variables, so it’s often supplemented by the correlation coefficient for
a more standardized assessment of the relationship.
3. Multivariate Analysis
Multivariate analysis examines the relationships between two or more
variables in the dataset. It aims to understand how variables interact with
one another, which is crucial for most statistical modeling techniques.
Techniques include:
 Pair plots: Visualize relationships across several variables
simultaneously to capture a comprehensive view of potential interactions.
 Principal Component Analysis (PCA): A dimensionality reduction
technique used to reduce the dimensionality of large datasets, while
preserving as much variance as possible.
Specialized EDA Techniques
In addition to univariate and multivariate analysis, there are specialized EDA
techniques tailored for specific types of data or analysis needs:
 Spatial Analysis: For geographical data, using maps and spatial plotting
to understand the geographical distribution of variables.
 Text Analysis: Involves techniques like word clouds, frequency
distributions, and sentiment analysis to explore text data.
 Time Series Analysis : This type of analysis is mainly applied to statistics
sets that have a temporal component. Time collection evaluation entails
inspecting and modeling styles, traits, and seasonality inside the statistics
through the years. Techniques like line plots, autocorrelation analysis,
transferring averages, and ARIMA (AutoRegressive Integrated Moving
Average) fashions are generally utilized in time series analysis.
Tools for Performing Exploratory Data Analysis
Exploratory Data Analysis (EDA) can be effectively performed using a variety
of tools and software, each offering unique features suitable for handling
different types of data and analysis requirements.
1. Python Libraries
 Pandas: Provides extensive functions for data manipulation and analysis,
including data structure handling and time series functionality.
 Matplotlib: A plotting library for creating static, interactive, and animated
visualizations in Python.
 Seaborn: Built on top of Matplotlib, it provides a high-level interface for
drawing attractive and informative statistical graphics.
 Plotly: An interactive graphing library for making interactive plots and
offers more sophisticated visualization capabilities.
2. R Packages
 ggplot2: Part of the tidyverse, it’s a powerful tool for making complex
plots from data in a data frame.
 dplyr: A grammar of data manipulation, providing a consistent set of
verbs that help you solve the most common data manipulation challenges.
 tidyr: Helps to tidy your data. Tidying your data means storing it in a
consistent form that matches the semantics of the dataset with the way it
is stored.

Steps for Performing Exploratory Data Analysis

Performing Exploratory Data Analysis (EDA) involves a series of steps
designed to help you understand the data you’re working with, uncover
underlying patterns, identify anomalies, test hypotheses, and ensure the data
is clean and suitable for further analysis.

Step 1: Understand the Problem and the Data

The first step in any information evaluation project is to sincerely apprehend
the trouble you are trying to resolve and the statistics you have at your
disposal. This entails asking questions consisting of:
 What is the commercial enterprise goal or research question you are
trying to address?
 What are the variables inside the information, and what do they mean?
 What are the data sorts (numerical, categorical, textual content, etc.) ?
 Is there any known information on first-class troubles or obstacles?
 Are there any relevant area-unique issues or constraints?
By thoroughly knowing the problem and the information, you can better
formulate your evaluation technique and avoid making incorrect assumptions
or drawing misguided conclusions. It is also vital to contain situations and
remember specialists or stakeholders to this degree to ensure you have
complete know-how of the context and requirements.
Step 2: Import and Inspect the Data
Once you have clean expertise of the problem and the information, the
following step is to import the data into your evaluation environment (e.g.,
Python, R, or a spreadsheet program). During this step, looking into the
statistics is critical to gain initial know-how of its structure, variable kinds, and
capability issues.
Here are a few obligations you could carry out at this stage:
 Load the facts into your analysis environment, ensuring that the facts are
imported efficiently and without errors or truncations.
 Examine the size of the facts (variety of rows and columns) to experience
its length and complexity.
 Check for missing values and their distribution across variables, as
missing information can notably affect the quality and reliability of your
evaluation.
 Identify facts sorts and formats for each variable, as these records may
be necessary for the following facts manipulation and evaluation steps.
 Look for any apparent errors or inconsistencies in the information, such
as invalid values, mismatched units, or outliers, that can indicate
exceptional issues with information.
Step 3: Handle Missing Data
Missing records is a joint project in many datasets, and it can significantly
impact the quality and reliability of your evaluation. During the EDA method,
it’s critical to pick out and deal with lacking information as it should be, as
ignoring or mishandling lacking data can result in biased or misleading
outcomes.
Here are some techniques you could use to handle missing statistics:
 Understand the styles and capacity reasons for missing statistics: Is
the information lacking entirely at random (MCAR), lacking at random
(MAR), or lacking not at random (MNAR)? Understanding the underlying
mechanisms can inform the proper method for handling missing
information.
 Decide whether to eliminate observations with lacking values
(listwise deletion) or attribute (fill in) missing values: Removing
observations with missing values can result in a loss of statistics and
potentially biased outcomes, specifically if the lacking statistics are not
MCAR. Imputing missing values can assist in preserving treasured facts.
However, the imputation approach needs to be chosen cautiously.
 Use suitable imputation strategies, such as mean/median imputation,
regression imputation, a couple of imputations, or device-getting-to-know-
 based imputation methods like k-nearest associates (KNN) or selection
trees. The preference for the imputation technique has to be primarily
based on the characteristics of the information and the assumptions
underlying every method.
 Consider the effect of lacking information: Even after imputation,
lacking facts can introduce uncertainty and bias. It is important to
acknowledge those limitations and interpret your outcomes with warning.
Handling missing information nicely can improve the accuracy and reliability
of your evaluation and save you biased or deceptive conclusions. It is
likewise vital to record the techniques used to address missing facts and the
motive in the back of your selections.
Step 4: Explore Data Characteristics
After addressing the facts that are lacking, the next step within the EDA
technique is to explore the traits of your statistics. This entails examining
your variables’ distribution, crucial tendency, and variability and identifying
any ability outliers or anomalies. Understanding the characteristics of your
information is critical in deciding on appropriate analytical techniques,
figuring out capability information first-rate troubles, and gaining insights that
may tell subsequent evaluation and modeling decisions.
Calculate summary facts (suggest, median, mode, preferred deviation,
skewness, kurtosis, and many others.) for numerical variables: These facts
provide a concise assessment of the distribution and critical tendency of
each variable, aiding in the identification of ability issues or deviations from
expected patterns.
Step 5: Perform Data Transformation
Data transformation is a critical step within the EDA process because it
enables you to prepare your statistics for similar evaluation and modeling.
Depending on the traits of your information and the necessities of your
analysis, you may need to carry out various ameliorations to ensure that your
records are in the most appropriate layout.
Here are a few common records transformation strategies:
 Scaling or normalizing numerical variables to a standard variety
(e.g., min-max scaling, standardization )
 Encoding categorical variables to be used in machine mastering fashions
(e.g., one-warm encoding, label encoding)
 Applying mathematical differences to numerical variables (e.g.,
logarithmic, square root) to correct for skewness or non-linearity
 Creating derived variables or capabilities primarily based on current
variables (e.g., calculating ratios, combining variables)
 Aggregating or grouping records mainly based on unique variables or
situations
By accurately transforming your information, you could ensure that your
evaluation and modeling strategies are implemented successfully and that
your results are reliable and meaningful.
Step 6: Visualize Data Relationships
Visualization is an effective tool in the EDA manner, as it allows to discover
relationships between variables and become aware of styles or trends that
may not immediately be apparent from summary statistics or numerical
outputs. To visualize data relationships, explore univariate, bivariate, and
multivariate analysis.
 Create frequency tables, bar plots, and pie charts for express variables:
These visualizations can help you apprehend the distribution of classes
and discover any ability imbalances or unusual patterns.
 Generate histograms, container plots, violin plots, and density plots to
visualize the distribution of numerical variables. These visualizations can
screen critical information about the form, unfold, and ability outliers within
the statistics.
 Examine the correlation or association among variables using scatter
plots, correlation matrices, or statistical assessments like Pearson’s
correlation coefficient or Spearman’s rank correlation: Understanding the
relationships between variables can tell characteristic choice,
dimensionality discount, and modeling choices.
Step 7: Handling Outliers
An Outlier is a data item/object that deviates significantly from the rest of the
(so-called normal)objects. They can be caused by measurement or
execution errors. The analysis for outlier detection is referred to as outlier
mining. There are many ways to detect outliers, and the removal process of
these outliers from the dataframe is the same as removing a data item from
the panda’s dataframe.
Identify and inspect capability outliers through the usage of strategies like
the interquartile range (IQR), Z-scores, or area-specific regulations: Outliers
can considerably impact the results of statistical analyses and gadget
studying fashions, so it’s essential to perceive and take care of them as it
should be.
Step 8: Communicate Findings and Insights
The final step in the EDA technique is effectively discussing your findings
and insights. This includes summarizing your evaluation, highlighting
fundamental discoveries, and imparting your outcomes cleanly and
compellingly.
Here are a few hints for effective verbal exchange:
 Clearly state the targets and scope of your analysis
 Provide context and heritage data to assist others in apprehending your
approach
 Use visualizations and photos to guide your findings and make them more
reachable
 Highlight critical insights, patterns, or anomalies discovered for the
duration of the EDA manner
 Discuss any barriers or caveats related to your analysis
 Suggest ability next steps or areas for additional investigation
Effective conversation is critical for ensuring that your EDA efforts have a
meaningful impact and that your insights are understood and acted upon
with the aid of stakeholders.

Data Filtering:
Data filtering is the process of narrowing down the most relevant information from a
large dataset using specific conditions or criteria. It makes the analysis more
focused and efficient.

Types of Data Filtering Techniques

Data filtering techniques can help you quickly access the data you need.

Basic Filtering Methods

Basic filtering involves simple techniques like range or set membership. For
example, in a database of temperatures recorded throughout a year,
a range filter could be used to select all records where the temperature was
between 20°C and 30°C. Similarly, a set membership filter could select
records for specific months, like June, July, and August.

Filtering by Criteria

Filtering by criteria involves more advanced filtering based on multiple

criteria or conditions. For instance, an e-commerce company might filter
customer data to target a marketing campaign. They could use multiple
criteria, such as customers who have purchased over $100 in the last month,
are in the 25-35 age range, and have previously bought electronic products.
Filtering by Time Range

Temporal filters work by selecting data within a specific time frame. A

financial analyst might use a time range filter to analyze stock market
trends by filtering transaction data to include only those that occurred in the
last quarter. This helps focus on recent market behaviors and predict future
trends.

Text Filtering

Text filtering includes techniques for filtering textual data, such as pattern
matching. For example, a social media platform might filter posts containing
specific keywords or phrases to monitor content related to a specific event or
topic. Using pattern matching, they can filter all posts with the hashtag
#EarthDay.

Numeric Filtering

Numeric filtering involves methods for filtering numerical data based on

value thresholds. A healthcare database might be filtered to identify patients
with high blood pressure by setting a numeric filter to include all records
where the systolic pressure is above 140 mmHg and the diastolic pressure is
above 90 mmHg.

Custom Filtering

Custom filtering refers to user-defined filters for specialized needs. A

biologist studying a species’ population growth might create a custom filter
to include data points that match a complex set of conditions, such as
specific genetic markers, habitat types, and observed behaviors, to study the
factors influencing population changes.

These techniques can be applied to extract meaningful information from

large datasets, aiding in analysis and decision-making processes.

Data Filtering Tools and Software

Data filtering can be performed via manual scripting or no-code solutions.
Here’s an overview of these methods:

Filtering Data Manually

Manual data filtering often involves writing custom scripts in programming

languages such as R or Python. These languages provide powerful libraries
and functions for data manipulation.

Example: In Python, the pandas library is commonly used for data analysis
tasks. A data scientist might write a script using pandas to filter a dataset of
customer feedback, selecting only entries that contain certain keywords
related to a product feature of interest. The script could look something like
this:

Python

import pandas as pd

# Load the dataset

df = pd.read_csv(‘customer_feedback.csv’)

# Define the keywords of interest

keywords = [‘battery life’, ‘screen’, ‘camera’]

# Filter the dataset for feedback containing the keywords

filtered_df = df[df[‘feedback’].[Link](‘|’.join(keywords))]

Using No-Code Data Filtering Software

No-code data filtering software allows you to filter data through a graphical
user interface (GUI) without writing code. These tools are designed to be
user-friendly and accessible to people with little programming experience.
With Regular Expressions capabilities, you have the flexibility to write
custom filter expressions.
How to use Hierarchical Indexes with Pandas ?
The index is like an address, that’s how any data point across the data frame or series
can be accessed. Rows and columns both have indexes, rows indices are called index
and for columns, it's general column names. Hierarchical Indexes are also known as
multi-indexing is setting more than one column name as the index.
# using the pandas columns attribute.
col = [Link]
print(col)
Output:
Index(['Unnamed: 0', 'region', 'state', 'individuals', 'family_members',
'state_pop'], dtype='object')

To make the column an index, we use the Set_index() function of pandas. If

we want to make one column an index, we can simply pass the name of the
column as a string in set_index(). If we want to do multi-indexing or
Hierarchical Indexing, we pass the list of column names in the set_index().
Below Code demonstrates Hierarchical Indexing in pandas:

# using the pandas set_index() function.

df_ind3 = df.set_index(['region', 'state', 'individuals'])

# we can sort the data by using sort_index()

df_ind3.sort_index()

print(df_ind3.head(10))
Output:
Now the dataframe is using Hierarchical Indexing or multi-indexing.
Note that here we have made 3 columns as an index ('region', 'state',
'individuals' ). The first index 'region' is called level(0) index, which is on top
of the Hierarchy of indexes, next index 'state' is level(1) index which is below
the main or level(0) index, and so on. So, the Hierarchy of indexes is formed
that's why this is called Hierarchical indexing.

Understanding Data Visualization

Data visualization translates complex data sets into visual formats that are
easier for the human brain to understand. This can include a variety of visual
tools such as:
 Charts: Bar charts, line charts, pie charts, etc.
 Graphs: Scatter plots, histograms, etc.
 Maps: Geographic maps, heat maps, etc.
 Dashboards: Interactive platforms that combine multiple visualizations.
The primary goal of data visualization is to make data more accessible and
easier to interpret allow users to identify patterns, trends, and outliers
quickly. This is particularly important in big data where the large volume of
information can be confusing without effective visualization techniques.
Why is Data Visualization Important?
Let’s take an example. Suppose you compile data of the company’s profits
from 2013 to 2023 and create a line chart. It would be very easy to see the
line going constantly up with a drop in just 2018. So you can observe in a
second that the company has had continuous profits in all the years except a
loss in 2018.
It would not be that easy to get this information so fast from a data table.
This is just one demonstration of the usefulness of data visualization. Let’s
see some more reasons why visualization of data is so important.
Importance of Data Visualization

1. Data Visualization Simplifies the Complex Data

Large and complex data sets can be challenging to understand. Data
visualization helps break down complex information into simpler, visual
formats making it easier for the audience to grasp. For example in a scenario
where sales data is visualized using a heat map on Tableau states that have
suffered a net loss are colored red. This visual makes it instantly obvious
which states are underperforming.

2. Enhances Data Interpretation

Visualization highlights patterns, trends, and correlations in data that might
be missed in raw data form. This enhanced interpretation helps in making
informed decisions. Consider another Tableau visualization that
demonstrates the relationship between sales and profit. It might show that
higher sales do not necessarily equate to higher profits this trend that could
be difficult to find from raw data alone. This perspective helps businesses
adjust strategies to focus on profitability rather than just sales volume.
3. Data Visualization Saves Time
It is definitely faster to gather some insights from the data using data
visualization rather than just studying a chart. In the screenshot below on
Tableau it is very easy to identify the states that have suffered a net loss
rather than a profit. This is because all the cells with a loss are coloured red
using a heat map, so it is obvious states have suffered a loss. Compare this
to a normal table where you would need to check each cell to see if it has a
negative value to determine a loss. Visualizing Data can save a lot of time in
this situation.
4. Improves Communication
Visual representations of data make it easier to share findings with others
especially those who may not have a technical background. This is important
in business where stakeholders need to understand data-driven insights
quickly. Let see the below TreeMap visualization on Tableau showing the
number of sales in each region of the United States with the largest
rectangle representing California due to its high sales volume. This visual
context is much easier to grasp rather than detailed table of numbers.
5. Data Visualization Tells a Data Story
Data visualization is also a medium to tell a data story to the viewers. The
visualization can be used to present the data facts in an easy-to-understand
form while telling a story and leading the viewers to an inevitable conclusion.
This data story should have a good beginning, a basic plot, and an ending
that it is leading towards. For example, if a data analyst has to craft a data
visualization for company executives detailing the profits of various products
then the data story can start with the profits and losses of multiple products
and move on to recommendations on how to tackle the losses.

Data visualization using Seaborn

Getting Started with Seaborn
Seaborn is a Python data visualization library that simplifies the process of
creating complex visualizations. It is specifically designed for statistical
data visualization, making it easier to understand data distributions
and relationships between variables. Seaborn integrates closely with
Pandas data structures, allowing seamless data manipulation and
visualization.
It is built on the top of matplotlib library and also closely integrated into the
data structures from pandas.
Key Features of Seaborn:
 High-level interface: Simplifies the creation of complex visualizations.
 Integration with Pandas: Works seamlessly with Pandas DataFrames
for data manipulation.
 Built-in themes: Offers attractive default themes and color palettes.
 Statistical plots: Provides various plot types to visualize statistical
relationships and distributions.
Installing Seaborn for Data Visualization
Before using Seaborn, you’ll need to install it. The easiest way to install
Seaborn is using pip, the Python package manager for python environment:
pip install seaborn

Creating Basic Plots with Seaborn

Before starting let’s have a small intro of bivariate and univariate data:
 Bivariate data: This type of data involves two different variables. The
analysis of this type of data deals with causes and relationships and the
analysis is done to find out the relationship between the two variables.
 Univariate data: This type of data consists of only one variable. The
analysis of univariate data is thus the simplest form of analysis since the
information deals with only one quantity that changes. It does not deal
with causes or relationships and the main purpose of the analysis is to
describe the data and find patterns that exist within it.
Seaborn offers a variety of plot types to visualize different aspects of
data. Seaborn helps to visualize the statistical relationships, to
understand how variables in a dataset are related to one another and
how that relationship is dependent on other variables, we perform
statistical analysis. This Statistical analysis helps to visualize the trends
and identify various patterns in the dataset. Below are some common plots
you can create using Seaborn:
1. Line plot
Lineplot is the most popular plot to draw a relationship between x and y with
the possibility of several semantic groupings. It is often used to track
changes over intervals.
Syntax : [Link](x=None, y=None)
Parameters:
x, y: Input data variables; must be numeric. Can pass data directly or
reference columns in data.
Let’s visualize the data with a line plot and pandas:
Python

import pandas as pd
import seaborn as sns

# initialise data of lists

data = {'Name':[ 'Mohe' , 'Karnal' , 'Yrik' , 'jack' ],
 'Age':[ 30 , 21 , 29 , 28 ]}
df = [Link]( data )

# plotting lineplot
[Link]( data['Age'], data['Weight'])
Output:

2. Scatter Plot
Scatter plots are used to visualize the relationship between two numerical
variables. They help identify correlations or patterns. It can draw a two-
dimensional graph.
Syntax: [Link](x=None, y=None)

Parameters:
x, y: Input data variables that should be numeric.
Returns: This method returns the Axes object with the plot drawn onto it.
Let’s visualize the data with a scatter plot and pandas:
Python

import pandas as pd
import seaborn as sns

# initialise data of lists

data = {'Name':[ 'Mohe' , 'Karnal' , 'Yrik' , 'jack' ],
'Age':[ 30 , 21 , 29 , 28 ]}
df = [Link]( data )

[Link](data['Age'],data['Weight'])
Output: