VII Semester

.IN
Course: Internet of Things
Course Code: BCS701

C
N
Credits: 04 & 2022 Scheme

SY
U
Module 5:Data Analytics for IoT
VT Soumya Jolad
[Link]@[Link]
Assistant Professor, Dept. of CSE, EPCET

Studied smart, not hard — thanks to [Link]

Outline

.IN
• Overview of Hadoop ecosystem
• MapReduce architecture

C
• MapReduce job execution flow

N
SY
• MapReduce schedulers

U
VT

Studied smart, not hard — thanks to [Link]

Hadoop Ecosystem

• Apache Hadoop is an open source framework for distributed batch processing of big data.

.IN
• Hadoop Ecosystem includes:
• Hadoop MapReduce

C
• HDFS

N
• YARN
• HBase

SY
• Zookeeper
• Pig
• Hive

U
• Mahout
• Chukwa VT
• Cassandra
• Avro
• Oozie
• Flume
• Sqoop

Studied smart, not hard — thanks to [Link]

Apache Hadoop

.IN
• A Hadoop cluster comprises of a Master node, backup node and a number
of slave nodes.
• The master node runs the NameNode and JobTracker processes and the
slave nodes run the DataNode and TaskTracker components of Hadoop.

C
• The backup node runs the Secondary NameNode process.

N
• NameNode
•

SY
NameNode keeps the directory tree of all files in the file system, and tracks
where across the cluster the file data is kept. It does not store the data of these
files itself. Client applications talk to the NameNode whenever they wish to
locate a file, or when they want to add/copy/move/delete a file.
• Secondary NameNode

U
• NameNode is a Single Point of Failure for the HDFS Cluster. An optional
Secondary NameNode which is hosted on a separate machine creates
VT
checkpoints of the namespace.
• JobTracker
• The JobTracker is the service within Hadoop that distributes MapReduce tasks to
specific nodes in the cluster, ideally the nodes that have the data, or at least are
in the same rack.

Studied smart, not hard — thanks to [Link]

Apache Hadoop

• TaskTracker

.IN
• TaskTracker is a node in a Hadoop cluster that accepts Map, Reduce
and Shuffie tasks from the JobTracker.
• Each TaskTracker has a defined number of slots which indicate the

C
number of tasks that it can accept.

N
• DataNode

SY
• A DataNode stores data in an HDFS file system.
• A functional HDFS filesystem has more than one DataNode, with data
replicated across them.

U
• DataNodes respond to requests from the NameNode for filesystem
operations. VT
• Client applications can talk directly to a DataNode, once the
NameNode has provided the location of the data.
• Similarly, MapReduce operations assigned to TaskTracker instances
near a DataNode, talk directly to the DataNode to access the files.
• TaskTracker instances can be deployed on the same servers that host
DataNode instances, so that MapReduce operations are performed
close to the data.

Studied smart, not hard — thanks to [Link]

MapReduce

• MapReduce job consists of two phases:

.IN
• Map: In the Map phase, data is read from a distributed
file system and partitioned among a set of computing

C
nodes in the cluster. The data is sent to the nodes as a set
of key-value pairs. The Map tasks process the input

N
records independently of each other and produce
intermediate results as key-value pairs. The intermediate

SY
results are stored on the local disk of the node running
the Map task.
• Reduce: When all the Map tasks are completed, the

U
Reduce phase begins in which the intermediate data with
the same key is aggregated.
VT
• Optional Combine Task
• An optional Combine task can be used to perform data
aggregation on the intermediate data of the same key for
the output of the mapper before transferring the output
to the Reduce task.

Studied smart, not hard — thanks to [Link]

 MapReduce Job Execution Workflow
• MapReduce job execution starts when the client applications submit jobs to the Job tracker.

.IN
• The JobTracker returns a JobID to the client application. The JobTracker talks to the NameNode to determine
the location of the data.

C
• The JobTracker locates TaskTracker nodes with available slots at/or near the data.

N
• The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the

SY
JobTracker that they are still alive. These messages also inform the JobTracker of the number of available
slots, so the JobTracker can stay up to date with where in the cluster, new work can be delegated.

U
VT

Studied smart, not hard — thanks to [Link]

 MapReduce Job Execution Workflow
• The JobTracker submits the work to the TaskTracker nodes when they poll for tasks. To choose a task for a

.IN
TaskTracker, the JobTracker uses various scheduling algorithms (default is FIFO).
• The TaskTracker nodes are monitored using the heartbeat signals that are sent by the TaskTrackers to

C
JobTracker.
• The TaskTracker spawns a separate JVM process for each task so that any task failure does not bring down

N
the TaskTracker.

SY
• The TaskTracker monitors these spawned processes while capturing the output and exit codes. When the
process finishes, successfully or not, the TaskTracker notifies the JobTracker. When the job is completed, the

U
JobTracker updates its status.
VT

Studied smart, not hard — thanks to [Link]

MapReduce 2.0 - YARN

.IN
• In Hadoop 2.0 the original processing engine of Hadoop
(MapReduce) has been separated from the resource
management (which is now part of YARN).

C
N
• This makes YARN effectively an operating system for

SY
Hadoop that supports different processing engines on a
Hadoop cluster such as MapReduce for batch processing,
Apache Tez for interactive queries, Apache Storm for

U
stream processing, etc.
VT
• YARN architecture divides architecture divides the two
major functions of the JobTracker - resource management
and job life-cycle management - into separate components:
• ResourceManager
• ApplicationMaster.

Studied smart, not hard — thanks to [Link]

YARN Components

.IN
• Resource Manager (RM): RM manages the global assignment of
compute resources to applications. RM consists of two main
services:

C
• Scheduler: Scheduler is a pluggable service that manages and enforces the
resource scheduling policy in the cluster.

N
• Applications Manager (AsM): AsM manages the running Application Masters in
the cluster. AsM is responsible for starting application masters and for monitoring

SY
and restarting them on different nodes in case of failures.

• Application Master (AM): A per-application AM manages the

application’s life cycle. AM is responsible for negotiating resources

U
from the RM and working with the NMs to execute and monitor
the tasks. VT
• Node Manager (NM): A per-machine NM manages the user
processes on that machine.
• Containers: Container is a bundle of resources allocated by RM
(memory, CPU, network, etc.). A container is a conceptual entity
that grants an application the privilege to use a certain amount of
resources on a given machine to run a component task.

Studied smart, not hard — thanks to [Link]

Hadoop Schedulers

.IN
• Hadoop scheduler is a pluggable component that makes it open to support
different scheduling algorithms.

C
• The default scheduler in Hadoop is FIFO.

N
• Two advanced schedulers are also available - the Fair Scheduler, developed

SY
at Facebook, and the Capacity Scheduler, developed at Yahoo.

U
• The pluggable scheduler framework provides the flexibility to support a
VT
variety of workloads with varying priority and performance constraints.
• Efficient job scheduling makes Hadoop a multi-tasking system that can
process multiple data sets for multiple jobs for multiple users
simultaneously.

Studied smart, not hard — thanks to [Link]

FIFO Scheduler

.IN
• FIFO is the default scheduler in Hadoop that maintains a work queue
in which the jobs are queued.

C
• The scheduler pulls jobs in first in first out manner (oldest job first)

N
for scheduling.

SY
• There is no concept of priority or size of job in FIFO scheduler.

U
VT

Studied smart, not hard — thanks to [Link]

Fair Scheduler

.IN
• The Fair Scheduler allocates resources evenly between multiple jobs and also provides capacity
guarantees.
• Fair Scheduler assigns resources to jobs such that each job gets an equal share of the available

C
resources on average over time.

N
• Tasks slots that are free are assigned to the new jobs, so that each job gets roughly the same
amount of CPU time.

SY
• Job Pools
• The Fair Scheduler maintains a set of pools into which jobs are placed. Each pool has a guaranteed capacity.

U
• When there is a single job running, all the resources are assigned to that job. When there are multiple jobs in
the pools, each pool gets at least as many task slots as guaranteed.
VT
• Each pool receives at least the minimum share.
• When a pool does not require the guaranteed share the excess capacity is split between other jobs.
• Fairness
• The scheduler computes periodically the difference between the computing time received by each job and
the time it should have received in ideal scheduling.
• The job which has the highest deficit of the compute time received is scheduled next.

Studied smart, not hard — thanks to [Link]

Capacity Scheduler

• The Capacity Scheduler has similar functionally as the Fair Scheduler but

.IN
adopts a different scheduling philosophy.
• Queues

C
• In Capacity Scheduler, you define a number of named queues each with a

N
configurable number of map and reduce slots.

SY
• Each queue is also assigned a guaranteed capacity.
• The Capacity Scheduler gives each queue its capacity when it contains jobs, and
shares any unused capacity between the queues. Within each queue FIFO scheduling

U
with priority is used.
VT
• Fairness
• For fairness, it is possible to place a limit on the percentage of running tasks per user,
so that users share a cluster equally.
• A wait time for each queue can be configured. When a queue is not scheduled for
more than the wait time, it can preempt tasks of other queues to get its fair share.

Studied smart, not hard — thanks to [Link]

Further Reading

.IN
• Apache Hadoop, [Link]
• Apache Hive, [Link]
• Apache HBase, [Link]

C
• Apache Chukwa, [Link]

N
• Apache Flume, [Link]

SY
• Apache Zookeeper, [Link]
• Apache Avro, [Link]
•

U
Apache Oozie, [Link]
• Apache Storm, [Link]
VT
• Apache Tez, [Link]
• Apache Cassandra, [Link]
• Apache Mahout, [Link]
• Apache Pig, [Link]
• Apache Sqoop, [Link]

Studied smart, not hard — thanks to [Link]