Multimedia

Computing
D ATA SO U R C ES

Abdulmotaleb El Saddik
FRSC, FIEEE, FCAE, FEIC
Distinguished Professor
[Link] Multimedia Communications Research Lab
[Link]
[Link] [Link]

[Link]
Digital Twins
Convergence of Multimedia TechnologiesMassive
throughput
Cybersecurity
(including biometric, Massive low
Data
privacy)
IoT
Social latency
Networks

Massive
Things and Being are interconnected sensing
Cybersecurity & Biometrics Finance
Medical
Massive
Source of data: People,
heterogeneity
Process, Product
AI Machine MMI
Data from
Data from
Learning Audio/
various
hard-sensors Cognition
surveillance AR/VR Fast feedback
cameras Video
Data from various
soft sensors Data from the
Ontology
crowd
QoE-powered
Demographic data
from various parts
Data from health
Deep Communications Massive
of city
Learning Haptics/
monitors of
citizens
Rules
(BT, Wi-Fi, 4G, Holograms
H-Robots
privacy
5G, etc.)
Security/Trust
© 2016, Prof. A. El Saddik, elsaddik@[Link], reproduce with permission

BigMM & AI Multimodal Interactions 5G & Tactile Internet

© 2002 Multimedia Communications research Laboratory (MCRLab) 3

Ecosystem of the Digital Twin for health and well-being.

Published in IEEE MMTC Communications – Frontiers. “Dtwins: A Digital Twins Ecosystem For Health
And Well-Being” by El Saddik et al.
© 2002 Multimedia Communications research Laboratory (MCRLab) 4
Multimedia Data Sources
Multimedia data sources are mainly:

– hard sensors

➔ They are part of the Internet of Things

– soft sensors

➔ Mainly social networks

© 2002 Multimedia Communications research Laboratory (MCRLab) 5

 Internet of Things

© 2002 Multimedia Communications research Laboratory (MCRLab) 6

Definition
• Internet of Things (IoT) is a network that
connects together physical things that are
provided with a unique identifier, and enables
them to transmit and receive data over a
network.

© 2002 Multimedia Communications research Laboratory (MCRLab) 7

What are the “Things”?
• Anything and everything
– Goods, objects, machines, applications, buildings, vehicles,
animals, people, plants, …

[Link]

© 2002 Multimedia Communications research Laboratory (MCRLab) 8

What common is with all things?

1. unique
identifier

4. controller

3. sensors [Link]
© 2002 Multimedia Communications research Laboratory (MCRLab) 9
• Why do we need IoT?
– Monitor: connectivity means more data, from many places ➔
more ways to increase efficiency and improve safety and
security.
– Control: when an object represents itself digitally, it can be
controlled from anywhere.
– Connect: with people, devices, pets, etc.
– Search: “Where’s my iPhone”, where are my keys?
– Play: social engagement, mobile games,

© 2002 Multimedia Communications research Laboratory (MCRLab) 10

IoT Statistics and Estimations

© 2002 Multimedia Communications research Laboratory (MCRLab) 11

IoT and Smart Objects
• Smart Objects are among the main building blocks of the
Internet of Things.
• Smart object: is an item equipped with a sensor or actuator,
a microprocessor, a communication device, and a power
source.

“Ordinary” chair Smart chair

© 2002 Multimedia Communications research Laboratory (MCRLab) 12

Characteristics
• What does a thing need to be part of IoT ?
– Have a unique identifier
– Either a sensor and/or actuator (receive and/or transmit data)
– Processor
– Communication
– Power source

© 2002 Multimedia Communications research Laboratory (MCRLab) 13

Unique Identifier
• IPv4 to IPv6 transition

© 2002 Multimedia Communications research Laboratory (MCRLab) 14

Unique Identifier

© 2002 Multimedia Communications research Laboratory (MCRLab) 15

Definition: Sensors
Sensors are transducers that can detect specific
events or changes in environment

© 2002 Multimedia Communications research Laboratory (MCRLab) 16

Sensors
• A sensor is a device which detects or measures a
physical property
• They need a communication module (sometimes
build-in) to send the information to other electronics
• Provide a means to objectively monitor or track
objects or humans or animals, etc.

• They transmit data from the physical world to the

Edge/Cloud
• They help build a virtual image of the physical world.

© 2002 Multimedia Communications research Laboratory (MCRLab) 17

Sensors
• Fixed Sensors:The fixed sensors are either
hardwired or wireless that can be used on site
to gather information from the surrounding
environment.
– Geomagnetic sensor
– Traffic enforcement camera

Red Light Speed

Camera Camera

© 2002 Multimedia Communications research Laboratory (MCRLab) 18

Sensors
• Mobile Sensors: mobile sensors depend on
the change of positions to acquire and
exchange information.
– GPS receiver
• Vehicles equipped with GPS receiver can be a traffic data provider.
• Position, speed → traffic condition.
• Communication delays and data packet losses→ appropriate processing
algorithm.
• Privacy legislation.
– Multi-Sensor Node

© 2002 Multimedia Communications research Laboratory (MCRLab) 19

Sensors
• Temperature Sensor
• Proximity Sensor
• Accelerometer
• IR Sensor (Infrared Sensor)
• Pressure Sensor
• Light Sensor
[Link]

• Ultrasonic Sensor
• Smoke, Gas and Alcohol Sensor
• Touch Sensor
• Color Sensor
• Humidity Sensor
• Tilt Sensor
• Flow and Level Sensor

[Link]

© 2002 Multimedia Communications research Laboratory (MCRLab) 20

Sensors
What can we do with collected data?
• Make Smart Decisions
• Analysis & Learn Patterns
• Control Devices Automatically
• Connectivity
• Security & Make Things Safer

© 2002 Multimedia Communications research Laboratory (MCRLab) 21

Physical World to Virtual World

• IoT sensors allow us to build models of

virtual things that represent the real world
things

Virtual Virtual Virtual

Virtual
Thing 1 Thing 2 Thing n
World
Data
Data

Data
Physical
Physical Physical Physical World
Thing 1 Thing 2 Thing n

© 2002 Multimedia Communications research Laboratory (MCRLab) 22

Actuators
• An actuator is a component of a physical
system that converts one type of energy into
another.
• An actuator requires:
– a signal
– A power/energy source.

© 2002 Multimedia Communications research Laboratory (MCRLab) 23

Actuators: examples
• Servo motor
• Vibration motor
• Thermoelectric cooler
• Sound speakers
• Etc.

© 2002 Multimedia Communications research Laboratory (MCRLab) 24

Virtual World to Physical World
• IoT actuators allow the virtual world to act
on / control the real world.
• Provide a means to take action in the
physical world and control aspects of it.
Virtual Virtual Virtual
Virtual
Thing 1 Thing 2 Thing n
World
Action

Action

Action
Physical
Physical Physical Physical World
Thing 1 Thing 2 Thing n

© 2002 Multimedia Communications research Laboratory (MCRLab) 25

Controllers
• Local computing/Embedded Intelligence
• Real time applications

© 2002 Multimedia Communications research Laboratory (MCRLab) 26

Controllers
• Microprocessor: contains a CPU only
• Microcontroller: a programmable device
that contains a microprocessor, memory
(RAM/ROM), and I/O signal lines on a single
chip.
• Microcomputer: contains a CPU, memory,
and I/O devices or ports

Microprocessor < Microcontroller < Microcomputer

© 2002 Multimedia Communications research Laboratory (MCRLab) 27

Connectivity
• Sensor Network communication
– Bluetooth Low Energy (BLE)
– Zigbee
– Wi-Fi
– Cellular
• Considerations
– Data rate and bandwidth requirements
– Transmission frequency
– Reliability requirements
– Security
– Power consumption

© 2002 Multimedia Communications research Laboratory (MCRLab) 28

Important Challenges
• Power
– Wired vs Wireless devices
– Energy saving techniques
– Renewable power sources: Solar, Piezoelectric generation, etc.
• Storage
– Local vs Cloud
• Security and Privacy
– Network security
– Physical devices
– Malware protection
• Standard
© 2002 Multimedia Communications research Laboratory (MCRLab) 29
 Networking Re-Cap

© 2002 Multimedia Communications research Laboratory (MCRLab) 30

Definitions
• Algorithm
– In general: a set of instructions that are followed in a fixed order and used for solving a mathematical
problem, or making a computer program, etc.
• Method
– A set of abstract-level steps to accomplish a goal.
• Methodology
– A set of methods and principles that used when studying a particular subject or doing a particular kind
of work.
• System
– In general means: any scheme of classification or arrangement; or a group or combination of
interrelated, interdependent, or interacting elements forming a collective entity.
– In Multimedia terminology; Multimedia system is a combination of discrete media, e.g., plain text and
continuous media such as video or audio and recently haptics.
• Framework
– A set of ideas, or rules from which something is developed, or on which decisions are addressed.
– In multimedia, it is a group of rules and formulas that used to build a multimedia system capable to
represent, transmit, receive, and store any kind of media data.
– It offers an intuitive API and a modular architecture to easily add a support for new audio, video and
container formats and transmission protocols.

© 2002 Multimedia Communications research Laboratory (MCRLab) 31

Definitions: Example
• Going to “Bayshore” Mall from Campus
– System: Transportation
– Methodology: Public transportation
– Method: Taking Bus
– Algorithm:
• Going to campus station
• Waiting for bus # 97 to Bayshore
• …
• Getting off from the bus at Bayshore Station

© 2002 Multimedia Communications research Laboratory (MCRLab) 32

Network Classification
⚫ Networks are often
classed according to how
S
p
1Gb/s
large they are. The
e
e 100Mb/s LAN classical classes are
d
10Mb/s called LAN, MAN and
o
MAN
f
1Mb/s WAN WAN.
N
e – A LAN (Local Area
t 100Kb/s
w Network) connects
o
r 10Kb/s hosts in a single building
k
s or across a single
1Kb/s
campus.
– A MAN (Metropolitan
0.1m 1m 10m 100m 1Km 10Km 100Km 1000Km Area Network) connects
Distances Spanned by Networks hosts across a town or a
city.
– A WAN (Wide Area
Network) connects
hosts across a country
or the world.

© 2002 Multimedia Communications research Laboratory (MCRLab) 33

Networking Technology
• Industrial communication networks can be
modeled according to the Open System
Interconnection (OSI) model:
– The International Organization for Standardization (ISO)
began developing the Open Systems Interconnection (OSI)
model in 1977
– It is now the most accepted standard for network modeling
– The OSI model is not a protocol; it is a model for
understanding and designing a network architecture that is
flexible, robust and interoperable.

© 2002 Multimedia Communications research Laboratory (MCRLab) 34

Open System Interconnection (OSI) model

Defines rules that apply to the following issues

[Link] network devices contact each other
[Link] network devices communicate with each other
[Link] has the right to transmit data
[Link] transmissions received correctly and by the right node
[Link] physical media are arranged and connected
[Link] that network devices maintain a proper rate of data
flow
[Link] bits are represented on the network media

© 2002 Multimedia Communications research Laboratory (MCRLab) 35

OSI model divides tasks into seven layers

All services directly called by the end user

7 Application (Mail, File Transfer,...) e.g. Telnet, SMTP
→ everything else!
Definition and conversion of the data
6 Presentation formats (e.g. ASN 1)
→ how to encode/decode msgs, security
Management of connections
5 Session (e.g. ISO 8326)
→ how to manage connections
End-to-end flow control and error recovery
4 Transport (e.g. TP4, TCP)
→ how to send packets reliably
Routing, possibly segmenting
3 Network (e.g. IP, X25)
→ how to route packets
Error detection, Flow control and error recovery,
2 Link medium access (e.g. HDLC)
→ how to transmit frames
Coding, Modulation, Electrical and
1 Physical mechanical coupling (e.g. RS485)
→ how to transmit bits

© 2002 Multimedia Communications research Laboratory (MCRLab) 36

Peer-to-Peer Communication Process

© 2002 Multimedia Communications research Laboratory (MCRLab) 37

3 Sub-groups
7 Application
User
support Allow interoperability among unrelated
6 Presentation software systems
layers

5 Session

Transport
layer
4 Transport links the two subgroups

3 Network

Network Deal with the physical aspects of

support 2 Link moving data from one device to another
layers
1 Physical

© 2002 Multimedia Communications research Laboratory (MCRLab) 38

Wi-Fi
• Standardized by IEEE 802.11
• Has become the dominant technology for
wireless LANs
• First important use was in the home to replace
Ethernet cabling for connecting desktop and
laptop computers with each other and with
the Internet
• Provides a cost effective way to the Internet
• Is essential to implementing the
Internet of Things (IoT)
© 2002 Multimedia Communications research Laboratory (MCRLab) 39
Wi-Fi
• Economic benefit of Wi-Fi is most clearly seen in
the enterprise
• Approximately half of all enterprise network
traffic is via Wi-Fi rather then the traditional
Ethernet
• Two trends have driven the transition to a Wi-Fi
centered enterprise:
– Demand has increased with more and more employees preferring to
use laptops, tablets, and smartphones to connect to the enterprise
network
– The arrival of Gigabit Ethernet allows the enterprise network to
support high-speed connections to mobile devices simultaneously

© 2002 Multimedia Communications research Laboratory (MCRLab) 40

Standards
• Interoperability is essential to the success of Wi-
Fi
• Wi-Fi enabled devices must be able to
communicate with Wi-Fi access points regardless
of the manufacturer of the device or access point
• Interoperability is guaranteed by:
– IEEE 802.11 wireless LAN committee develops the protocol and
signaling standards
– The Wi-Fi Alliance creates test suites to clarify interoperability for
commercial products that conform to various IEEE 802.11 standards
– The term Wi-Fi (wireless fidelity) is used for products certified by the
Alliance

© 2002 Multimedia Communications research Laboratory (MCRLab) 41

Wi-Fi Data Rates
• 802.11 (1997): 2 Mbps (megabits per second,
million bits per second)
• 802.11a (1999): 54 Mbps
• 802.11b (1999): 11 Mbps
• 802.11n (1999): 600 Mbps
• 802.11g (2003): 54 Mbps
• 802.11ad (2012): 6.76 Gbps (60-GHz band)
• 802.11ac (2014): 3.2 Gbps (5-GHz band
similar to older generations)

© 2002 Multimedia Communications research Laboratory (MCRLab) 42

Analog Signals
• Spectrum: Range of frequencies (number of cycles in second) contained in signal.
• Absolute bandwidth: Width of spectrum
• Effective bandwidth : ( bandwidth) Narrow band of frequencies containing most of the energy.
• According to Fourier analysis, any composite signal can be
represented as a combination of simple sine waves with different
frequencies, phases, and amplitudes

Figure from signal and system by Simon Haykin

© 2002 Multimedia Communications research Laboratory (MCRLab) 43

Digital Signal

• Most of the digital signals are aperiodic and, thus period or frequency is not appropriate, as it has infinite
bandwidth!

• Bit interval (instead of period) and Bit rate (instead of frequency)

• The bit interval is the time required to send one single bit.

• The bit rate is the number of bit intervals in one second, usually expressed in bits per second (bps).

Figure from signal and system by Simon Haykin

© 2002 Multimedia Communications research Laboratory (MCRLab) 44

Relation Between Hz and b/sec

© 2002 Multimedia Communications research Laboratory (MCRLab) 45

Wireless agencies and standards

Agency Purpose Website

Institute of electrical and Creates and maintain operational [Link]
electronics (IEEE) standards

Federal communications Regulates the use of wireless device [Link]

commission (FCC) in the US

European Telecommunication Charted to produce common [Link]

standard institute (ETSi) standards in Europe

Wi-Fi Alliance Promotes and tests for WLAN [Link]

interoperability

46
© 2002 Multimedia Communications research Laboratory (MCRLab) 46
1G
• Developed in 1980s & completed
in early 1990s
• Based on analog system
• Speed up to 2.4 kbps
• AMPS (Advance Mobile Phone System) was
launched by the US & it was the 1G mobile
system
• Allows user to make voice calls in 1 country

© 2002 Multimedia Communications research Laboratory (MCRLab) 47

2G
• Developed in late 1980s & completed
in late 1990s
• Based on digital system
• Speed up to 64 kbps
• Services such are digital voice & SMS with
more clarity
• Semi global facility

© 2002 Multimedia Communications research Laboratory (MCRLab) 48

3G
• Developed late 1990s & early 2000s
until present day
• Transmission speed from 125 kbps to
2 Mbps
• Superior voice quality
• Good clarity in video conference
• E-mail, PDA, information surfing, on-line
shopping/ banking, games, etc.
• Global roaming

© 2002 Multimedia Communications research Laboratory (MCRLab) 49

4G
• Developed in 2010
• Faster & more reliable
• Speed up to 100 Mbps
• High performance
• Easy roaming
• Low cost

© 2002 Multimedia Communications research Laboratory (MCRLab) 50

5G
• Next major phase of mobile
telecommunication & wireless
system
• 10 times more capacity than
others
• Expected speed up to 1 Gbps
• More faster & reliable than 4G
• Lower cost than previous
generations

© 2002 Multimedia Communications research Laboratory (MCRLab) 51

Key concepts
• Real wireless world with no more limitations with access &
zone issues
• Wearable devices
• IPv6, where a visiting care of mobile IP address is assigned
according to location & connected network
• One unified global standard
• Smart radio
• The user can simultaneously be connected with several
wireless access technology
• Multiple concurrent data transfer path

© 2002 Multimedia Communications research Laboratory (MCRLab) 52

Beyond 5G: The 6G Horizon
Conceptual Promises of 6G:
• Ubiquitous Artificial Intelligence (AI): 6G
will integrate AI deeply into the network
infrastructure to enable intelligent
responses to network demands and user
needs.
• Holographic Communication: Anticipates
the norm of holographic types of
communication, requiring massive data
and extremely low latency.
• Deep Connectivity: Envisions seamless
communication between land, sea, air,
and space, including satellite networks.

Performance
Targets: Speed: Targets speeds Latency: Aims for sub- Reliability: Expects Spectrum Utilization:
up to 1 terabit per millisecond latencies to near-perfect reliability Innovations in terahertz
second, significantly support real-time for critical applications, (THz) bands and
higher than 5G. applications beyond like remote surgery or intelligent surface
current capabilities. autonomous transport. materials for better
signal propagation.

© 2002 Multimedia Communications research Laboratory (MCRLab) 53

Overall Objectives of the Future 6G Development

Feature/metric 4G 5G 6G

Per device peak data rate 1 Gbps 10 Gbps 1 Tbps

End-to-end latency >100 ms >10 ms >1 ms

Maximum spectral efficiency 15 bps/Hz 30 bps/Hz 100 bps/Hz

Mobility support [km/h] Up to 350 Up to 500 Up to 1000

Satellite integration No Partial Fully

AI No Partial Fully

Haptic communication No Partial Fully

THz communication No Very limited Widely

Architecture MIMO Massive Intelligent

MIMO surface

Service level Video VR, AR Tactile

Maximum frequency 6 GHz 90 GHz 10 THz

© 2002 Multimedia Communications research Laboratory (MCRLab) 54

 IoT Applications

© 2002 Multimedia Communications research Laboratory (MCRLab) 55

How everything works together
4 main blocks of IoT structure:

• Hardware Sensors Platforms

Data Acquisition
• Soft Sensors Platforms

Information
Creation
• Software Processing

Sense and
Meaning Making
• Information Visualization

Action Taking • Do Something

© 2002 Multimedia Communications research Laboratory (MCRLab) 56

Application: Smart Homes
Smart Home Control from a Virtual Environment

[Link]

© 2002 Multimedia Communications research Laboratory (MCRLab) 57

IoT for Smart Homes
• Common IoT devices used for smart homes

• Other devices include sound sensor, camera, gas sensor…

© 2002 Multimedia Communications research Laboratory (MCRLab) 58
Application: IoT Enabler System for Smart
Homes
• System that allows users to specify and centrally
control Internet of Things smart objects
• Offers simple language for the specification of
control rules for smart objects
• Provides a user interface to graphically illustrate
the data received from smart objects
➔This allows non-IT users to build smart objects and
define control rules in a smart home environment.

Source for IoT Enabler for Smart Homes, including next 2 slides: Basim Hafidh, Hussein Al Osman, Juan
Arteaga-Falconi, Haiwei Dong, and Abdulmotaleb El Saddik, “SITE: The Simple Internet of Things
Enabler for Smart Homes”, IEEE Access, 5, (2017): 2034-2049.

© 2002 Multimedia Communications research Laboratory (MCRLab) 59

Application: IoT Enabler System for Smart
Homes (cont.)
Turn any object into a smart object using this system.
Use a main block provided by the system and
attached designed blocks to it depending on the
desired outcome.
Define the rules: - what is the trigger and
- what is the desired output

© 2002 Multimedia Communications research Laboratory (MCRLab) 60

IoT for human Digital Twin
• Aspects of well-being, fitness, etc
• Digital twin of physical health
• Can IoT assist in building a DT model of the
RT ? What can we achieve ?

© 2002 Multimedia Communications research Laboratory (MCRLab) 61

Application: Development of an Automatic
3D Human Head Scanning

[Link]

© 2002 Multimedia Communications research Laboratory (MCRLab) 62

Application: Digital Twin for Health and
Well-being
• What types of sensors ?
– Wearables and personal health devices

© 2002 Multimedia Communications research Laboratory (MCRLab) 63

Smart Insole
Forward Walking Backward Walking

© 2002 Multimedia Communications research Laboratory (MCRLab) 64

Smart Insole for health and sport

© 2002 Multimedia Communications research Laboratory (MCRLab) 65

Smart Insole: Results

Faisal Arafsha, Fedwa Laamarti, Abdulmotaleb El Saddik, “Cyber-Physical System Framework for
Measurement and Analysis of Physical Activities”, Electronics, 2019, vol. 8, issue 2.

© 2002 Multimedia Communications research Laboratory (MCRLab) 66

Application: IoT for Sport

Sport and
Fitness Data

© 2002 Multimedia Communications research Laboratory (MCRLab) 67

© 2002 Multimedia Communications research Laboratory (MCRLab) 68