Module 1

INTRODUCTION TO VR
● Introduction to VR

● Modern Experiences,

● Historical Perspective.

● VR Applications.

● Birds-Eye View for the Hardware-Sensors, Displays

● Software-Virtual World Generator, Game Engines, Human Senses,

● Human Psychology and Perceptions.

 Introduction
Virtual reality is the technology that teleports you to an immersive 3D environment.
It can be designed for human interaction for specific reasons in order to create experiences not
otherwise possible.
Virtual reality (VR) is a simulated experience that uses pose tracking and 3D near-eye displays to
provide an immersive feel of a virtual world.

Standard virtual reality systems use virtual reality headsets to generate realistic images, sounds
and other sensations that simulate user presence in a virtual environment.

Virtual reality equipment enables users to look around the artificial world, move around in it, and
interact with virtual items.
Definition of VR: Inducing targeted behavior in an organism by using artificial sensory
stimulation, while the organism has little or no awareness of the interference.

1. Targeted behavior: The organism is having an “experience” that was designed by the creator.
Examples include flying, walking, exploring, watching a movie, and socializing with other
organisms.

2. Organism: This could be you, someone else, or even another life form such as a fruit fly,
cockroach, fish, rodent, or monkey

3. Artificial sensory stimulation: Through the power of engineering, one or more senses of the
organism become co-opted, at least partly, and their ordinary inputs are replaced or enhanced by
artificial stimulation.

4. Awareness: While having the experience, the organism seems unaware of the interference,
thereby being “fooled” into feeling present in a virtual world. This unawareness leads to a sense of
presence in an altered or alternative world. It is accepted as being natural.
Types of VR

● Non-immersive Virtual Reality.

● Fully Immersive Virtual Reality.

● Semi-Immersive Virtual Reality.

● Augmented Reality.

● Collaborative VR.
Non-immersive VR
Non-immersive virtual reality refers to a virtual experience through a computer where you can

control some characters or activities within the software, but the environment is not directly

interacting with you.

For example, when you play video games such as World of WarCraft, you can control characters

within the game that have their own animations and attributes.

All basic forms of gaming devices, such as PlayStation, Xbox, Computer, etc, provide you with

a non-immersive virtual reality experience.

 Fully immersive VR
On contrary to non-immersive virtual reality, a fully immersive virtual technology ensures that
you have a realistic experience within the virtual world.
This is an expensive form of virtual reality that involves helmets, gloves, and body connectors
with sense detectors.
These are connected to a powerful computer. Your movements, reactions, and even a blink of an
eye are detected and projected within the virtual world.
You will feel like you are within the virtual world physically.
One example could be a Virtual Shooter gaming zone where you will be equipped with the gears
in a small room and you will be viewing a virtual world through the helmet where you are facing
other shooters trying to kill you.
You will move your arms and body to run, jump, crouch, shoot, throw, and many more within the
game.
Semi-immersive VR
A semi-immersive virtual reality is a mixture of non-immersive and fully immersive virtual
reality.
This can be in the form of a 3D space or virtual environment where you can move about on your
own, either through a computer screen or a VR box/headset.
So all activities within the virtual world are concentrated toward you.
However, you have no real physical movements other than your visual experience.
On a computer, you can use the mouse to move about the virtual space, and on mobile devices,
you can touch and swipe to move about the place.
 Augmented Reality
Augmented Reality is when a certain entity or device seems to be present in reality but is actually
not.
Rather than putting you into a virtual world, a virtual entity is placed in the real world through any
device.
For example, through your mobile screen, you can view your room, and probably place a cartoon
character in the corner. You will be able to see the character through your mobile screen and not in
reality.
It is mostly used by businesses such as furniture suppliers or decorators.
For example, a person willing to buy a table will be able to place the table in his room through his
phone display.
This will let him understand if this table is suitable and looks good in his room or if he has to
choose another design.
Collaborative VR

This is a form of a virtual world where different people from various locations can come into contact
within a virtual environment, usually in the form of 3D or projected characters.
For example, Unity game development has also embraced the concept of virtual collaboration, much like
the video game called PUBG (Players Unknown Battleground), where tons of players come into existence
as individual virtual characters that they can control.
Much like the transformative impact of igaming development in its sector, VR technologies are paving the
way for innovative experiences across various domains.
Here they can interact with each other through microphones, headsets, and chatting.
Recently people are getting used to virtual meeting rooms to conduct business meetings remotely, or for
conducting virtual debate competitions.
For those interested in enhancing their virtual collaboration, learning how to conduct virtual meetings
effectively is crucial, offering a blend of convenience and efficiency in today’s digital landscape.
The main goal of this form of VR is to create collaboration between people. As virtual reality continues to
improve over time, the advancement of translation for VR will pave the way for immersive multilingual
 Modern VR Experiences
The current generation of VR systems was brought about by advances in display, sensing,
and computing technology from the smartphone industry.

From Palmer Luckey’s 2012 Oculus Rift design to building a viewing case for
smartphones, the world has quickly changed as VR headsets are mass produced and placed
onto the heads of millions of people.

This trend is similar in many ways to the home computer and web browser revolutions; as
a wider variety of people have access to the technology, the set of things they do with it
substantially broadens.
 Video games

● People have dreamed of entering their video game worlds for decades.
● By 1982, this concept was already popularized by the Disney movie Tron.
● Most gamers currently want to explore large, realistic worlds through an avatar.
….Video Games Example

(a) Valve’s Portal 2 demo, which shipped with The Lab for the HTC Vive
headset, is a puzzle-solving experience in a virtual world.
(b) The Virtuix Omni treadmill for walking through first-person shooter
games.
(c) Lucky’s Tale for the Oculus Rift maintains a third-person perspective as
the player floats above his character.
(d) In the Dumpy game from DePaul University, the player appears to have
a large elephant trunk. The purpose of the game is to enjoy this unusual
embodiment by knocking things down with a swinging trunk.
 Immersive cinema
Hollywood movies continue to offer increasing degrees of realism.

Movie directors are entering a fascinating new era of film.

The tricks of the trade that were learned across the 20th century need to be investigated because they are
based on the assumption that the cinematographer controls the camera viewpoint.

In VR, viewers can look in any direction, and perhaps even walk through the scene.

What should they be allowed to do?

How do you make sure they do not miss part of the story?

Should the story be linear, or should it adapt to the viewer’s actions?

Should the viewer be a first-person character in the film, or a third-person observer who in
invisible to the other characters?

How can a group of friends experience a VR film together?

When are animations more appropriate versus the capture of real scenes?

It will take many years to resolve these questions and countless more that will arise. In the
meantime,

VR can also be used as a kind of “wrapper” around existing movies.

Figure shows the VR Cinema application, which allows the user to choose any seat in a virtual
movie theater.
Whatever standard movies or videos that are on the user’s hard drive can be
streamed to the screen in the theater.

These could be 2D or 3D movies.

A projector in the back emits flickering lights and the audio is adjusted to mimic
the acoustics of a real theater.

This provides an immediate way to leverage all content that was developed for
viewing on a screen, and bring it into VR.
In 2015, Oculus Story Studio produced Emmy-winning Henry, an immersive short story about an unloved hedgehog who hopes to
make a new friend, the viewer

[Link]
gZjaHJvbWUyBggAEEUYOTIHCAEQIRigAdIBCTEzOTcwajBqOagCALACAA&sourceid=chrome&ie=UTF-8#fpstate=ive&vld=cid:7f
ba400f,vid:IUY2yI5F16U,st:0
VR Cinema, developed in 2013 by Joo-Hyung Ahn for the Oculus Rift. Viewers
could choose their seats in the theater and watch any movie they like.
Telepresence
The first step toward feeling like we are somewhere else is capturing a panoramic view of
the remote environment.
Google’s Street View and Earth apps already rely on the captured panoramic images
from millions of locations around the world.
Simple VR apps that query the Street View server directly enable to user to feel like he is
standing in each of these locations, while easily being able to transition between nearby
locations.
Panoramic video capture is even more compelling.
Even better is to provide live panoramic video interfaces, through which people can
attend sporting events and concerts.
● Through a live interface, interaction is possible.
● People can take video conferencing to the next level by feeling present at the remote
location.
● Current VR technology allows us to virtually visit far away places and interact in
most of the ways that were previously possible only while physically present.
● This leads to improved opportunities for telecommuting to work. This could
ultimately help reverse the urbanization trend sparked by the 19th-century industrial
revolution, leading to urbanization as we distribute.
An important component for achieving telepresence is to capture a panoramic view: (a) A car with
cameras and depth sensors on top, used by Google to make Street View. (b) The Insta360 Pro captures
and streams omnidirectional videos.
A simple VR experience that presents Google Street View images through a VR headset: (a) A familiar
scene in Paris. (b) Left and right eye views are created inside the headset, while also taking into
account the user’s looking direction
Virtual societies
● Whereas telepresence makes us feel like we are in another part of the physical world, VR also
allows us to form entire societies that remind us of the physical world, but are synthetic worlds
that contain avatars connected to real people.
● Groups of people could spend time together in these spaces for a variety of reasons, including
common special interests, educational goals, or simply an escape from ordinary life.

Virtual societies develop through

interacting avatars that meet in virtual
worlds that are maintained on a
common server. A snapshot from
Second Life is shown here.
Empathy
The first-person perspective provided by VR is a powerful tool for causing people to feel empathy for
someone else’s situation.

The world continues to struggle with acceptance and equality for others of different race, religion, age,
gender, sexuality, social status, and education, while the greatest barrier to progress is that most people
cannot fathom what it is like to have a different identity.

In Clouds Over Sidra, 2015, film producer

Chris Milk offered a first person perspective
on the suffering of Syrian refugees (figure by
Within, Clouds Over Sidra)
In 2014, BeAnotherLab, an interdisciplinary collective, made “The Machine to Be Another” where you
can swap bodies with the other gender. Each person wears a VR headset that has cameras mounted
on its front. Each therefore sees the world from the approximate viewpoint of the other person. They
were asked to move their hands in coordinated motions so that they see their new body moving
appropriately.
Education
In addition to teaching empathy, the first-person perspective could revolutionize many
areas of education. In engineering, mathematics, and the sciences,

VR offers the chance to visualize geometric relationships in difficult concepts or data that
are hard to interpret.

Furthermore, VR is naturally suited for practical training because skills developed in a

realistic virtual environment may transfer naturally to the real environment.

The motivation is particularly high if the real environment is costly to provide or poses
health risks.
A flight simulator in use by the US Air Force (photo by Javier Garcia, U.S. Air Force). The user
sits in a physical cockpit while being surrounded by displays that show the environment.
A tour of the Nimrud palace of Assyrian King Ashurnasirpal II, a VR experience
developed by Learning Sites Inc. and the University of Illinois in 2016.
Virtual prototyping
In the real world, we build prototypes to understand how a proposed design feels or functions.

Virtual prototyping enables designers to inhabit a virtual world that contains their prototype.

They can quickly interact with it and make modifications.

They also have opportunities to bring clients into their virtual world so that they can communicate
their ideas. Imagine you want to remodel your kitchen.

You could construct a model in VR and then explain to a contractor exactly how it should look.

Virtual prototyping in VR has important uses in many businesses, including real estate,
architecture, and the design of aircraft, spacecraft, cars, furniture, clothing, and medical
instruments.
Architecture is a prime example of where a virtual prototype is invaluable. This demo, called Ty Hedfan, was
created by IVR-NATION. The real kitchen is above and the virtual kitchen is below.
Health care
Although health and safety are challenging VR issues, the technology can also
help to improve our health.
There is an increasing trend toward distributed medicine, in which doctors train
people to perform routine medical procedures in remote communities around the
world.
Doctors can provide guidance through telepresence, and also use VR technology
for training.
In another use of VR, doctors can immerse themselves in 3D organ models that
were generated from
They can also explain medical options to the patient or his family so that they may
make more informed decisions.
In yet another use, VR can directly provide therapy to help patients. Examples
include overcoming phobias and stress disorders through repeated exposure,
improving or maintaining cognitive skills in spite of aging, and improving motor
skills to overcome balance, muscular, or nervous system disorders. VR systems
could also one day improve longevity by enabling aging people to virtually travel,
engage in fun physical therapy, and overcome loneliness by connecting with family
and friends through an interface that makes them feel present and included in
remote activities.
A heart visualization system based on images of a real human heart. This was developed by
the Jump Trading Simulation and Education Center and the University of Illinois.
New human experiences
Finally, the point might be to simply provide a new human experience.

Through telepresence, people can try experiences through the eyes of robots or
other people.

However, we can go further by giving people experiences that are impossible (or
perhaps deadly) in the real world.

Most often, artists are the ones leading this effort.

(a) In 2014, Epic Games created a wild roller coaster ride through virtual living
room. (b) A guillotine simulator was made in 2013 by Andre Berlemont, Morten
Brunbjerg, and Erkki Trummal. Participants were hit on the neck by friends as the
blade dropped, and they could see the proper perspective as their heads rolled.
Historical Perspective
Staring at rectangles

We start with a history that predates what most people would consider to be VR,
but includes many aspects crucial to VR that have been among us for tens of
thousands of years.

Long ago, our ancestors were trained to look at the walls and imagine a 3D world
that is part of a story.
(a) A 30,000-year-old painting from the Bhimbetka rock shelters in India (photo by Archaeological
Survey of India).
(b) An English painting from around 1470 that depicts John Ball encouraging Wat Tyler rebels
(unknown artist).
(c) A painting by Hans Vredeman de Vries in 1596.
(d) An impressionist painting by Claude Monet in 1874.
Moving pictures

Once humans were content with staring at rectangles on the wall, the next step was to put them into
motion.

The phenomenon of stroboscopic apparent motion is the basis for what we call movies or motion
pictures today.

Flipping quickly through a sequence of pictures gives the illusion of motion, even at a rate as low as
two pictures per second.

Above ten pictures per second, the motion even appears to be continuous, rather than perceived as
individual pictures
Toward convenience and portability

Further motivations for accepting lower levels of realism are cost and portability even though they
could go to theaters and watch high-resolution, color, panoramic, and 3D movies at the time.
Such tiny, blurry, black-and-white television sets seem comically intolerable with respect to our
current expectations.

The next level of portability is to carry the system around with you. Thus, the progression is from:

1) having to go somewhere to watch it,

2) being able to watch it in your home,

3) being able to carry it anywhere. Whether pictures, movies, phones, computers, or video
games, the same progression continues.

We can therefore expect the same for VR systems. At the same time, note that the gap is closing
between these levels: The quality we expect from a portable device is closer than ever before to
the version that requires going somewhere to experience it.
Video games
Motion pictures yield a passive, third-person experience, in contrast to video games which are closer
to a first-person experience by allowing us to interact with him.

Video games are an important step toward closed-loop VR, whereas motion pictures are open-loop.

First-person shooter (FPS) games such as Doom gave the player a first-person perspective and
launched a major campaign over the following decade toward higher quality graphics and realism.

Assassin’s Creed shows a typical scene from a modern, realistic video game. At the same time, wildly
popular games have emerged by focusing on simplicity. Angry Birds looks reminiscent of games from
the 1980s, and Minecraft allows users to create and inhabit worlds composed of course blocks.

Note that reduced realism often leads to simpler engineering requirements; in 2015, an advanced FPS
game might require a powerful PC and graphics card, whereas simpler games would run on a basic
smartphone.
VR headsets

● Once again, the trend toward portability appears. An important step for VR was taken in 1968
with the introduction of Ivan Sutherland’s Sword of Damocles, which leveraged the power of
modern displays and computers.
● He constructed what is widely considered to be the first VR headset.
● As the user turns his head, the images presented on the screen are adjusted to compensate so that
the virtual objects appear to be fixed in space.
● The perception of stationarity. To make an object appear to be stationary while you move your
sense organ, the device producing the stimulus must change its output to compensate for the
motion.
● This requires sensors and tracking systems to become part of the VR system. Commercial VR
headsets started appearing in the 1980s with Jaron Lanier’s company VPL, thereby popularizing
the image of goggles and gloves;
(a) CAVE virtual environment, Illinois
Simulator Laboratory, Beckman Institute,
University of Illinois at Urbana-Champaign,
1992 (photo by Hank Kaczmarski).
(b) Sutherland’s Ultimate Display, 1968.
(c) VPL Earphones, 1980s.
(d) Virtuality gaming, 1990s.
(e) Nintendo Virtual Boy, 1995. (f) Oculus
Rift, 2016.
Bringing people together

● We have so far neglected an important aspect, which is human-to-human or social interaction.

● We use formats such as a live theater performance, a classroom, or a lecture hall for a few people
to communicate with or entertain a large audience. We write and read novels to tell stories to
each other.
● Prior to writing, skilled storytellers would propagate experiences to others, including future
generations.
● We have communicated for centuries by writing letters to each other.
● More recent technologies have allowed us to interact directly without delay. The audio part has
been transmitted through telephones for over a century, and now the video part is transmitted as
well through videoconferencing over the Internet.
● At the same time, simple text messaging has become a valuable part of our interaction,
providing yet another example of a preference for decreased realism.
● Communities of online users who interact through text messages over the Internet have been
growing since the 1970s.
● In the context of games, early Multi-User Dungeons (MUDs) grew into Massively Multiplayer
Online Games (MMORPGs) that we have today.
● In the context of education, the PLATO system from the University of Illinois was the first
computer-assisted instruction system, which included message boards, instant messaging, screen
sharing, chat rooms, and emoticons.
● This was a precursor to many community-based, online learning systems, such as the Khan
Academy and Coursera.
Hardware
● The hardware produces stimuli that override the senses of the user.
● The VR hardware accomplishes this by using its own sensors, thereby tracking motions of the
user.
● Head tracking is the most important, but tracking also may include button presses, controller
movements, eye movements, or the movements of any other body parts.
● Finally, it is also important to consider the surrounding physical world as part of the VR system.
● In spite of stimulation provided by the VR hardware, the user will always have other senses that
respond to stimuli from the real world.
● She also has the ability to change her environment through body motions.
● The VR hardware might also track objects other than the user, especially if interaction with them
is part of the VR experience.
● Through a robotic interface, the VR hardware might also change the real world. One example is
teleoperation of a robot through a VR interface.
A third-person perspective of a VR system. It is wrong to assume that the engineered
hardware and software are the complete VR system: The organism and its interaction
with the hardware are equally important. Furthermore, interactions with the surrounding
physical world continue to occur during a VR experience.
Sensors and sense organs
● In engineering, a transducer refers to a device that converts energy from one form to another.
● A sensor is a special transducer that converts the energy it receives into a signal for an electrical
circuit.
● This may be an analog or digital signal, depending on the circuit type.
● A sensor typically has a receptor that collects the energy for conversion.
● Organisms work in a similar way. The “sensor” is called a sense organ, with common examples
being eyes and ears.
● Because our “circuits” are formed from interconnected neurons, the sense organs convert energy
into neural impulses.
● They are measuring the same things and sometimes even function in a similar manner.
● This should not be surprising because we and our engineered devices share the same physical
world:.
● The laws of physics and chemistry remain the same.
 Configuration space of sense organs
● As the user moves through the physical world, his sense organs move along with him.
● Furthermore, some sense organs move relative to the body skeleton, such as our eyes rotating
within their sockets.
● Each sense organ has a configuration space, which corresponds to all possible ways it can be
transformed or configured.
● The most important aspect of this is the number of degrees of freedom or DOFs of the sense
organ.
● Three DOFs correspond to its changing position in space:
● 1) side-to-side motion, 2) vertical motion, and 3) closer-further motion.
● The other three DOFs correspond to possible ways the object could be rotated; in other words,
exactly three independent parameters are needed to specify how the object is oriented. These are
called yaw, pitch, and roll.
● As an example, consider your left ear.
● As you rotate your head or move your body through space, the position of the ear changes, as
well as its orientation.
● This yields six DOFs.
● The same is true for your right eye, but it also capable of rotating independently of the head.
● Keep in mind that our bodies have many more degrees of freedom, which affect the
configuration of our sense organs.
● A tracking system may be necessary to determine the position and orientation of each sense
organ that receives artificial stimuli
Under normal conditions, the brain (and body parts) control the configuration of
sense organs (eyes, ears, fingertips) as they receive natural stimulation from the
surrounding, physical world.
In comparison to previous figure, a VR system “hijacks” each sense by replacing
the natural stimulation with artificial stimulation that is provided by hardware called
a display. Using a computer, a virtual world generator maintains a coherent, virtual
world. Appropriate “views” of this virtual world are rendered to the display.
If done well, the brain is “fooled” into believing that the virtual world is in fact the
surrounding physical world and natural stimulation is resulting from it.
Aural: world-fixed vs. user-fixed
Seven speakers distributed around the room
periphery generate most of the sound, while a
subwoofer (the “1” of the “7”) delivers the lowest
frequency components. The aural displays are
therefore world-fixed.

A listener wearing headphones, as shown in

Figure. In this case, the aural displays are
user-fixed.
What are the key differences?
● In the surround-sound system, the generated sound (or stimulus) is far away from the ears, whereas it is
quite close for the headphones.
● One implication of the difference in distance is that much less power is needed for the headphones to
generate an equivalent perceived loudness level compared with distant speakers.
● Another implication based on distance is the degree of privacy allowed by the wearer of headphones.
● A surround-sound system at high volume levels could generate a visit by angry neighbors.
● Wearing electronics on your head could be uncomfortable over long periods of time, causing a
preference for surround sound over headphones.
● Several people can enjoy the same experience in a surround-sound system (although they cannot all sit
in the optimal location). Using headphones, they would need to split the audio source across their
individual headphones simultaneously.
● They are likely to have different costs, depending on the manufacturing difficulty and available
component technology. At present, headphones are favored by costing much less than a set of
surround-sound speakers (although one can spend a large amount of money on either).
Visual: world-fixed vs. user-fixed

(a) A CAVE VR system developed at Teesside University, UK.

(b) A 90-year-old woman (Rachel Mahassel) wearing the Oculus Rift DK1
headset in 2013.
CAVE (Cave Automatic Virtual Environment) is a virtual reality (VR) environment consisting of a
cube-shaped VR room or a room-scale area in which the walls, floors and ceilings are projection screens.

Key Differences
● Suppose the screen in front of the user’s eyes shows a fixed image in the headset.
If the user rotates his head, then the image will be perceived as being attached to
the head. This would occur, for example, if you rotate your head while using the
Viewmaster.
● If you would like to instead perceive the image as part of a fixed world around you,
then the image inside the headset must change to compensate as you rotate your
head.
● The surrounding virtual world should be counter-rotated,
● Once we agree that such transformations are necessary, it becomes a significant engineering
challenge to estimate the amount of head and eye movement that has occurred and apply the
appropriate transformation in a timely and accurate manner.
● If this is not handled well, then users could have poor or unconvincing experiences. Worse yet,
they could fall prey to VR sickness.
● This is one of the main reasons why the popularity of VR headsets waned in the 1990s.
● The component technology was not good enough yet. Fortunately, the situation is much
improved at present.
● For audio, few seemed to bother with this transformation, but for the visual counterpart, it is
absolutely critical. One final note is that tracking and applying transformations also becomes
necessary in CAVE systems if we want the images on the screens to be altered according to
changes in the eye positions inside of the room
● The hardware components of VR systems are conveniently classified as:
• Displays (output): Devices that each stimulate a sense organ.
• Sensors (input): Devices that extract information from the real world.
• Computers: Devices that process inputs and outputs sequentially.
Displays
A display generates stimuli for a targeted sense organ. Vision is our dominant sense, and any display constructed for the eye must
cause the desired image to be formed on the retina. Because of this importance,

For CAVE systems, some combination of digital projectors and mirrors is used. Due to the plummeting costs, an array of
large-panel displays may alternatively be employed.

For headsets, a smartphone display can be placed close to the eyes and brought into focus using one magnifying lens for each
eye.

Screen manufacturers are currently making custom displays for VR headsets by leveraging the latest LED display technology from
the smartphone industry.

Some are targeting one display per eye with frame rates above 90 Hz and over two megapixels per eye.

Now imagine displays for other sense organs. Sound is displayed to the ears using classic speaker technology. Bone conduction
methods may also be used, which vibrate the skull and propagate the waves to the inner ear; this method appeared Google Glass.

For the sense of touch, there are haptic displays.

Haptic feedback can be given in the form of vibration, pressure, or temperature.

Sensors
● For visual and auditory body-mounted displays, the position and orientation of the sense
organ must be tracked by sensors to appropriately adapt the stimulus.
● The orientation part is usually accomplished by an inertial measurement unit or IMU.
● The main component is a gyroscope, which measures its own rate of rotation; the rate is
referred to as angular velocity and has three components namely precession,nutation and
spin.
● Measurements from the gyroscope are integrated over time to obtain an estimate of the
cumulative change in orientation.
● The error resulting due to wrong measurement, called drift error, would gradually grow
unless other sensors are used.
● To reduce drift error, IMUs also contain an accelerometer and possibly a magnetometer.
● Over the years, IMUs have gone from existing only as large mechanical systems in aircraft
and missiles to being tiny devices inside of smartphones.
● Due to their small size, weight, and cost, IMUs can be easily embedded in wearable devices.
● They are one of the most important enabling technologies for the current generation of VR headsets and
are mainly used for tracking the user’s head orientation.
● Digital cameras provide another critical source of information for tracking systems.
● Like IMUs, they have become increasingly cheap and portable due to the smartphone industry, while at
the same time improving in image quality.
● Cameras enable tracking approaches that exploit line-of-sight visibility.
● The idea is to identify features or markers in the image that serve as reference points for an moving object
or a stationary background.
● Such visibility constraints severely limit the possible object positions and orientations.
● Standard cameras passively form an image by focusing the light through an optical system, much like the
human eye.
● Once the camera calibration parameters are known, an observed marker is known to lie along a ray in
space.
● Cameras are commonly used to track eyes, heads, hands, entire human bodies, and any other objects in the
physical world.
● One of the main challenges at present is to obtain reliable and accurate performance without placing
special markers on the user or objects around the scene.
● As opposed to standard cameras, depth cameras work actively by
projecting light into the scene and then observing its reflection in the
image.
● This is typically done in the infrared (IR) spectrum so that humans do
not notice.
● In addition to these sensors, we rely heavily on good-old mechanical
switches and potentiometers to create keyboards and game controllers.
● An optical mouse is also commonly used.
● One advantage of these familiar devices is that users can rapidly input
data or control their characters by leveraging their existing training.
● A disadvantage is that they might be hard to find or interact with if their
faces are covered by a headset.
Computers
A computer executes the virtual world generator (VWG).

Although unimportant for world-fixed displays, the location is crucial for body-fixed displays.

If a separate PC is needed to power the system, then fast, reliable communication must be
provided between the headset and the PC.

This connection is currently made by wires, leading to an awkward tether; current wireless
speeds are not sufficient.

As you have noticed, most of the needed sensors exist on a smartphone, as well as a
moderately powerful computer.

Therefore, a smartphone can be dropped into a case with lenses to provide a VR experience
with little added costs .
● The limitation, though, is that the VWG must be simpler than in the case of a separate PC so that it
runs on less-powerful computing hardware.
● In the near future, we expect to see wireless, all-in-one headsets that contain all of the essential
parts of smartphones for delivering VR experiences.
● These will eliminate unnecessary components of smartphones (such as the additional case), and
will instead have customized optics, microchips, and sensors for VR.
● In addition to the main computing systems, specialized computing hardware may be utilized.
● Graphical processing units (GPUs) have been optimized for quickly rendering graphics to a screen
and they are currently being adapted to handle the specific performance demands of VR.
● Also, a display interface chip converts an input video into display commands.
● Finally, microcontrollers are frequently used to gather information from sensing devices and send
them to the main computer using standard protocols, such as USB.
Figure shows the hardware components for the
Oculus Rift DK2, which became available in late
2014. In the lower left corner, you can see a
smartphone screen that serves as the display. Above
that is a circuit board that contains the IMU, display
interface chip, a USB driver chip, a set of chips for
driving LEDs on the headset for tracking, and a
programmable microcontroller. The lenses, shown in
the lower right, are placed so that the smartphone
screen appears to be “infinitely far” away, but
nevertheless fills most of the field of view of the
user. The upper right shows flexible circuits that
deliver power to IR LEDs embedded in the headset
(they are hidden behind IRtransparent plastic). A
camera is used for tracking, and its parts are shown
in the center.
Software
● The VWG receives inputs from low-level systems that indicate what the user is doing in the real world.
● A head tracker provides timely estimates of the user’s head position and orientation.
● Keyboard, mouse, and game controller events arrive in a queue that are ready to be processed.
● The key role of the VWG is to maintain enough of an internal “reality” so that renderers can
extract the information they need to calculate outputs for their displays.

The Virtual World Generator

(VWG) maintains another world,
which could be synthetic, real, or
some combination. From a
computational perspective, the
inputs are received from the user
and his surroundings, and
appropriate views of the world are
rendered to displays.
Virtual world: real vs. synthetic
● At one extreme, the virtual world could be completely synthetic.
● In this case, numerous triangles are defined in a 3D space, along with material properties that indicate how
they interact with light, sound,forces, and so on.
● The field of computer graphics addresses computer-generated images from synthetic models, and it remains
important for VR.
● At the other extreme, the virtual world might be a recorded physical world that was captured using modern
cameras, computer vision, and Simultaneous Localization and Mapping (SLAM) techniques. Many
possibilities exist between the extremes.
● For example, camera images may be taken of a real object, and then mapped onto a synthetic object in the
virtual world.

Using both color and depth information from

cameras, a 3D model of the world can be extracted
automatically using Simultaneous Localization and
Mapping (SLAM) techniques.
Matched Motion
● The most basic operation of the VWG is to maintain a correspondence between user motions in the real world
and the virtual world.
● In the real world, the user’s motions are confined to a safe region, which we will call the matched zone.
● Imagine the matched zone as a place where the real and virtual worlds perfectly align. One of the greatest
challenges is the mismatch of obstacles.
● What if the user is blocked in the virtual world but not in the real world? The reverse is also possible.
● In a seated experience, the user sits in a chair while wearing a headset. The matched zone in this case is a small
region, such as one cubic meter, in which users can move their heads. Head motions should be matched
between the two worlds.
● If the user is not constrained to a seat, then the matched zone could be an entire room or an outdoor field.
● Note that safety becomes an issue because the user might spill a drink, hit walls, or fall into pits that exist only
in the real world, but are not visible in the virtual world. Larger matched zones tend to lead to greater safety
issues. Users must make sure that the matched zone is cleared of dangers in the real world, or the developer
should make them visible in the virtual world.
● Which motions from the real world should be reflected in the virtual world? This varies among VR
experiences.
● In a VR headset that displays images to the eyes, head motions must be matched so that the visual renderer uses
the correct viewpoint in the virtual world.
● Other parts of the body are less critical, but may become important if the user needs to perform hand-eye
coordination or looks at other parts of her body and expects them to move naturally.
A matched zone is maintained
between the user in their real world
and his representation in the virtual
world.
The matched zone could be moved in
the virtual world by using an interface
such as a game controller, while the
user does not correspondingly move
in the real world.
User Locomotion
In many VR experiences, users want to move well outside of the matched zone. This motivates locomotion,
which means moving oneself in the virtual world, while this motion is not matched in the real world.
Imagine you want to explore a virtual city while remaining seated in the real world. How should this be
achieved?
You could pull up a map and point to where you want to go, with a quick teleportation operation sending you
to the destination.
A popular option is to move oneself in the virtual world by operating a game controller, mouse, or keyboard.
By pressing buttons or moving knobs, yourself in the virtual world could be walking, running, jumping,
swimming, flying, and so on.
You could also climb aboard a vehicle in the virtual world and operate its controls to move yourself. These
operations are certainly convenient, but often lead to sickness because of a mismatch between your balance
and visual senses.
Physics
● The VWG handles the geometric aspects of motion by applying the appropriate mathematical
transformations.
● In addition, the VWG usually implements some physics so that as time progresses, the virtual world behaves
like the real world.
● In most cases, the basic laws of mechanics should govern how objects move in the virtual world. For
example, if you drop an object, then it should accelerate to the ground due to gravitational force acting on it.
● One important component is a collision detection algorithm, which determines whether two or more bodies
are intersecting in the virtual world.
● If a new collision occurs, then an appropriate response is needed.
● For example, suppose the user pokes his head through a wall in the virtual world. Should the head in the
virtual world be stopped, even though it continues to move in the real world? To make it more complex, what
should happen if you unload a dump truck full of basketballs into a busy street in the virtual world?
Simulated physics can become quite challenging, and is a discipline in itself. There is no limit to the
[Link] addition to handling the motions of moving objects, the physics must also take into account
how potential stimuli for the displays are created and propagate through the virtual world.
Networked experiences
● In the case of a networked VR experience, a shared virtual world is maintained by a
server.
● Each user has a distinct matched zone.
● Their matched zones might overlap in a real world, but one must then be careful so
that they avoid unwanted collisions.
● Most often, these zones are disjoint and distributed around the Earth. Within the
virtual world, user interactions, including collisions, must be managed by the VWG.
● If multiple users are interacting in a social setting, then the burdens of matched
motions may increase.
● As users meet each other, they could expect to see eye motions, facial expressions,
and body language;
Developer choices for VWGs

● A developer could start with a basic Software Development Kit (SDK) from a VR headset
vendor and then build her own VWG from scratch.
● The SDK should provide the basic drivers and an interface to access tracking data and
make calls to the graphical rendering libraries.
● In this case, the developer must build the physics of the virtual world 56 S. M. LaValle:
Virtual Reality from scratch, handling problems such as avatar movement, collision
detection, lighting models, and audio.
● This gives the developer the greatest amount of control and ability to optimize performance;
however, it may come in exchange for a difficult implementation burden.
● In some special cases, it might not be too difficult. For example, in the case of the Google
Street view , the “physics” is simple: The viewing location needs to jump between
panoramic images in a comfortable way while maintaining a sense of location on the Earth.
● In the case of telepresence using a robot, the VWG would have to take into account movements in
the physical world.
● Failure to handle collision detection could result in a broken robot (or human!). At the other extreme,
a developer may use a ready-made VWG that is customized to make a particular VR experience by
choosing menu options and writing high-level scripts.
● Examples available today are OpenSimulator, Vizard by WorldViz, Unity 3D, and Unreal Engine by
Epic Games. The latter two are game engines that were adapted to work for VR, and are by far the
most popular among current VR developers.
● The first one, OpenSimulator, was designed as an open-source alternative to Second Life for
building a virtual society of avatars.
● As already stated, using such higher-level engines make it easy for developers to make a VR
experience in little time; however, the drawback is that it is harder to make highly original
experiences that were not imagined by the engine builders.
Human Physiology and Perception
● Our bodies were not designed for VR.
● By applying artificial stimulation to the senses, we are disrupting the operation
of biological mechanisms that have taken hundreds of millions of years to
evolve in a natural environment.
● We are also providing input to the brain that is not exactly consistent with all
of our other life experiences.
● In some instances, our bodies may adapt to the new stimuli. This could cause
us to become unaware of flaws in the VR system.
● In other cases, we might develop heightened awareness or the ability to
interpret 3D scenes that were once difficult or ambiguous.
● Unfortunately, there are also many cases where our bodies react by increased fatigue or
headaches, partly because the brain is working harder than usual to interpret the stimuli.
● Finally, the worst case is the onset of VR sickness, which typically involves symptoms of
dizziness and nausea.
● Perceptual psychology is the science of understanding how the brain converts sensory
stimulation into perceived phenomena.
● Our body is not designed for VR hence we should keep in mind about

1) basic physiology of the human body, including sense organs and neural pathways,

2) the key theories and insights of experimental perceptual psychology, and

3) the interference of the engineered VR system with our common perceptual

processes and the resulting implications or side effects.
● The perceptual side of VR often attracts far too little attention among developers.
● In the real world, perceptual processes are mostly invisible to us.
● When you see someone you know well, the process starts automatically, finishes
immediately, and seems to require no effort.
● Scientists have conducted experiments that reveal how much work actually occurs in
this and other perceptual processes.
● Through brain lesion studies, they are able to see the effects when a small part of the
brain is not functioning correctly.
● Some people suffer from prosopagnosia, which makes them unable to recognize the
faces of familiar people, including themselves in a mirror, even though nearly
everything else functions normally.
● Scientists are also able to perform single-unit recordings, mostly on animals, which
reveal the firings of a single neuron in response to sensory stimuli. Imagine, for
example, a single neuron that fires whenever you see a sphere.
Optical illusions

● One of the most popular ways to appreciate the complexity of our perceptual
processing is to view optical illusions.
● These yield surprising results and are completely unobtrusive.
● Each one is designed to reveal some shortcoming of our visual system by providing a
stimulus that is not quite consistent with ordinary stimuli in our everyday lives.
Optical illusions present an unusual stimulus that
highlights limitations of our vision system. (a)
The Ponzo illusion causes the upper line
segment to appear larger than the lower one,
even though they are the same length. (b) The
checker shadow illusion causes the B tile to
appear lighter than the A tile, even though they
are the exactly the same shade of gray (figure by
Adrian Pingstone)
 Classification of senses
● Perception and illusions are not limited to our eyes.
● In the human body stimulus is converted into neural impulses.
● For each sense, the below table indicates the type of energy for the stimulus and the receptor that converts the
stimulus into neural impulses.
● Each receptor as a sensor that targets a particular kind of stimulus. This is referred to as sensory system
selectivity.
● This is referred to as sensory system selectivity. In each eye, over 100 million photoreceptors target
electromagnetic energy precisely in the frequency range of visible light.
● Different kinds even target various colors and light levels.
● The auditory, touch, and balance senses involve motion, vibration, or gravitational force; these are sensed by
mechanoreceptors.
● The sense of touch additionally involves thermoreceptors to detect change in temperature.
● Our balance sense helps us to know which way our head is oriented, including sensing the direction of “up”.
● Finally, our sense of taste and smell is grouped into one category, called the chemical senses, that relies on
chemoreceptors; these provide signals based on chemical composition of matter appearing on our tongue or in
our nasal passages.
● Note that senses have engineering equivalents, most of which appear in VR systems. Imagine you a designing a
humanoid telepresence robot, which you expect to interface with through a VR headset.
● You could then experience life through your surrogate robotic self. Digital cameras would serve as its eyes, and
microphones would be the ears.
● Pressure sensors and thermometers could be installed to give a sense of touch.
● For balance, we can install an IMU. In fact, the human vestibular organs and modern IMUs bear a striking
resemblance in terms of the signals they produce.
● We could even install chemical sensors, such as a pH meter, to measure aspects of chemical composition to
provide taste and smell.
Big brains
● Perception happens after the sense organs convert the stimuli into neural impulses.
● According to latest estimates, human bodies contain around 86 billion neurons. Around 20 billion are
devoted to the part of the brain called the cerebral cortex, which handles perception and many other
high-level functions such as attention, memory, language, and consciousness.
● It is a large sheet of neurons around three millimeters thick and is heavily folded so that it fits into our
skulls.
● In case you are wondering where we lie among other animals, a roundworm, fruit fly, and rat have
302, 100 thousand, and 200 million neurons, respectively. An elephant has over 250 billion neurons,
which is more than us.
● Only mammals have a cerebral cortex. The cerebral cortex of a rat has around 20 million neurons.
● Cats and dogs are at 300 and 160 million, respectively. A gorilla has around 4 billion. A type of dolphin
called the long-finned pilot whale has an estimated 37 billion neurons in its cerebral cortex, making it
roughly twice as many as in the human cerebral cortex; however, scientists claim this does not imply
superior cognitive abilities
● Another important factor in perception and overall cognitive ability is the interconnection
between neurons.
● Imagine an enormous directed graph, with the usual nodes and directed edges. The
nucleus or cell body of each neuron is a node that does some kind of “processing”. Figure
shows a neuron.
● The dendrites are essentially input edges to the neuron, whereas the axons are output
edges.
● Through a network of dendrites, the neuron can aggregate information from numerous
other neurons, which themselves may have aggregated information from others.
● The result is sent to one or more neurons through the axon.
● For a connected axon-dendrite pair, communication occurs in a gap called the synapse,
where electrical or chemical signals are passed along.
● Each neuron in the human brain has on average about 7000 synaptic connections to other
neurons, which results in about 1015 edges in our enormous brain graph
 A typical neuron receives signals
through dendrites, which interface to
other neurons. It outputs a signal to
other neurons through axons.

The stimulus captured by receptors works its way through a hierarchical network of neurons. In the early
stages, signals are combined from multiple receptors and propagated upward. At later stages, information
flows bidirectionally.
Hierarchical processing
● Upon leaving the sense-organ receptors, signals propagate among the neurons to eventually reach the cerebral
cortex.
● Along the way, hierarchical processing is performed.
● Through selectivity, each receptor responds to a narrow range of stimuli, across time, space, frequency, and so
on.
● After passing through several neurons, signals from numerous receptors are simultaneously taken into
account.
● This allows increasingly complex patterns to be detected in the stimulus.
● In the case of vision, feature detectors appear in the early hierarchical stages, enabling us to detect features
such as edges, corners, and motion.
● Once in the cerebral cortex, the signals from sensors are combined with anything else from our life
experiences that may become relevant for making an interpretation of the stimuli.
● Various perceptual phenomena occur, such as recognizing a face or identifying a song. Information or
concepts that appear in the cerebral cortex tend to represent a global picture of the world around us.
● Surprisingly, topographic mapping methods reveal that spatial relationships among receptors are maintained
in some cases among the distribution of neurons.
Proprioception
● In addition to information from senses and memory, we also use proprioception, which is the ability to sense
the relative positions of parts of our bodies and the amount of muscular effort being involved in moving them.
● Close your eyes and move your arms around in an open area. You should have an idea of where your arms are
located, although you might not be able to precisely reach out and touch your fingertips together without using
your eyes.
● This information is so important to our brains that the motor cortex, which controls body motion, sends signals
called efference copies to other parts of the brain to communicate what motions have been executed.
● Proprioception is effectively another kind of sense.
● Continuing our comparison with robots, it corresponds to having encoders on joints or wheels, to indicate how
far they have moved.
● One interesting implication of proprioception is that you cannot tickle yourself because you know where your
fingers are moving; however, if someone else tickles you, then you do not have access to their efference
copies.
● The lack of this information is crucial to the tickling sensation.
Fusion of senses
● Signals from multiple senses and proprioception are being processed and combined with our experiences by
our neural structures throughout our lives.
● In ordinary life, without VR or drugs, our brains interpret these combinations of inputs in coherent, consistent,
and familiar ways.
● Any attempt to interfere with these operations is likely to cause a mismatch among the data from our senses.
The brain may react in a variety of ways.
● It could be the case that we are not consciously aware of the conflict, but we may become fatigued or develop
a headache.
● Even worse, we could develop symptoms of dizziness or nausea. In other cases, the brain might react by
making us so consciously aware of the conflict that we immediately understand that the experience is artificial.
● This would correspond to a case in which the VR experience is failing to convince people that they are present
in a virtual world. To make an effective and comfortable VR experience, trials with human subjects are
essential to understand how the brain reacts. It is practically impossible to predict what would happen in an
unknown scenario, unless it is almost identical to other well-studied scenarios.
● One of the most important examples of bad sensory conflict in the context of VR is vection, which is the
illusion of self motion.
● The conflict arises when your vision sense reports to your brain that you are accelerating, but your balance
sense reports that you are motionless.
● As people walk down the street, their balance and vision senses are in harmony. You might have experienced
vection before, even without VR.
● If you are stuck in traffic or stopped at a train station, you might have felt as if you are moving backwards
while seeing a vehicle in your periphery that is moving forward. In the 1890s, Amariah Lake constructed an
amusement park ride that consisted of a swing that remains at rest while the entire room surrounding the swing
rocks back-and-forth.
● For example, if you accelerate yourself forward using a controller, rather than moving forward in the real
world, then you perceive acceleration with your eyes, but not your vestibular organ.
Adaptation
● A universal feature of our sensory systems is adaptation, which means that the perceived
effect of stimuli changes over time.
● This may happen with any of our senses and over a wide spectrum of time intervals.
● For example, the perceived loudness of motor noise in an aircraft or car decreases within
minutes.
● In the case of vision, the optical system of our eyes and the photoreceptor sensitivities
adapt to change perceived brightness.
● Over long periods of time, perceptual training can lead to adaptation.
● In military training simulations, sickness experienced by soldiers appears to be less than
expected, perhaps due to regular exposure.
● Anecdotally, the same seems to be true of experienced video game players.
● Those who have spent many hours and days in front of large screens playing first-person
shooter games apparently experience less vection when locomoting themselves in VR.
● Adaptation therefore becomes a crucial factor for VR. Through repeated exposure, developers
may become comfortable with an experience that is nauseating to a newcomer.
● This gives them a terrible bias while developing an experience the problem of confusing the
scientist with the lab subject in the VR experiment. On the other hand, through repeated, targeted
training developers may be able to improve their debugging skills by noticing flaws in the system
that an “untrained eye” would easily miss. Common examples include
● A large amount of tracking latency has appeared, which interferes with the perception of
stationarity.
● The left and right eye views are swapped.
● Objects appear to one eye but not the other.
● One eye view has significantly more latency than the other.
● Straight lines are slightly curved due to uncorrected warping in the optical system. This
disconnect between the actual stimulus and one’s perception of the stimulus leads to the next
topic.
Psychophysics
● Psychophysics is the scientific study of perceptual phenomena that are produced by physical stimuli.
● For example, under what conditions would someone call an object “red”? The stimulus corresponds
to light entering the eye, and the perceptual phenomenon is the concept of “red” forming in the brain.
● Other examples of perceptual phenomena are “straight”, “larger”, “louder”, “tickles”, and “sour”.
● Figure shows a typical scenario in a psychophysical experiment. As one parameter is varied, such
as the frequency of a light, there is usually a range of values for which subjects cannot reliably
classify the phenomenon.
● For example, there may be a region where they are not sure whether the light is red. At one
extreme, they may consistently classify it as “red” and at the other extreme, they consistently classify
it as “not red”.
● For the region in between, the probability of detection is recorded, which corresponds to the
frequency with which it is classified as “red”.
The most basic psychometric
function. For this example, as the
stimulus intensity is increased, the
percentage of people detecting the
phenomenon increases. The point
along the curve that corresponds to 50
percent indicates a critical threshold
or boundary in the stimulus intensity.
The curve above corresponds to the
cumulative distribution function of
the error model (often assumed to be
Gaussian).
● Stevens’ power law One of the most known results from psychophysics is Steven’s
power law, which characterizes the relationship between the magnitude of a physical
stimulus and its perceived magnitude.
● The hypothesis is that an exponential relationship occurs over a wide range of
sensory systems and stimuli: p = cmx in which
○ • m is the magnitude or intensity of the stimulus,
○ • p is the perceived magnitude,
○ • x relates the actual magnitude to the perceived magnitude, and is the most important part of the
equation, and • c is an uninteresting constant that depends on units.
● Note that for x = 1, is a linear relationship, p = cm; An example of this is our
perception of the length of an isolated line segment directly in front of our eyes.
● The length we perceive is proportional to its actual length. The more interesting
cases are when x 6= 1.
● For the case of perceiving the brightness of a target in the dark, x = 0.33, which
implies that a large increase in brightness is perceived as a smaller increase.
● In the other direction, our perception of electric shock as current through the fingers
yields x = 3.5. A little more shock is a lot more uncomfortable!
Steven’s power law captures the
relationship between the
magnitude of a stimulus and its
perceived magnitude. The model
is an exponential curve, and the
exponent depends on the stimulus
type.