Imaging of the Liver and Intra-hepatic Biliary Tract: Volume

1: Imaging Techniques and Non-tumoral Pathologies Emilio

Quaia
[Link]
hepatic-biliary-tract-volume-1-imaging-techniques-and-non-tumoral-
pathologies-emilio-quaia/

Or type in your browser (Write

continuously without a period.):

textbookfull . com/?p=84925

LAST UPDATED: MAR 2026

Scan to Download

Imaging of the Liver and Intra-

hepatic Biliary Tract: Volume 1:
Imaging Techniques and Non-tumoral
Pathologies Emilio Quaia

2026 Next Generation Release

★★★★★ 4.8/5.0 - 12 downloads

'A reliable PDF learning resource.' - Irene M.

PDF Available Immediately

Medical Radiology · Diagnostic Imaging
Series Editors: Hans-Ulrich Kauczor · Paul M. Parizel · Wilfred C. G. Peh

Emilio Quaia Editor

Imaging
of the Liver and
Intra-hepatic
Biliary Tract
Volume 1: Imaging Techniques and Non-tumoral
Pathologies
Medical Radiology
Diagnostic Imaging

Series Editors
Hans-Ulrich Kauczor
Paul M. Parizel
Wilfred C. G. Peh

For further volumes:

[Link]
Emilio Quaia
Editor

Imaging of the Liver

and Intra-hepatic Biliary
Tract
Volume 1: Imaging Techniques
and Non-tumoral Pathologies
Editor
Emilio Quaia
Radiology Unit, Department of Medicine - DIMED
University of Padova
Padova, Italy

ISSN 0942-5373     ISSN 2197-4187 (electronic)

Medical Radiology
ISBN 978-3-030-38982-6    ISBN 978-3-030-38983-3 (eBook)
[Link]

© Springer Nature Switzerland AG 2021

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or
part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way,
and transmission or information storage and retrieval, electronic adaptation, computer software,
or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in
this book are believed to be true and accurate at the date of publication. Neither the publisher nor
the authors or the editors give a warranty, expressed or implied, with respect to the material
contained herein or for any errors or omissions that may have been made. The publisher remains
neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To my beloved Luigi
Explore our comprehensive library of test banks and solution
manuals at [Link] and take advantage of
current promotions.
Contents

Part I Embriology and Normal Radiologic Anatomy


Embryology and Development of the Liver������������������������������������������   3
Lorenzo Ugo and Emilio Quaia
Liver Anatomy������������������������������������������������������������������������������������������ 15
Lorenzo Ugo, Silvia Brocco, Arcangelo Merola,
Claudia Mescoli, and Emilio Quaia

Part II Imaging and Interventional Modalities

Ultrasound of the Liver �������������������������������������������������������������������������� 51

Vasileios Rafailidis and Paul S. Sidhu

Computed Tomography of the Liver������������������������������������������������������ 77
Domenico De Santis, Federica Landolfi, Marta Zerunian,
Damiano Caruso, and Andrea Laghi

Magnetic Resonance Imaging of the Liver:
Technical Considerations������������������������������������������������������������������������ 99
António Pedro Pissarra, Raquel Madaleno,
Manuela França, and Filipe Caseiro-Alves

Nuclear Medicine of Hepatobiliary System
(SPECT and PET)������������������������������������������������������������������������������������ 113
Pietro Zucchetta and Diego Cecchin
Liver Biopsy���������������������������������������������������������������������������������������������� 119
Valentina Bernardinello, Silvia Ceccato, Antonio Giangregorio,
Serena Magnaguagno, Filippo Crimí, and Emilio Quaia

Hepatic Angiography and Vascular Interventional Radiology������������ 133
Alessandro Pauro, Amalia Lupi, Chiara Mattolin,
Mirko Lazzarin, and Emilio Quaia

Intrahepatic Biliary Tract Interventional Radiology���������������������������� 175
Alessandro Rago, Francesca Zavan, Sofia Moschi,
Paolo De Vincentis, Filippo Crimí, and Emilio Quaia

vii
viii Contents

Part III Hepatic and Biliary Tract Non-Tumoral Pathology

Congenital and Development Disorders of the Liver��������������������������� 199

Anna Florio, Lorenzo Ugo, Filippo Crimí, and Emilio Quaia
Fibropolycystic Liver Diseases���������������������������������������������������������������� 225
Carlos Bilreiro and Inês Santiago
Inflammatory Liver Diseases������������������������������������������������������������������ 241
İlkay S. İdilman, Deniz Akata, and Muşturay Karçaaltıncaba

Liver Steatosis and NAFLD�������������������������������������������������������������������� 251
Manuela França and João Mota Louro

Liver Increased Iron Deposition and Storage Diseases������������������������ 267
Manuela França and João Pinheiro Amorim

Chronic Hepatitis and Liver Fibrosis/Cirrhosis����������������������������������� 281
Sara Mehrabi, Enza Genco, Mariacristina Maturi,
and Mirko D’Onofrio
 ascular Liver Diseases �������������������������������������������������������������������������� 295
V
Manuela França and Joana Pinto
Imaging of Liver Trauma������������������������������������������������������������������������ 323
Stefania Romano, Tomas Andrasina, Hana Petrasova,
Luca Tarotto, Gianluca Gatta, Alfonso Ragozzino,
and Vlastimil Valek
Non-tumoral Pathology of the Intrahepatic
Biliary Tract���������������������������������������������������������������������������������������������� 337
Jelena Kovač
Abbreviations

AASLD American Association for the Study of Liver Diseases

AAST American Association for Surgery and Trauma
AATD Alpha-1 antitrypsin deficiency
ABC Acute bacterial cholangitis
ABCR α-Fetoprotein Barcelona Clinic Liver Cancer Child-Pugh
and Response (score)
ACT Adoptive cell transfer
ADC Apparent diffusion coefficient
ADPKD Autosomal dominant polycystic kidney disease
ADPLD Autosomal dominant polycystic liver disease
AE Adverse event
AFP Alpha-fetoprotein
AGA American Gastroenterology Association
AGS Alagille syndrome
AI Artificial intelligence
AIF Arterial input function
AIH Autoimmune hepatitis
AIOM Italian Association of Medical Oncology
AISF Italian Association for the Study of the Liver
AKI Acute kidney injury
AML Angiomyolipoma
AP Arterial Phase
APASL Asian Pacific Association for the Study of the Liver
APHE Arterial phase hyperenhancement
APS Arterioportal shunt
APTT Activated partial thromboplastin time
ARFI Acoustic radiation force impulse
ARPKD Autosomal recessive polycystic kidney disease
ART Assessment for Retreatment with TACE (score)
ASQ Acoustic structure quantification
AST Serum aspartate aminotransferase
AUC Area Under the Curve
AVM Arteriovenous malformation
BA Biliary atresia
BASM Biliary atresia splenic malformation
BCLC Barcellona Clinic Liver Cancer
BCS Budd–Chiari syndrome

ix
x Abbreviations

BD Bile duct
β-HCA β-Catenin-mutated type with upregulation of glutamine
synthetase hepatocellular adenoma
BF Blood flow
BH Biliary hamartomas
BMD Basis material decomposition
BMI Body mass index
BRCLM Breast cancer liver metastases
BSA Body surface area
BSC Best supportive care
BV Blood volume
CAP Controlled attenuation parameter
CBCTA Cone-beam computed tomographic angiography
CCC (or CCA) Cholangiocarcinoma
CDDP Cisplatin
CDUS Color Doppler ultrasound
CE Contrast enhancement
CECT
(or CE-CT) Contrast-enhanced CT
CEMRI
(or CE-MRI) Contrast-enhanced MRI
CEUS Contrast-enhanced ultrasound
CHA Common hepatic artery
cHC Mixed or combined (or biphenotypic) hepatocellular carci-
noma and cholangiocarcinoma
CHD Common hepatic duct
CHF Congenital hepatic fibrosis
CKD-EPI Chronic kidney disease epidemiology collaboration
CM Contrast medium
CMV Cytomegalovirus
COACH Cerebellar vermis hypo/aplasia, oligophrenia, ataxia con-
genital, coloboma, and hepatic fibrosis syndrome
CP Child-Pugh
CPSS Congenital portosystemic shunts
CR Complete response
CRA Cryoblation
CRC Colorectal cancer
CRLM Colorectal cancer liver metastases
CRP C-reactive protein
CS Caroli syndrome
CSPH Clinically significant portal hypertension
CT Computed tomography
cTACE Conventional TACE
CTAP CT arterial portography
CTHA CT hepatic arteriography
CTP CT perfusion
CTPI CT perfusion index
CV Central vein of the hepatic lobule
Abbreviations xi

DCE Dynamic contrast enhanced

DCE-US Dynamic contrast-enhanced ultrasound
DEBIRI Drug-eluting beads irinotecan-loaded
DEB-TACE Drug-eluting bead TACE
DECT Dual-energy CT
DILI Drug-induced liver injury
DISIDA Disofenin
DKI Diffusion kurtosis imaging
dlDECT Dual-layer detector DECT
DN Dysplastic nodule
DP Delayed phase
DSA Digital subtraction angiography
dsDECT Dual-Source dual-energy CT
DWI
(or DW imaging) Diffusion-weighted imaging sequence (MRI)
EAP Early arterial phase
EASL European Association for the Study of the Liver
ECCM Extracellular contrast media
ECOG Eastern Cooperative Oncology Group
EDV End diastolic velocity
EES Extravascular-extracellular space
EF Enhancing fraction
EFSUMB European Federation of Societies for Ultrasound in
Medicine and Biology
EGDS Esophago-gastro-duodenoscopy
eGFR Estimated glomerular filtration rate
EHD Extrahepatic disease
EHE Epithelioid hemangioendothelioma
EHPVO Extrahepatic portal vein obstruction
EHS Extrahepatic spread
EIS Endoscopic injection sclerotherapy
EMH Extramedullary hematopoiesis
ENETS European Neuroendocrine Tumour Society
EOB-DTPA Ethoxybenzyl diethylenetriamine pentaacetic acid (gadox-
etic acid)
EORTC European Organization for Research and Treatment of
Cancer
EPSVS Extrahepatic portosystemic venous shunt
e-PTFE Expanded polytetrafluoroethylene
ERCP Endoscopic retrograde cholangiopancreatography
ESGAR European Society of Gastrointestinal and Abdominal
Radiology
ESMO European Society for Medical Oncology
EUS Endoscopic ultrasound
EVL Endoscopic variceal band ligation
FAST Focused assessment with sonography in trauma
FBP Filtered back projection
FDG-PET Fluorodeoxyglucose-positron emission tomography
Discovering Diverse Content Through
Random Scribd Documents
Table of Contents
One to Wake
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
About the Author
 One to Wake
On the night of Nuawa’s execution, she saw the Winter Queen for the
first time.
The wind was inert and the night ignited by frostworks, teeth of ice
biting flowers into the sky. The soldiers had given her something and it
made her world both heavy and light, her thoughts dragging behind her like
a train. She was thinking not of the ghost-kiln before her—though even at
six, she knew what it was and what it did—or even of her mother holding
her hand. Instead she was wondering how it was that the queen could walk
so evenly and stride so fast in her armor. Frost and iron, the coronet more
helm than crown, the sword at her hip as broad as Nuawa. It seemed
impossible for a single person to move under so much weight, shoulder so
much heft. She imagined then that the queen was iron and rime underneath
too, dense strong bones inside the annihilating white of her skin.
A soldier lowered Nuawa into the ghost-kiln’s petal mouth. It clenched
shut with a small hiss. Later she would try to recall whether the soldier
flinched as they did this, whether it mattered to them that she was a child—
small for her age—but she would not remember.
Inside the stomach-chamber the air was thick with the smell of dying,
the odor of bodies that had already succumbed. Her giving-mother held
Nuawa, pressing something sharp into her mouth, whispering that she
would live. Nuawa made an obedient noise and swallowed, like a good
daughter. It cut going down and filled her mouth with blood, sweet and
staccato, though it didn’t hurt. Most of her seemed asleep, swaddled in a
warm and distant place while here her bare feet turned numb. She lay
against her mother’s breast and dreamed of a painted blue sky.
After a time, her mother’s arms fell limp. The world turned inside
Nuawa and the dark scraped at her ankles, her wrists, the base of her skull.
It seeped into her mouth and her nose until she felt feverish with secrets.
She was a fish, swimming through dark waters. She was a bird, fluttering up
a sky she’d only seen as oil on canvas.
When the machine’s belly opened, it was still night, or perhaps night
again. Nuawa’s sense of time had slipped like water on stone. Weak as she
was, she looked to make sure, but the queen was no longer about.
People were drawing blanched, drained bodies out of the machine.
They pulled. They pried. They wept and carried away their dead. Nuawa
waited for her turn without sound or protest. She did not cry for help. Her
giving-mother had told her to be quiet until her bearing-mother came.
And she did come, her bearing-mother Indrahi. They were the last. A
few other bodies remained, kinless on the gray, frozen ground.
“Are we not taking her home?” Nuawa asked through cracked lips as
her bearing-mother lifted her from her giving-mother’s breast.
“That is not her. She is gone. From now on you are to understand that
she was never your kin.” In the cold, her bearing-mother’s expression was
stone, was ice. She tipped hot ember wine into Nuawa’s mouth. “Are you
afraid?”
Nuawa considered her fears: freezing to death, falling into the canals
where the ice thinned and cracked under her weight, being a disappointment
in her arithmetic. But already she was forgetting everything from before her
passage through the ghost-kiln, as though it’d purged her, as though it’d
remade her anew. Something was gone. Something else was gained. At last
she said, “No, I don’t think so, Mother.” It was strange, to have only one
now. One parent, one mother.
Mother took Nuawa’s hand. “Good. For your mind is a weapon,
Nuawa, and we shall nurture it in the absence of fear. One day you will fire
all that you are, like a bullet, into the heart of the Winter Queen.”
 Chapter 1
The season’s last match brings with it a press of audience, the mass and
noise of them audible even in the preparation vestibule where silence is
meant to be the final word. There’s nothing for it, Nuawa supposes, as she
tightens the seals on her armor and checks her gun one last time. Everything
is oiled, ready.
The gladiator’s bell rings. The arena gate lifts slowly, a hum of
blindfolds and lion helms, a susurrus of tiger tails and specters. She knows
the mechanisms are lubricated well, the ghosts fed a rich diet of incense and
candlewicks, but the tournament masters like their theatrics.
She steps into a dome of obsidian glass and agate tiles. It is opaque
from inside, transparent from the outside. If she falls, they will hear every
last noise: the rattle of her final breath and the wet slap of viscera meeting
glass, while she will never see their faces. Their rapt faces, empty-eyed,
mesmerized by spectacle. So it goes.
The opposite gate unfurls, dove wings and mandarin petals. For half a
moment, she sees nothing at all, then discerns the solid outlines of the
muzzles, the light-drinking coat, the sleek knotted limbs. They have sent her
leopards to fight.
She hears the whirr of their articulated legs, the scrape of their curse-
alloyed claws, and knows they are more than animal. Guided by a human
mind, potent with thaumaturgy. She counts: four pairs of jade-dark eyes,
four tails like whips.
An instant’s calculation for angle and trajectory, and she fires. The
leopards are fast, upon her far quicker than any human or natural beast
could be. Her bullet ricochet off the dome, piercing a leopard’s shadow; its
flesh corresponds in a rip of meat, a spray of gore. Her second shot catches
another in the haunch, interrupting it mid-pounce.
 Her drop to the floor is a fraction too late. Claws screech across the
metal of her armor, not penetrating but leaving a slick of concentrated
grudges: pain flashes down her vertebrae, bright turquoise synesthetic
across her vision. Her gauntleted arm is all that keeps her face from being
shredded to cartilage and gore.
She pulls a polynomial from her belt, tearing off its safety with her
teeth. An implosive flash, more light than heat, blinds the puppeteer behind
those feline eyes. Nuawa uses the pause to gain distance, rolling away,
drawing her blade. Her sword’s beaked shadows click and clatter, a spread
of five today: thanks to the lighting, all far longer than the blade itself or her
reach. More than sufficient.
Blade shadows roar as they meet the leopards’. Fur tears; arteries
rupture and tendons snap.
Nuawa beheads the animals, for theater and for good measure. Even
then she half-expects each to get up for a rematch, but apparently they
haven’t been witched to work beyond stopped hearts and spilled brains. A
ground fog of expended power rises, is quick to dissipate. She wonders
what shape the puppeteer is in. Incapacitated, with luck. In agony, she
hopes.
Her gate lifts. There is no announcement of her victory, no applause.
The Marrow is too refined for that.
Back in the vestibule there are attendants waiting, sent by her manager
Tezem. One is moon-dusted, the other with a face painted half white and
half green. Both are slim, male, adolescent: the diametric opposite of
Nuawa’s preferences, Tezem’s idea of a joke. When they offer her purifying
balm and cleansing ointments, she takes the bottles and jars from them. “I’ll
go up for a bath.” Many of Tezem’s duelists enjoy being pampered, with
attendants to scrub their backs and lather their hair, oil their limbs and
perfume their throats. Nuawa prefers to be left well enough alone.
She steps into an elevator; here no sense of drama interferes with
function and so the ghosts are efficient, the ride smooth and fast. From
overhead, a portrait of the queen looks down, the royal coiffure as iridescent
as borealis light. Winter’s visage is everywhere, austere in its gauntness,
alien in its sclera the black of frostbitten flesh. Speculations as to the
queen’s origins run abundant, in euphemism and guarded whispers, and
most say she is from the distant isle of Yatpun: a snow-woman from
permafrost peaks, sick of the mountain gods’ tyranny and determined to be
lord and deity of her fate. But Yatpun has been inaccessible for centuries
behind its event-horizon wall, and if she is indeed from the island nation,
the queen is the sole individual alive certain of that truth.
Nuawa presents her credentials, a chalcedony cube at her wrist, to the
toothed locks that guard the Marrow’s highest, most exclusive floor.
A low humming and a haze of steam. Yifen is coming out of the bath,
toweling her hair dry and unselfconsciously nude. At the sight of Nuawa her
eyes brighten, her mouth widening into a grin, openly voracious. “Nuawa!
I’m just done cleaning up but I don’t think I have gotten all the grime out of
my hair. May I join you?”
Nuawa holds up the purification jars. “Only if you want to risk getting
curses on you. This was a dirty fight.”
“Please, stuff that weak can’t begin to touch me.” The other duelist
widens her eyes, tilting her head, coquettish. “Or have I begun to bore
you?”
In truth, a little. She enjoys Yifen’s company well enough most times,
but while Yifen is a fine lover, her appetites tend to exceed Nuawa’s. Still, a
good source of information. “If I tire of you, I would be tiring of life.” She
disrobes and follows Yifen into the bath. There’s only one other occupant, a
sour-faced, taciturn duelist from abroad; they don’t greet or so much as look
up, attention fixed on the pane displaying an intake audition. Nuawa takes a
look. Duelists at the Marrow go into each match blind, but the auditions
give her some idea of new challenges.
Yifen has opened the jars, spooned out ointment, and warmed it
between her tattoo-protected palms. She is inked everywhere, inscribed for
stamina and luck, for senses beyond the five physical ones; their first time
together was an educational experience for Nuawa. “Scouting for a fling?”
Yifen asks, following Nuawa’s gaze.
“I’d never slight you to your face.” On the pane, Nuawa spots two
duelists she’s fought in other venues. But most of the aspirants are foreign,
a few occidentals whose faces and coloring are only slightly less alien than
the queen’s. “Why so many travelers?”
“You haven’t heard? You must mingle more, visit the commoners once
in a moon.” Yifen lowers her voice as she spreads purifying balm down
Nuawa’s spine and hips. “We’re getting our first tribute game. The winner
gets an officer’s commission and the queen’s general will train them as her
very own protégé. Imagine if one of us gets chosen? The first officer ever
from Sirapirat.”
Despite herself, Nuawa is intrigued. Sirapirat citizens have never been
allowed to enlist in the queen’s army, let alone rise to an officer. They have
to give the queen tribute the same as any other territory, but never in a
game, never with the promise of reward. “How is it going to work?”
“Ah, now you’re more interested in that than in me. How I self-
sabotage.” Yifen makes a moue. “Our rankings will give us no head start, I
fear. All participants will begin equal. The first part will be a survival
course. Ten to thirteen winners out of nearly three hundred applicants will
then proceed to fight in single-combat matches.”
“And?” She inches her legs apart, an invitation. Behind her, the foreign
duelist makes a disgusted noise and climbs out of their bath, then to the
dressing parlor.
“And I can get a roster of the applicants, should you wish to …”
Yifen’s thumbs a warm, oiled line up Nuawa’s thigh. “Get preemptive.”
“Which ex-partner or enemy do you want gotten rid of?”
“Little cynic. What if I just want you to win and bring Sirapirat the
glory we’ve long been denied, even though we produce fighters as fit as any
for the army?”
Nuawa slides into one of the pools, down until the water comes up to
her throat then her face. Her hair flares in naga corona, snake-tendrils
floating in the water. She shuts her eyes, feels the grime of perspiration and
leopard-carried grudge sluice away. When she emerges, she finds Yifen laid
flat against the floor, chin in hands and watching with undisguised interest.
The gaze of a hawk on prey.
Nuawa wipes hot water from her mouth. “Are you not participating,
then, in this tournament of tournaments?”
“Given what’ll happen to the losers? I’d rather not.” Yifen cards her
fingers through Nuawa’s drenched hair, tickling an earlobe. “The soldier’s
life is not for me, besides. Too demanding and I’d look terrible in that
uniform. You, though, have just the right mindset and could go far. Only
don’t forget me when you rise high in the general’s favor, hmm?”
“You overestimate me,” Nuawa murmurs. The pool is churning, curse-
vestiges calling out hungry ghosts from the pipes. She climbs out. “But tell
me more and I’ll do my best.”
***
 Chapter 1 | 3

about in or upon the structure (the live load), and event-driven loads originated from the occurrence of
earthquakes, strong wind, or heavy snow. Structural design aims at providing a structure with sufficient
level of resistance against these loads with minimum cost. Within structural engineering, there are several
technical sub-areas. Some are named according to the type of structure. Some are named according to the
type of load.
Earthquake Engineering. The suddenness of earthquakes and the damage they could cause in a matter
of seconds inspired the study of the nature of earthquakes and the effects they inflict on structures. The
effects of earthquake ground motion create vertical and horizontal forces that change violently within a
short duration. The time-varying nature and the multi-directional nature of the earthquake-induced load
require special design and analysis considerations. The fundamental approach in earthquake engineering
is not to design a structure to withstand any earthquake at all costs but to design a structure that will not
inflict injury to human lives at a reasonable cost.
Wind Engineering. Strong wind caused by a hurricane, a tornado, or a storm creates effects on struc-
tures that are also time-varying and multi-directional. Strong wind around a structure may push against
a surface while creating a partial vacuum behind another structural surface. Unlike earthquakes, which
occur infrequently, especially the damaging ones, strong wind in some areas occurs frequently and so is the
damage it incurs. Design against such wind-related effects is the realm of wind engineering. Here again, the
design approach is to protect human lives with a reasonable cost.
Structural Reliability. The many loads a structure must withstand during its life span are mostly of a
“random” nature, meaning it cannot be defined precisely with respect to its magnitude and time of occur-
rence. So are the resistance provided by the size and material of structural components. Design in the face
of uncertainty requires the application of probability and statistics. Structural reliability is the methodology
applying these mathematical tools to the load-resistance analysis in structural design. It is used in the
development of design codes and specifications that are followed by designers to provide acceptable levels
of safety against all loads.
Fire Engineering. In the event of a fire in a building, the high temperature created by the fire may
cause the building material to lose its strength and eventually fail under the weight of the building. Fire
engineering in the context of structural engineering deals with the effective application of protective materi-
als to the structural components such as steel beams and columns such that sufficient time is provided for
the occupants to escape and the firefighters to arrive. The research in fire engineering provides data to be
incorporated into design and construction codes and specifications.
Bridge Engineering. Some structural engineers specialize in bridge design and construction. Bridge
design can be categorized according to material and bridge type. One unique feature of bridge design is it
is closely integrated with construction. From the bridge foundation to the superstructure, the process of
construction and erection often requires detailed analysis by the design engineers and likely dictates the
designers’ choice of bridge type.
Dam Engineering. The design of dams requires detailed study of the geological characteristics of the site
and the mechanical properties of the foundation before the dam type is selected. For some types of dams,
it is necessary to ensure the dam material is placed in such a manner that seepage of water through and
under the dam body is within acceptable limits. Dam engineers also design all details on how to divert water
during construction and specify maintenance and operations procedures post construction.
4 | Introduction to Civil Engineering

Building Engineering. Structural engineers often become building design specialists because building
design is more frequently in demand than bridge or dam designs, especially in urban centers. Building
engineers also design special buildings such as stadiums and large dome structures.
Forensic Engineering. Forensic engineering refers to the study of causes of an engineering event, usually
a disaster or failure of some kind. In the context of structural engineering, it refers to the investigation
of a structural failure. There are no courses or programs for structural forensic engineering training, but
experienced structural engineers who have investigated past failures are often called upon to investigate a
new event. In case of major disasters, often a team of experts are assembled to study the cause of the disaster
and to make recommendations to prevent future disasters. Even when the cause of disaster is terrorism,
forensic engineering would reveal the weakness in design and provide guidance for future designs. A good
example is the 1995 bombing of the Alfred P. Murrah Federal Building in Oklahoma City. The collapse of
the building was caused by the bombing that destroyed ground-level columns in the front of the building.
Experts recommended that future buildings should have sufficient redundancy in the design of supporting
columns so that the damage of one or more columns would not lead to the collapse of the whole building.
In structural design in the context of civil engineering, there are three construction materials that are
dominant: steel, reinforced concrete, and timber. Each has its own design specifications. Thus, steel structure
design, reinforced-concrete structure design, and timber structure design are three main design disciplines.

1.3 Geotechnical Engineering

Most civil engineering structures are earthbound. They sit on soil and rock ground directly or on constructed
foundations that transfer the load to the soil or rock below. Geotechnical engineering is the technical
specialty that deals with soil and rock as supporting materials for structures. It deals with the various
foundation types that work between the structure and the ground. In addition it deals with the stability
of soil or rock slopes whose failure may cause loss of human lives or damage to property. There are several
technical areas of study that are pertinent to geotechnical engineering.
Engineering Geology. While geology is a basic science that is concerned with macroscopic earth struc-
tures or movements, engineering geology provides geological data pertinent to constructed structures. One
obvious example of the application of engineering geology is the mapping of active seismic faults that are
to be avoided when making plans for human habitat development, roadway construction, or power plant
construction. At a more fundamental level, understanding various geological formation and rock types
provides geotechnical engineers the knowledge necessary in assessing the suitability of a site for human
activities.
Soil Mechanics. Most people would not consider soil as an engineering material, but it is, because most
constructed structures are situated on it by necessity. Without due consideration of soil’s bearing capacity
under various circumstances, a structure built over it may sink, tilt, or outright turn over. Soil mechanics
is a branch of mechanics that studies the mechanical properties of various types of soil and its strength at
different moisture-content levels. It provides the scientific base upon which design formulas and codes are
developed for everyday engineering design practice.
 Chapter 1 | 5

Rock Mechanics. The properties of rock become relevant when it is used as the foundation of a high-rise
building or a large dam. It is also relevant when one examines the stability of the slope of a mountain or a
tunnel. It is also the subject of study for the occurrence of earthquakes.
Foundation Engineering. A foundation is the interface between a superstructure and its supporting
soil. A common type of foundation for single-family homes consists of strip footings placed under load-
bearing basement walls. Another common practice is to use a concrete slab to spread the weight of the
building over the soil underneath. Foundation engineering is the study of different types of foundation
and their proper applications. Depending on the properties of soil at a site, shallow or deep piles may be
deployed. The construction of a bridge over water may require the use of deep caissons on which piers are
constructed. To stabilize an excavated slope, various types of methods may be used including retaining walls
and slope-protection vegetation growth.
Soil Improvement. When a structure must be placed at a site with very weak soil, various techniques
can be used to improve the soil properties. These typically involve the use of replacement material through
excavation or the injection of special material (grouts) into the original soil to change its properties. Another
special technique is to place geo-synthetic fabrics or textiles in horizontal layers to strengthen the soil or
to limit soil’s permeability, which is essential in the design for landfill and hazardous material deposit sites.
Tunnel Engineering. Tunneling through soil or rock is sometimes necessary in the construction of
roadways or special storage spaces. Tunnel engineering deals with the route determination, selection of
tunneling machines, and the analysis and design of the tunnel structure.
Most of the things designed by geotechnical engineers are not as visible as those by structural engineers
because they are underground or under the superstructure above. But, it is safe to state that no civil engi-
neering work can be constructed without the contribution of geotechnical engineers.

1.4 Environmental Engineering

Environmental engineering is the application of engineering means to protect human health and to preserve
the natural environment by managing and developing water, air, and land resources. The application of
environmental engineering relies on the fundamental sciences of chemistry, biology, ecology, and health sci-
ences. Most modern environmental engineering projects are planned and implemented under the auspices
of the Clean Water Act, Safe Drinking Water Act, Clean Air Act, and other federal and state environmental
legislation. Several technical areas in environmental engineering are described below.
Water Treatment and Supply. Before water is consumed, it has to be collected first from either un-
derground or above-ground sources. Therefore, source control is one of the most important tasks of water
supply. Except for a few municipalities where the source water derived from deep aquifers, source water has
to be treated to remove contaminants such as pathogenic bacteria, heavy metals, and pesticide residues. The
process of treatment involves the removal of suspended solids and the use of chemicals or ultraviolet (UV)
radiation to disinfect unwanted organisms so that the effluent water satisfies quality requirements dictated
by the Federal Safe Drinking Water Act. For water used by industrial plants such as paper mills or nuclear
power plants, special treatment is needed and its discharge is regulated.
Wastewater Treatment and Disposal. In a modern municipality, household wastewater is collected
through underground pipelines to a treatment plant. The wastewater treatment process is very different
6 | Introduction to Civil Engineering

from drinking water treatment and is classified into primary treatment, secondary treatment, and tertiary
advanced treatment. Primary treatment removes suspended solids from wastewater by a sedimentation
process. Secondary treatment is to remove dissolved organic wastes from wastewater by biochemical decom-
position followed by further sedimentation. The Federal Clean Water Act establishes nationwide minimum
treatment requirements for all wastewater. For municipal wastewater discharge, the minimum treatment is
the secondary treatment, which removes 85% of biochemical oxygen demand (BOD) and total suspended
solids (TSS). BOD is a measurement of oxygen-demanding organic wastes. In situations in which these
minimum treatment levels are not sufficient, the Clean Water Act requires additional treatment, which is
accomplished by membrane filtration and other physical-chemical processes.
The outcomes of the wastewater treatment are solid sludge and effluent water. The solid sludge sometimes
can be used for landfill or even as fertilizers. The effluent water can be used for irrigation or groundwater
recharge or may be directly discharged into river, stream, or lake or sea. For a municipality, the amount of
rainfall determines the ways of collecting and treating wastewater. Obviously if a large amount of rainfall
is expected, especially when storm water comes in a very short period of time, rainwater runoff should
be separated or diverted either temporarily or permanently from household wastewater in order to avoid
overwhelming the treatment plant. Thus, a wastewater collecting system can either be combined (for more
arid areas) or separated. Some industrial plants produce special wastewater that requires the removal of
heavy metal or hazardous chemicals before being discharged.
Air Pollution. Environmental engineers monitor, analyze, and assess the air quality around municipali-
ties. Air pollution comes from natural and human-activity sources. Volcanic eruption is a major natural
source of air pollution. The gaseous and particulate contents of a volcanic eruption are often studied by
scientists rather than engineers. Around a large municipality, however, air pollutants come from automobile
emissions, nearby industrial plant emission, and even from faraway sources. Health science advances have
discovered that tiny solid particles in the air such as soot are hazardous to human health. The monitor-
ing of these particles is as important as that for gases. Tracing plant emission in the atmosphere, called
plume analysis, is important in the assessment of the environmental impact of a plant. Another form of
air pollution is sand storm. Monitoring of sand storms may lead to the sources of the storm and policy for
conservation or planting of new vegetation.
Solid Waste Disposal. Solid wastes, commonly known as trash and garbage, from domestic, commer-
cial, and industrial sources are to be collected, separated, and partially recycled, and disposed of in landfills
and special disposal sites. Environmental engineers, working with other civil engineers, select, design, and
construct sanitary landfill sites. Water percolating through a sanitary landfill is intercepted, collected, and
treated in order to prevent the seepage of hazardous materials into ground water strata. Some solid waste
maybe burned by specially designed incinerating plants.
Nuclear Waste Disposal. Nuclear waste comes from used fuel rods in nuclear power plants. Though the
degree of radiation from these spent fuel rods is low, long-term exposure to low-level radiation is hazardous
to human health. Disposal of these wastes has few options. The basic approach is to store them in places
far from human habitat. Furthermore, it must be assured that the storage containers will not leak to the
environment in any way. Leakage to underground water would be disastrous because the contaminants can
travel far and reach sources for human water consumption.
Noise Pollution. In modern municipalities, human activities often generate sustained high levels
of sound that are hazardous to the physical and mental wellbeing of habitants. Sound barriers are often
needed to shield neighborhoods from highway traffic noises. Power plants or air-conditioning plants on
 Chapter 1 | 7

large campuses produce high levels of noises that also require containment and shielding. Environmental
engineers monitor the noise levels and design and implement mitigation strategies.
Environmental Impact Assessment. Environmental engineers are often called upon to assess the
impacts on human health and the natural environment by a new development, a new industrial plant, or
even a new commercial establishment such as a large shopping mall. Such assessment may entail the study
of noise, traffic, water consumption and discharge, power requirements, air pollution potential, and other
factors.
Environmental engineering as a part of civil engineering is unique in its extensive applications of
knowledge from health sciences and biology and chemistry. Its practice is also very much impacted by
environmental laws enacted at the state and national levels.

1.5 Water Resources Engineering

Water resources engineering is a specialty dealing with the use of water in support of modern living, includ-
ing the agricultural, industrial, domestic, recreational, and environmental needs. Its scope includes the
finding and preservation of above- and underground water sources, understanding the movement of water
in nature, engineering the transport of water, and managing erosive effects of water wave and current on
shorelines. Some core and related specialties are described below.
Water Resources System Engineering. The understanding of the circulation of water on earth and
managing the sources of water in a region requires a system approach. Decision on the water supply for a
city or a region requires the knowledge of water sources and the quality and quantity of each source. The ap-
plication of system analysis in water resources management and the design and operations of multipurpose
reservoir and river systems is at the core of water resources system engineering.
Hydraulic Engineering. Hydraulic engineers design artificial waterways such as canals, channels, and
aqueducts as well as manage water movement by designing and constructing dams, levees, canal locks, and
other water-regulating devices. For many regions a major task of hydraulic engineers is flood prevention
and control, which entails the assessment of potential rainfall quantity, prediction of water levels along
natural rivers, streams, or channels, and strategies to mitigate flooding hazards by improving the natural
topography. Hydraulic engineering is also fundamental to hydraulic-power generation. In hydraulic-power
generation a prerequisite is a high water-level differential (water head). When water moves from a high level
to a lower level, the difference in the water levels provides the energy potential for power generation. Some
dams are constructed mainly for power generation although usually a dam also has the potential to be used
for flood control. The stored water in a dam’s reservoir can be used for agricultural, industrial, and domestic
consumption as well as recreational sports.
Coastal Engineering. The movement of water in oceans and lakes has erosive effects on their shorelines.
The preservation of wetland for flood mitigation or marine ecology requires the knowledge of such effects.
Use of artificial barriers such as breakwaters or dikes at a shore or a harbor can result in reducing the water
wave level within protected areas, eliminate or reduce the effects of shoreline erosion, and redirect natural
sediment so that new land can be created over time.
Ocean Engineering. Ocean engineering deals with the effects of ocean currents and waves on ocean-
bound structures and the analysis and design of such structures to withstand the wave forces. The most
8 | Introduction to Civil Engineering

prominent ocean-bound structures are offshore platforms for oil exploration and production. Ocean
engineers provide estimates of forces generated from waves and currents and the interaction of wave and
structure so that structural engineers can design a platform to withstand such forces. Other ocean-bound
structures include offshore wind farms and pipelines to transport materials from offshore to shore. While
ships are obvious ocean-bound structures, their design usually falls in the realm of naval architecture, which
integrates several engineering disciplines: structural, ocean, mechanical, and electrical for the design of
ships. Naval architecture is not considered as a part of civil engineering.

1.6 Transportation Engineering

Transportation engineering deals with the efficient transport of people and goods. The content of transpor-
tation engineering changes whenever a new mode of transportation becomes viable. For example the advent
of airplanes and air travel led to new technical fields such as airport design and air traffic control. Several
sometimes overlapping technical specialties are part of transportation engineering:
Transportation Planning. Transportation infrastructure is mostly government funded or at least
government approved. Before any physical facilities are designed and built for moving people and goods,
decisions must be made from policy and political considerations. Transportation planning considers policy
formation processes, cost, financing, and projected performance of potential transportation systems, includ-
ing inter-modal transportation that involves more than one mode of travel such as sea-land-air travels.
Transportation System Engineering. Transportation System Engineering entails the efficient manage-
ment and operation practices, design, and assessment of the cost-effectiveness of transportation systems.
The assessment of transportation systems requires performance modeling techniques, traffic simulation, and
environmental impact (noise and air pollution) analyses.
Highway Engineering. Highway engineering focuses on the planning, design, construction, and op-
eration and maintenance of highways. Unique to highway engineering is the design and construction of
highway pavements and foundations, and the design of highway interchanges. The operation of highways
includes the use of high-occupancy lanes and networked signals and displays that can alter lane direction
during rush traffic and warn travelers of road conditions ahead. Design and construction of toll booths and
ways of collecting tolls are part of highway engineering as well.
Railway Engineering. With the advent of high speed railway, light-rail systems, and magnetic levitation
systems, railway engineering gained renewed interest in civil engineering. Railway transportation remains
a cost-effective way of transporting large quantities of goods on land. Railway engineering focuses on the
planning, design, construction, operation, and maintenance of railways. Advances in electronic signal design
and communication technology provide new tools in the control of railway traffic for efficiency and safety.
Port and Harbor Engineering. Even under the most favorable natural conditions, a port requires ad-
ditional engineering to make sure ships can safely navigate through and dock, and cargoes can be efficiently
unloaded and uploaded for shipping elsewhere. Some harbors require routine dredging to maintain the
navigation channel. Some harbors need breakwaters to tame the ocean waves. These are the scope of port
and harbor engineering.
Airport Engineering. The construction of a new airport requires extensive planning, including the study
on demand, environmental impact, cost analysis, and investment returns. Since an airport is part of an air
transport system, the impact of a new airport must consider the regional air traffic demand. Airport site
 Chapter 1 | 9

selection requires extensive study of regional topography, including local constructed structures, prevailing
winds, and, for airplane safety, movements of birds. The impact on nearby neighborhoods includes the new
land traffic generated by travelers and the inevitable noise generated along air traffic routes. The engineering
of airport infrastructure such as runways, terminals, and signals falls in the domain of other civil engineering
specialties.
Traffic Engineering. Traffic engineering sometimes is considered as synonymous to transportation engi-
neering but it is usually defined as the narrower field of management of traffic flow. From huge metropolises
to small towns, surface traffic must be controlled and modulated for safety and speed. Traffic engineers use
projected and monitored traffic patterns and volume to design automated or centrally controlled street
signals to modulate traffic. Tools used for traffic control include weight sensors for triggering of left-turn
signals and ramp-entry signals for freeway entry during rush hours.
Intelligent Transportation Systems. New and emerging electronic and computer technologies make
it possible to efficiently monitor, evaluate, and improve the performance of transportation systems. An
example of the applications to new vehicle engineering is the possibility of high-speed automated highway
travel by automobiles in groups. Another example is the centrally controlled traffic monitoring and manage-
ment systems used in large cities. Fully automated automobiles without drivers are in development and
testing.
Transportation engineering is a specialty whose content is changing with emerging technologies. The
operational aspect of transportation engineering is heavily influenced by the advances in new technologies.

1.7 Construction Engineering

Transforming design details on paper or a computer file into physical reality is the task for construction
engineers and managers. The successful completion of a construction project requires the integration of sev-
eral areas: human resources management; financial resources and cost management; construction processes;
schedule design and control; construction machinery; electric and mechanical facilities; legal, health, and
safety issues; and risk management. Construction firms may specialize in one or more types of structures:
buildings, bridges, dams, highways, airports, and ports and harbors. In the following some of the engineer-
ing and management aspects of construction engineering are described.
Construction Processes. Depending on the types of structures to be constructed and the materials to
be used, each project has a unique process to follow to erect the structure step by step. The execution of a
physical construction process often requires the knowledge of geotechnical engineering, structural engineer-
ing, construction materials, and site surveying. For example the construction of a multi-story building starts
with the placement of foundations, which may consist of multiple steel or concrete piles. The story-by-story
erection process depends on whether the building is of concrete or steel construction. These processes follow
well-developed practices deeply rooted in accumulated experience of construction engineers. Intertwined
with the construction process is construction scheduling, which lays out in detail the daily activities to be
performed. Creating a construction schedule must consider the critical phases that determine the time
length of the project.
10 | Introduction to Civil Engineering

Electric and Mechanical Facilities. In most construction projects electric and mechanical facilities are
to be installed. Basic knowledge of common electrical and mechanical facilities is required for the installa-
tion of these facilities and their integration into the main structure.
Construction Machinery. From conveyer belts and bulldozers to scrapers, excavators, loaders, graders,
compactors, cranes, and pipe-layers, many different machines are used in construction. Knowledge of these
machines’ capabilities is required in planning the types and quantity needed for a construction project.
Financial and Cost Management. From the bidding for a project to the actual execution, financial
management is at the center of concern of construction management. Bidding is the critical process by
which potential contractors compete to earn the contract from the owner of the project. Cost estimating is
done at the beginning of a potential project and updated throughout the project. During the project period,
cash flow management is needed to keep the project going. Successful contractors often have sufficient cash
reserve or credit to ride through tough periods of low income cash flow and high outward cash payments.
Contracts and Specifications. A contract is a legal document that specifies the responsibilities of both
the contractor who delivers the service and the owner who receives the service. Specifications are also
legal documents that the contractor must follow throughout the project. Specifications are often based on
well-developed construction practices but sometimes may include special requirements added by the owner.
These legal documents are developed, negotiated, and decided upon between the contractor and the owner.
Health and Safety Issues. The health and safety of not only construction workers but also neighboring
residents or passersby is regulated by the government. The Occupational Safety and Health Administration
(OSHA) of the Department of Labor enforces regulations and laws pertaining to construction. Knowledge
of these regulations and the strict adherence to the regulations are the responsibilities of the construction
engineers or managers.
Legal Issues and Risk Management. Examples of potential liabilities of a construction project include
negative cash flow, missing construction time milestones, mistakes in the constructed configurations or
the strength of materials used, and accidents resulting in human injury or death. The prevention of any of
the above and the timely management of the damage once any of them has occurred are the realm of risk
management. Legal and ethical guidelines are to be followed at all times.

1.8 Geomatics (Surveying Engineering)

Geomatics is the new name for an expanded technical area of what used to be called surveying or surveying
engineering. Surveying engineering is the technical specialty that applies the science of measurement to the
assembling of spatial data on land and sea and any natural or constructed objects thereof. Its applications
often involve legally required documentation related to the transaction of property, the location of routes or
points needed in construction, and the collecting of global data for resources analysis and utilization. Some
of the technical areas in surveying engineering are described herein.
Plane Surveying. The earth is a near circular sphere with a curved surface. When the area of interest
on the earth’s surface is small enough such that the curvature throughout the area can be neglected in
measuring and locating any point in the area without causing any significant amount of error, the earth
surface is treated as if it is a plane. With this simplification, the measuring and locating of points or routes
are mathematically much simpler. Plane surveying deals with the techniques and skills in carrying out such
 Chapter 1 | 11

tasks within the limit of the simplification. In plane surveying, optical instruments are used in conjunction
with basic mathematics to accurately locate points and lines, to calculate areas and volumes, and to map
existing surface features. Since no scientific measurements can be made without error because of limited
resolution of instruments or even human mistakes, techniques are developed to minimize measurement
errors.
Route Location. For highway or railway construction, the roadway configuration contains straight lines
and curves in the horizontal and vertical planes necessitated by the change of natural topography and the
need to minimize the cost of changing the natural topography. These lines and curves are designed and
documented on paper or in computer files. Route location is the technique to transform points on these
lines and curves on file to points on location with reference to some existing points on the earth’s surface.
Land Surveying. Land surveying pertains to the measuring and documentation of property lines, streets
and alleys, subdivisions and lots within a city or a village. The map of the area is called a plat, which forms
the legal document in ownership transactions.
Geodetic Surveying. In contrast to plane surveying, geodetic surveying includes the earth’s surface
curvature in its calculations and mapping. More advanced mathematical tools such as spherical geometry
and curvilinear coordinate systems are utilized for the computation of measure data and locating of points
and lines. Surveying instruments may include the use of aerial photography, satellite imaging, and the global
positioning system (GPS). Geodetic survey is necessary in the construction of long-span bridges and other
structures covering a large stretch of area (a long tunnel, for example).
Aerial Photography and Satellite Imaging. With the advances in optical and electronics technologies
resulting in improved measurement resolution and accuracy, images produced from airplane- or satellite-
based instruments can be used in the mapping of ground features for various applications. Advances in
digital photography and computer automation make the mapping efficient and much less time consum-
ing. Applications of such images and maps range from real estate information generation and agricultural
resources evaluation to military and intelligence information evaluations.
Geographical Information System. Surveying engineering provides the basis for the geographical
information system (GIS) of a state, city, town, or smaller unit. Useful information such as basic population
data, infrastructure, average income of population, demographic distribution, educational and recreational
resources, crime rate, etc., is compiled in a single database for easy retrieving and application. GIS is also an
example of the increasing application of information technology to surveying engineering.
It should be mentioned that surveying nowadays is often considered a technical area outside of civil en-
gineering or engineering in general. After all, the organization in charge of testing for professional licensing
in engineering and surveying, National Council of Examiners for Engineering and Surveying (NCEES), has
separate exams for engineers and surveyors. Some civil engineering programs do not require any surveying
courses.

1.9 Urban Planning

Urban planning integrates land use planning, infrastructure planning, and public policy for new develop-
ments or renewal of urbanized communities. Successful urban planning requires the application of the
knowledge developed in social, economic, architectural, and engineering studies. Educational programs for
12 | Introduction to Civil Engineering

future urban planners can be found in Bachelor of Arts degree programs in arts, architecture, and public
policy colleges. Bachelor of Science degree programs are also available in some universities. The application
of civil engineering to urban planning emphasizes the physical aspects of urban planning: street patterns,
park and recreational areas, industrial and residential areas, transportation systems such as freeway access,
airport access, public works and utilities, infrastructure management, etc.

1.10 Related Disciplines

Some of the disciplines that interact or overlap with civil engineering are described here.
Applied Mechanics. Mechanics is one of the early and basic areas in physics. It studies the nature and
effects of force. Applied or engineering mechanics emphasizes the application aspects of the theory of
mechanics. Civil engineering structures are subjected to the effects of gravitational force, wind, and earth-
quakes, and effects of temperature change. Various applied mechanics areas ranging from the fundamental
statics, dynamics, fluid mechanics, and mechanics of materials to more advanced areas such as thermal stress
and wave propagation have direct applications in structural engineering, geotechnical engineering, water
resources engineering, environmental engineering, and construction engineering.
Architectural Engineering. Architectural engineering specializes in the aesthetics and structural design
of buildings. The structural design aspect of architectural engineering overlaps with structural engineering.
The aesthetical design aspect of architectural engineering applies the knowledge developed in architecture
studies. Architectural engineers also study and apply knowledge in electrical and mechanical systems to
building designs.
Agricultural Engineering. Agricultural engineering traditionally entails two technical areas: irriga-
tion engineering and mechanization. Irrigation engineering is part of hydraulic engineering while the
mechanization of agricultural engineering is part of mechanical engineering. Since the advent of DNA
engineering, the use of basic molecular biology techniques to change the properties of a specific crop
becomes an important addition to agricultural engineering and a new and broader field of bioengineering
emerges.
Aerospace Engineering. Aerospace Engineering entails aeronautical engineering and astronautical
engineering, which develops vehicles that fly through the earth’s atmosphere and beyond, respectively. The
structural analysis and design of the flying vehicles, airplanes, and spacecraft, is most similar to that of civil
structural engineering. The difference is in the nature of forces acting on the structures.
Biomedical Engineering. Biomedical engineering specializes in the applications of engineering to the
medical field, including medical imaging, surgical devices, and implant devices. The structural analysis and
design of medical devices and implants overlaps with civil structural engineering in the basic tools used and
differs in the nature of forces acting on the devices.
Naval Architecture. As briefly described before, naval architecture specialized in the design of ships.
The analysis and design of ship structures is similar to that of civil structures and uses similar computational
tools. The difference is in the nature of forces acting on the structures.
Preface

Formation control of distributed agents (in short, distributed formation control) has
a unique feature, different from the traditional formation control problems. The
traditional formation control problems attempt to control the positions of agents
directly, which are usually modeled as nonlinear or high-order dynamical systems.
Since the positions of agents are controlled, it is basically assumed that the position
information is available to each agent. That is, each agent is assumed to be able to
sense its position pi with respect to a global coordinate frame. However, since the
positions are expressed with respect to a common global coordinate frame, it is not
a distributed approach from a sensing perspective. With the introduction of con-
sensus algorithms, the interaction features between neighboring agents have been
utilized for a formation control. The interactions are usually characterized by dif-
fusive couplings such as pi
pj where agents i and j are neighboring agents. The
diffusive coupling can be considered as a displacement vector between two agents i
and j. So, in consensus approaches, agents are assumed to be able to measure the
displacements pi
pj . Let agent i be able to measure the displacement pji , pj
pi
and agent k can measure the displacement pjk , pj
pk . But, we can see that the
two displacements pji and pjk are still expressed with respect to the same common
coordinate frame. Thus, the consensus-based formation control requires a global
coordinate frame as a base for defining the displacement vectors. However, if a
global sensing information or interactions with a centralized coordinator are
required, it may reduce the performance of formation systems or increase the
implementation costs. Let us imagine a group of unmanned aerial vehicles that need
to move or behave together in certain formation configurations. A leader in this
group is moving autonomously or by a remote control. The other agents need to
move with respect to the leader in specific relative positions and/or relative ori-
entations. If the follower agents can sense their states with respect to other agents
including the leader, then they do not need to have interactions with the central
computer. Even though global position information is available to each agent, it
should be transformed to relative information between agents since the formation
configuration is defined with respect to the motion of the leader. Of course, we can

ix
x Preface

assign the desired trajectories of agents in real-time or in an offline manner. In this

case, for the real-time trajectory assignment, each agent needs to communicate with
the central coordinator, and for the offline trajectory uploading, only the previously
designed trajectories can be pursued by the group. Thus, the distributed formation
control has more autonomy and can save the implementation costs since motions of
agents are defined in a relative way by local sensing. This monograph is motivated
by this observation and attempts to deliver the recent developments in the dis-
tributed formation control.
Historically, the distributed formation control began with gradient control laws
that try to reduce a potential function that is a function of errors of distances
between neighboring agents. In Part II, we introduce gradient control laws for
global and local stabilizations for general undirected graph cases. These results are
presented in Chaps. 3 and 4 respectively. When the interactions between neigh-
boring agents are directed, without cycles, it is relatively clear to ensure a global
convergence. Chapter 5 introduces global stabilizations for distributed formations
with specific directed graphs called persistent formations. However, the gradient
control laws introduced in Part II can only ensure local stabilizations for general
cases. To remedy this weakness, in Part III, we introduce formation control via
orientation alignment. Note that when the directions of axes of a local coordinate
frame of an agent are not same to the directions of axes of a global reference
coordinate frame, it is called that the orientation of an agent is not aligned to the
global coordinate frame. Similarly, when the directions of axes of two local
coordinate frames are not the same, it is said that the two local coordinate frames
are not aligned. Thus, by performing orientation alignment as well as formation
control simultaneously, the misaligned orientations of the agents can be aligned and
a desired formation can be achieved. The orientation alignment implies that the
formation control in distributed setups can be transformed into the consensus-based
displacement control. Thus, global or quasi-global convergences (see Chap. 6 for
the definition of quasi-global stability) can be guaranteed depending on initial
angles and communication variables. In Chap. 6, quasi-global convergence is
ensured while in Chap. 8, global convergence is ensured. Note that for the orien-
tation alignments in distributed agent systems, it is required to have communica-
tions between neighboring agents for exchanging certain information. Thus, the
orientation alignment-based formation control schemes introduced in Part III have
advantages of (quasi-) global stabilization but has disadvantages of requiring
communications between agents. Also, note that the quasi-global stabilization in
Chap. 6 requires less information exchange and less amount of computations than
the global stabilization schemes introduced in Chap. 8. In Chap. 7, a more advanced
formation control scheme is presented, which estimates the positions of agents with
only local relative displacements. Then, the estimated positions are directly used for
the formation control of distributed agents. Thus, the formation control in dis-
tributed setups can be transformed into the position-based formation control. In
Part IV, distributed formation control problems with bearing measurements are
introduced. Also, in this part, for a global stabilization of agents to a desired
formation configuration, the orientation alignment schemes are utilized. In Part V,
Preface xi

recent developments in advanced topics are introduced. The advanced topics

include formation control with a moving leader and formation control under a
moving frame in Chap. 11 and global stabilization of any complete graphs under
gradient control laws in Chap. 12. Since a leader is moving with a constant velocity,
other agents need to estimate the velocity in their own local coordinate frames. It is
also shown that when the agents are modeled by double integrators, a desired
formation configuration can be achieved while making agents move together with a
velocity matching consensus. Without orientation alignment, it is shown that for-
mations modeled by complete graphs in any dimensional spaces can be stabilized to
a desired formation. But, in this case, it is required to have all-to-all communica-
tions among agents. Depending upon applications, it may be necessary to expand
and contract the size of formation only using local measurements. This topic is also
introduced in Chap. 12. The objective of formation control is to achieve a desired
configuration in Euclidean space; but when orientations of agents also need to be
controlled relatively, the positions and orientations of a group of agents may be
controlled simultaneously. This problem may be called formation control of special
Euclidean groups, and will be partially addressed in Chaps. 8 and 9 seeking a
synchronization of orientation angles of agents. Finally, it has been observed that
the distributed formation control setups can be exactly used to localize agents’
positions. This problem may be called a duality between formation control and
network localization. This issue and some recent results are presented in Chap. 13.
Fig. 1 shows the outline of the monograph.
The works of Chaps. 3, 4, 5, 11 and Sect. 12.1 can be classified as the results in
gradient-based formation control laws. The works in Chaps. 6–8 can be classified as
the results in orientation alignment-based formation control laws, and the works in
Chaps. 9, 10 and Sect. 11.3 can be classified as the results in bearing-based for-
mation control laws. The resizing and scaling problems in Sect. 12.2 are developed
under gradient-based and bearing-based approaches, and the network localizations
in Chap. 13 can be developed under all three classes.
In Part I, the preliminary notations and essential mathematics are included as
background of this monograph. In Chap. 1, we include the basic concepts and
definition for distributed formation control. It defines formation control problems
motivated from collective behaviors in nature and in human society. Then, this
chapter includes basic mathematical concepts and the properties of distributed agent
systems. The distributed agent systems are characterized by distributed sensings,
distributed communications, and distributed computation and control. With these
concepts, in Chap. 1, we formulate the formation control problems of distributed
multi-agent systems. Since the results in this monograph are highly based on
concepts in graph theory, graph rigidity, persistence, consensus, and some control
theory, these are included in Chap. 2 as mathematical background. These mathe-
matical backgrounds are essential for understanding distributed formation control
problems. If a reader does not have enough background, it is recommended to read
xii Preface

GbFC OAbFC BbFC

Chapter 9: Quasi-
Chapter 3: Global Chapter 6: Quasi-
global convergence
stabilization global stabilization
& triangular

Chapter 4: Local Chapter 7: Chapter 10: Global

stabilization Position-based convergence

Chapter 5: Chapter 8: Global Section 11.3:

Directed graphs stabilization Double integrators

Section 11.1,
Section 12.2: Re-
11.2, 11.3:
sizing and scaling
Moving formation

Section 12.1:
Chapter 13: Net-
Modified gra-
work localization
dient for K(n)

Fig. 1 Outline of monograph: GbFC—Gradient-based formation control laws. OAbFC—

Orientation alignment-based formation control laws. BbFC—Bearing-based formation control laws

Chap. 2 first. But, if the author is familiar with rigidity theory, graph theory and
nonlinear control, this chapter can be skipped. To increase the readability of
monograph, some proofs which are lengthy or direct extensions of the previous
proofs are placed at the Appendix.

Buk-gu, Gwangju, Korea (Republic of) Hyo-Sung Ahn

Acknowledgements

This monograph is based on my previous works, which were collaborated with my

M.S. and Ph.D. students, and my colleague friends. I reused and reproduced the
mathematical parts from my conference and journal papers with permissions from
publishers including IEEE, Elsevier, Wiley, Springer, IET, IFAC, and ICROS.
I have obtained the permissions for reuse and reproduce of the previous publica-
tions published by IEEE, Elsevier, Wiley, and Springer for this monograph through
Copyright Clearance Center’s RightsLink® service. I have obtained the permissions
from the Institution of Engineering & Technolog (IET) and IFAC through email
contacts and from the Institute of Control, Robotics and Systems (ICROS) of Korea
by the returned rights. I clearly indicated and cited the appropriate conference or
journal papers that were reused and reproduced for this monograph at the end of
each chapter. This monograph is fully rewritten in a consistent way, with standard
notations and symbols, and all the figures and simulation results are newly pro-
duced or regenerated. All the numerical simulations were performed by
MATLAB®. The MATLAB source codes for numerical simulations and source
files of figures can be downloaded from [Link] Some
new results, background material, and examples have been added to this mono-
graph. I would like to acknowledge that the works of this monograph have been
supported from the National Research Foundation (NRF) of Korea under the grant
NRF-2017R1A2B3007034.
I would like to appreciate my students for their works during their degree period
at Gwangju Institute of Science and Technology (GIST). Without their works, I
could not have completed this monograph. I appreciate my former students Hwan
Hur, Kwang-Kyo Oh, Byeong-Yeon Kim, Young-Cheol Choi, Ji-Hwan Son,
Sang-Chul Lee, Seung-Ju Lee, Young-Hun Lim, Myoung-Chul Park, Byung-Hun
Lee, Sung-Mo Kang, Han-Eol Kim, Tae-Kyung Lee, Tong Duy Son, Sang-Hyuk
Yun, Je-Young Jeong, Yun-Tae Kim, Seok-Young Han, Hae-Jun Kim, Viet Hoang
Pham, Han-Young Park, Kyu-Hwan Kim, and Phuong Huu Hoang, and current
students Gwi-Han Ko, Jae-Gyeong Lee, Minh Hoang Trinh, Koog-Hwan Oh,
Yoo-Bin Bae, Seong-Ho Kwon, Quoc Van Tran, Young-Hun John, Jin-Hee Son,

xiii
xiv Acknowledgements

Jae-Ryoung Byun, Hong-Jun Lee, Chuong Van Nguyen, Hong-Kyong Kim,

Shin-Hyun Park, and Chan-Yeong Jeong. I also appreciate the colleagues Zhiyong
Sun, Shiyu Zhao, Mengbin Ye, Daniel Zelazo, and Brian D. O. Anderson for their
contributions on formation control theory presented in this monograph. I specially
appreciate Prof. Brian D. O. Anderson for his encouragement, and insight, technical
details and scholarly attitude throughout our collaborations. Special thanks also
should go to Dr. Zhiyong Sun for his helpful and detailed technical comments in
this monograph and go to the editor Oliver Jackson for his professional comments.
This monograph is highly dependent on the works of my former students conducted
at the distributed control and autonomous systems lab. (DCASL) of GIST and the
collaborations with the colleagues.

Gwangju, Korea (Republic of) Hyo-Sung Ahn

January 2019 Gwangju Institute of Science and Technology
email: hyosung@[Link]
Contents

Part I Background
1 Preliminary Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Formations and Collective Behaviors . . . . . . . . . . . . . . . . . . . . 3
1.2 Concepts and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.1 Distributed Sensings . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.2 Distributed Communications . . . . . . . . . . . . . . . . . . . . 16
1.3.3 Distributed Computation and Control . . . . . . . . . . . . . 17
1.4 Formation Control of Multi-agent Systems . . . . . . . . . . . . . . . . 18
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2 Mathematical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.1 Basics of Graph Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.2 Rigidity Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.2.1 Graph Rigidity (Distance Rigidity) . . . . . . . . . . . . . . . 31
2.2.2 Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.2.3 Bearing Rigidity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2.4 Weak Rigidity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.3 Consensus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.4 Basics of Control Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.4.1 Essential Concepts for Nonlinear Control . . . . . . . . . . . 50
2.4.2 LaSalle’s Invariance Principle and Barbalat’s
Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.4.3 Input-to-State Stability . . . . . . . . . . . . . . . . . . . . . . . . 53
2.4.4 Finite-Time Stability . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

xv
xvi Contents

Part II Gradient Control Laws

3 Global Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.1 Gradient Control Laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2 Global Convergence of Three-Agent Formations . . . . . . . . . . . . 66
3.3 Global Convergence of Polygon Formations . . . . . . . . . . . . . . . 72
3.4 Global Convergence of Kð3Þ þ 1 Edge Formations . . . . . . . . . . 78
3.5 Global Convergence of Kð4Þ
1 Edge Formations . . . . . . . . . . 85
3.6 Global Convergence in 3-Dimensional Space . . . . . . . . . . . . . . 88
3.7 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4 Local Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.1 Inter-agent Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2 Exponential Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.3 Local Asymptotic Stability in d-Dimensional Space . . . . . . . . . 107
4.4 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5 Persistent Formations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.1 Acyclic Minimally Persistent Formations in 2-Dimensional
Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.2 Acyclic Minimally Persistent Formations in 3-Dimensional
Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.3 Acyclic Persistent Formations in 2-Dimensional Space . . . . . . . 128
5.4 Finite-Time Convergence of Formations . . . . . . . . . . . . . . . . . . 131
5.5 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Part III Orientation Alignment-based Approaches

6 Formation Control via Orientation Alignment . . . . . . . . . . . . . . . . 143
6.1 Formation Control via Orientation Estimation . . . . . . . . . . . . . . 143
6.2 Switching Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.3 Formation Control via Orientation Control . . . . . . . . . . . . . . . . 154
6.4 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Contents xvii

7 Formation Control via Orientation and Position Estimation . . . . . . 165

7.1 Formation Control via Orientation Control and Position
Estimation in 2-Dimensional Space . . . . . . . . . . . . . . . . . . . . . 165
7.2 Formation Control via Orientation Control and Position
Estimation in 3-Dimensional Space . . . . . . . . . . . . . . . . . . . . . 171
7.3 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
8 Formation Control via Global Orientation Alignment . . . . . . . . . . 185
8.1 Formation Control in 2-Dimensional Space . . . . . . . . . . . . . . . 185
8.2 Formation Control in 3-Dimensional Space . . . . . . . . . . . . . . . 191
8.3 Orientation Control in 2-Dimensional Space . . . . . . . . . . . . . . . 196
8.4 Orientation Control in 3-Dimensional Space . . . . . . . . . . . . . . . 204
8.4.1 Convergence Analysis of Auxiliary Variables . . . . . . . 206
8.4.2 Consensus Protocol on SO(3) . . . . . . . . . . . . . . . . . . . 207
8.5 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
8.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Part IV Bearing-Based Formation Control

9 Formation Control via Bearing Measurements . . . . . . . . . . . . . . . . 221
9.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.2 Bearing-Based Formation Control . . . . . . . . . . . . . . . . . . . . . . 224
9.3 Bearing-Based Formation Control via Orientation
Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
9.4 Bearing-Based Control of a Cyclic Triangular Formation . . . . . 246
9.5 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
9.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
10 Bearing-Based Formation Control via Global Orientation
Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
10.1 Displacement-Transformed Formation Control . . . . . . . . . . . . . 255
10.2 Position-Transformed Formation Control . . . . . . . . . . . . . . . . . 261
10.3 Summary and Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
10.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

Part V Advanced Topics

11 Moving Formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
11.1 Moving Formation of 3 Agents . . . . . . . . . . . . . . . . . . . . . . . . 271
11.2 Shape and Direction Angle Control . . . . . . . . . . . . . . . . . . . . . 277
 2
Examining the Balance
of Plot, Exploration,
and Combat
Wide and Narrow Levels. . . 10

The Short Version: The first half of this section attempts to quantify the three
constituent elements of FFVI: plot, exploration, and combat, and weigh them in
proportional measure. This measurement is based on the notion that in Final Fantasy VI,
like many RPGs, the overall structure of the game is that of linked quests. From the
measurement of these elements on a graph, the pacing of the quests becomes very
clear and its pointedness is explained. The second part of this chapter analyzes the
how the level-up system is both wide and tall, two technical concepts which help tie
the second half of the game together structurally.

We’ll get to characters (and music!) in the next section, but before we do, there’s
one last way to analyze how the plot fits in with the game. As I see it, there are
three elements to an RPG, and while they frequently overlap, it’s fairly easy to
distinguish them. Those three elements are plot, exploration, and combat.
Plot is fairly obvious: it is the story that the game tells to frame the events of
the narrative. Some games don’t have plots, but most RPGs do. Obviously, it is
possible to have many diverging plots in one game; to some degree, the second half
of FFVI dabbles in multiple storylines, but not in an especially complicated way.
When quantifying plot, I counted all the script words spoken on a given quest.

7
As to how I divided quests, I usually looked for a moment in the script where the
stated objective changed.
Exploration is all of the non-combat action performed in the world. Exploring
towns, viewing optional scenes, talking to NPCs, and many other things are all
exploration activities. There is often a lot of overlap between exploration and
plot, but not too much in FFVI. Moreover, even in games where there is overlap,
it is still possible to quantify and analyze the design choices made to create
exploration.
Combat seems simple enough: How much fighting is involved in each quest?
This turned out to be trickier to tabulate than I expected. How do you assess how
much “fighting time” each dungeon represents? There are actually numerous ways
to do it, although each of these ways brings up more questions than results. For
the purposes of the pacing graph, however, we want to know about is quest design,
not dungeon design because we’re still analyzing how the plot, exploration, and
combat interact to form our perception of the game’s overall experience. (An
analysis of the dungeon design, which was often deceptively elaborate, follows in
Chapter 6.) To analyze how the combat is constructed to form quests, I analyze it
using what I call measured time (this is a meaningful term, I promise; see below).
FFVI measures time by using a step count, which is also used to figure out when
to start a random encounter. Dungeons are constructed on an invisible grid, each
block of which represents a step for the party. The distance from the entrance of
any given dungeon to its exit is a concrete number of steps, given the ideal path.
Thus, you can rate a dungeon based on the minimum number of steps that it takes
to go through it.
Unfortunately, nobody travels the perfect path through a dungeon or, at the
very least, they don’t do it their first time through a dungeon. There are often
distractions like puzzles and treasure chests. Final Fantasy, as a general rule, is
not as puzzle-heavy as something like Legend of Zelda or Lufia. In fact, in the
first half of the game, there is only one dungeon with real puzzles—the Cave to
the Sealed Gate. Accordingly, when fashioning a combat “score” for each quest,
instead of relying upon puzzles to inform me of the accurate length of a dungeon,
I rely on treasure chests.
FFVI’s treasure chests are unusually rich; there are a lot of them and they
frequently yield something very useful. (This is especially true in the second half
of the game, but the first half has its fair share of great loot.) Players learn quickly
that it helps to go off the critical path to get a chest because they know that its
contents could be extremely useful. What I have noticed is that because of the
usefulness of chests, every chest in a dungeon makes a player more eager to search
for the next chest. Accordingly, I gave extra combat weight to dungeons with more
chests. Finally, I added extra points to the combat score for having more bosses
in a quest. In any case, the graph is below, and we’ll break down its meaning and
trends.

8 2. Examining the Balance of Plot, Exploration, and Combat

Of course, the “scores” on the graph were adjusted so that they all fit in the same
space—that’s why there are no numbers. There’s no way to make these three
elements equivalent in a strict mathematical sense, and I don’t even try. The
point of having them all on the same graph is to show how the designers chose
to use each element in what relative proportion for each quest. For example, you
can see that the quest to find Terra has relatively less of all three game elements.
There aren’t too many dungeons, not a huge amount of exploration, nor much
plot. This I attribute to pacing. If every quest were longer than the last, the game
might feel tedious. The same pacing move happens when the party journeys to the
Sealed Gate. The designers are making the middle of the World of Balance flow by
alternating short and long quests. One other important thing to note is how the
combat score actually has a downward trend from beginning to end. Exploration
and plot jump up and down quite a bit, but the combat score starts high, climbs to
its highest point in the second quest, and then descends, plateaus, descends again,
and never makes it back to previous levels.
I admit that my scoring system has its flaws, but I don’t think that fact alone
accounts for what the graph shows. Rather, I think it actually shows that earlier
dungeons are designed to be bigger (and have more battles), but not harder. Later
dungeons are shorter but much denser with treasure and difficult encounters.
Ultimately, this is a design feature that has to do with real durability (RDur)
and damage against the average enemy (DAE), a couple of statistics that describe
exactly what these dungeons are doing. But that’s in Chapter 7—we’re still talking
about story and presentation right now.

2. Examining the Balance of Plot, Exploration, and Combat 9

 Of course, those who have played the game (and perhaps some who haven’t)
will immediately recognize that this kind of three-pronged analysis can’t really be
made to work on the second half of the game. While there is plot, it’s not linear.
Moreover, the gameplay structure of the World of Ruin is flexible as well. The
player’s ability to choose the structure of the second half means that in order to
analyze how the World of Ruin is constructed, we must use a different technique.

Wide and Narrow Levels

Why is it that the World of Ruin (the second half of the game) feels artistically
complete? Why does it feel like half the game when it’s only 25% of the script? Is it
just all the dungeons you have to do? I’d say no; there is something else at work. In
the mid and late 90s, there were numerous mediocre Japanese role-playing games
(JRPGs) that attempted to follow in the wake of Final Fantasy’s success. Generally,
these games were able to get right the large worlds, well-drawn characters, and
even interesting combat concepts. Those games usually failed because the end of
the game was a series of gigantic, interminable dungeons that added nothing to
the game except boring length. How did Squaresoft avoid that?
To understand how FFVI is different, we have to understand how players perceive
the game, retrospectively. Specifically, we have to understand what stands in for the
natural increase in player skill. A player of a real-time action game (including genres
like first-person shooter [FPS] and fighting games) remembers the game they’ve
played in two ways. First, they have a general impression of the experience. This
impression is what they’ll probably form their first opinion about. Second, and
perhaps more important, is their skills; players remember a game through, and
because of, skills. More than one accomplished designer has argued that videogames
are nothing more than learning experiences and that games are engaging because
learning cumulative skills is inherently fun. If that’s true, what players remember
of an action game in the long term is heavily influenced by the skills they acquire
while playing it. Of course, the more skills they learn, the better the game will seem,
retrospectively. (Sometimes this is confounded by other factors.) Playing through
the game again will remind them of how much they’ve learned, how much they’ve
mastered. The greater the distance between the player’s first fumbling attempts and
their current level of mastery, the greater the retrospective impression of the game
will be for them. “Wow,” they tell themselves. “I sure came a long way. What a game!”
Games like FFVI, which lean heavily simulated skills, cannot be remembered
quite in the same way because the player does not gain most of the skills,
the characters do. (This is what is meant by “simulated skill;” it is any skill a
character learns rather than a player. I.e., the player doesn’t have to learn a button
combination to cast Fire 3—Terra learns it when she reaches level 43.) Generally,
the acquisition of skills by characters is not a process of mastery, but a process
of discovery. The player has to locate and unlock character skills and equipment,
which are simulated substitutes for mastery. Hidden spells like Melton, hidden
weapons like Illumina, hidden characters like Gogo—these are the trophies of a

10 2. Examining the Balance of Plot, Exploration, and Combat

game rich in simulated skills. The player has to work hard to find these assets for
their party. Thus, when the player is remembering the game and thinking about
the artistic completeness of what they finished, they remember the things they
discovered rather than the things they mastered.
There’s a problem for games like FFVI. Even mediocre action games will
clearly leave the player more skilled at the end than they were at the beginning,
and this accumulation of player skill is gratifying. Every twitch of the thumbs
causes the player’s brain to recall all the skills that went into the action he or she
is performing now. RPGs do not share this trait. In a mediocre RPG, a player
replaces all of their gear with gear from the next shop. I.e., the player replaces
all of his leather armor with iron armor, and then iron armor with silver armor,
and so on. Finally, the player replaces all of that gear with treasure found in the
last couple of dungeons. This is the worst kind of linearity. Because the best gear
(or most of it) in this circumstance is simply plopped in front of the player, their
impression of the game will suffer, in retrospect. For one thing, the game is too
easy; it’s just a series of inflating stats that are placed, unavoidably, in the way of
the player’s party. What is the player going to remember about their agency in the
game? That they pressed “A” at all the right times? They didn’t have to be clever or
curious, or work hard for their skills and gear. That can be really disappointing.
How does this affect the design of the World of Ruin in FFVI? The second half
of the game is non-linear, so the designers are already at a disadvantage when
trying to make their game feel complete. Only a few quests in the second half
(like Cyan’s) build on other quests. How did the designers make them feel like
they add up to a cohesive whole? The answer has to do with the way that level-
ups are structured.
First, I want to define what I mean when I say “level-up.” I define a level-up
as a permanent, periodic increase in simulated skills. Let’s break this down term
by term. Level-ups are permanent; once Sabin goes from level 21 to level 22, he’s
not going back down to level 21. Periodic means that Sabin gains levels at regular
intervals when he has gained enough experience points. The differences between
levels are fairly regular; after level 30, each level-up only requires 4%–6% more
experience than the previous one. The last item in the definition is an increase
in simulated skills. At level 22, Sabin has just a little bit more HP and MP than
he does at level 21 and does slightly more damage with all his attacks. Those are
all simulated skills for surviving and casting magic, but Sabin’s character level
is only one small facet of how powerful he is. This is the basic damage formula
for FFVI:[3]

Damage = Battle Power + ((Level * Level * Attack)/256) * 3/2

This is for the “fight” command, but almost all of the damage formulae are similar.
While Sabin’s fight damage goes up with his level, it only goes up a little bit. Stats
other than his level affect his fight and special ability more than his character
level does.

Wide and Narrow Levels 11

 The most important thing to understand about this is that the other parts of
the formula are also level-ups. If a level-up is a permanent, periodic increase in
simulated skills, finding new weapons is a level-up, too. In every town (for most
of the game), there are new weapons and armor that replace the old weapons and
armor. Each new weapon increases Sabin’s simulated fighting skill just the same
as an increase in character level would. In fact, there’s even a statistic used later
in this book called LEQ which measures this. Battle power, as described in the
equation above, includes weapon power. If Sabin were to go from his first weapon
in the World of Ruin to his last weapon found in the World of Ruin, the increase
in damage would be equivalent to about six character levels gained. Because
these weapons come so regularly, we can safely say that (at least for most of the
game) they too are a permanent, periodic increase in simulated skills. If you
think they aren’t permanent, just remember that players equip their characters
with new weapons in succession. The increase in damage output from the
Dragon Claw is permanent until it’s replaced by the greater increase of the Tiger
Fangs. It’s the same as saying that the increase in stats of level 21 is permanent
until replaced by level 22; there’s no functional difference. Even when players
move gear from one character to another, those levels aren’t lost, they’re just
re-allocated from a pool of levels that can be disbursed to many characters. Thus,
gear is effectively permanent until replaced by something clearly better, which
happens on a regular basis.
Final Fantasy VI’s levels are memorable because there are a great variety of
them and the player has to be curious and creative in order to access all of their
power. There are a number of interlocking systems at play in FFVI that serve as
paths toward level-ups. Character levels are one, and they are acquired through
experience points. Purchased gear is another way to increase a character’s level,
acquired through another kind of experience system: GP (or Gil in later versions).
The “magic points” (later AP) system also works as a kind of experience system for
learning magic. Last but not least, there is also the Esper stat-boost system, which
appends bonuses to core character stats like vigor and stamina. This system is
the most complex, because it requires players to plan and ration their level-ups
in strategic ways.
There are also other systems outside of battle that behave in the same way,
requiring the player to be clever in his or her search of level-ups, rather than
merely grinding. The most obvious example of this is the search for gear. In the
World of Ruin, where there is no fixed quest order after the acquisition of the
airship, the available dungeon loot greatly expands the player’s path to power.
For the most part, this is because most of that loot is of permanent use. In
many RPGs, the best loot is acquired near the end of the game and it replaces
all the gear that came before it. FFVI avoids this pattern in more than one way.
While there are two sets of armor that are the “best” in terms of protection,
there are many sets of armor that almost as good, but those pieces of armor
are scattered about the entire world. In a bad RPG, all of the best armor and
weapons would be in the second-to-last dungeon. In FFVI, there is no real

12 2. Examining the Balance of Plot, Exploration, and Combat

second-to-last stop to provide this gear. If there were such a dungeon, it would
destroy pacing of the second half of the game. Why would the player do any
of the other optional content, when they can simply do that one dungeon to
get all the gear they need? Instead, what you see is a division of permanently
useful gear into a number of dungeons, starting even before the retrieval of the
second airship (Table 2.1).

Table 2.1 Endgame Loot from World of Ruin Quests

Quest Level Weapons Armor Accessories Espers

Figaro Cave/ 27 Soul Sabre Regal Crown

Basement
Daryl’s Tomb 29 Genji Helmet, EXP Egg
Czarina Gown
Narshe 30 Ribbon, Ice Tritoch
Shield
Umaro’s Cave 33 Minerva Gauntlet Terrato
Mount Zozo 34 Flame Shield, Gold Hairpin
Thunder
Shield, Red
Cap
Owzer’s House 36 Starlet
Cave on the 36 Striker, Tiger Fangs Behemoth Suit
Veldt
Cyan’s Soul 39 Aura Flame Shield, Alexander
Genji Glove
Gogo’s Realm 43 Genji Armor,
Red Jacket,
Thunder Shield
Ancient Castle 45 Gradeus, Scimitar, Gold Hairpin Odin/
Doom Darts, Wing Raiden
Edge, Punisher
Phoenix Cave 47 Wing Edge, Valiant Flame Shield Ribbon Phoenix
Knife
Fanatics Tower 52 Stopper, Pearl Lance Force Armor Gem Box

The correlation between dungeon level and the gear available within is not nearly
as strong as it would be in a strictly linear game, so there’s no covert linearity
lurking in an allegedly open environment. The later dungeons tend to have a
little bit more gear, but not definitively higher-quality gear. This is most obvious
with the Ancient Castle, which is enormously rich. It’s certainly no picnic, but
it’s a short dungeon and doesn’t require the player to be at a very high level.

Wide and Narrow Levels 13

It’s relatively easy to exit and has a save point mid-way through. It also has an
enormous amount of powerful loot—especially weapons. After the player finishes
this dungeon, their party will have a significant increase in power.
To demonstrate the psychological results of a dungeon like the Ancient
Castle, let us imagine a fictional player. He or she enters one of the non-linear
dungeons of the World of Ruin and gets badly beaten by the monsters there. He
or she then resets the game, goes to a different dungeon, and finds a bunch of
loot. Remember that most of the dungeons in the World of Ruin have at least
one piece of gear or an Esper that teaches spells which are viable for endgame
content. Returning to that first dungeon with new gear and spells, the players
succeeds with an inflated sense of accomplishment. “This used to be so hard,”
he chuckles. Linear games can’t really match this. Players can’t skip to later
dungeons and can’t foresee the difficulty that they have to overcome. This makes
the levels that the player gains seem fuller and more meaningful than they
might in a more linear game.
This scenario illustrates the meaning of the terms “wide” and “tall” levels.
A level up system can be tall or short. Final Fantasy games feature 99 possible
character levels, compared to something like Baldur’s Gate, which comes with a
cap at level 20. Final Fantasy VI therefore features a “tall” level-up system, in that
the player builds upon the same stats and equipment slots over and over up to 98
times. Final Fantasy VI also has “wide” levels. As explained above, all those pieces
of gear and spells are level-ups, too. Instead of upgrading that gear and those spells
in a boring linear fashion, they have to deeply explore the world of the game to
find them. What’s more, there’s a great variety in those spells and gear. There are
numerous combat spells like Fire 3 and Ultima, but there are also utility spells like
Haste 2 and Osmose. There is armor with ultra-high stats like the Genji Armor,
but there are also great accessories like the Economizer and Ribbon, which make
many parts of the game more convenient for the player. That’s the meaning of
width in a level-up system.
Final Fantasy VI is both tall and wide in its level-up systems. The height gives
the game structure because in order to take on certain tasks (especially the last
few dungeons), the player needs to acquire power over a long period. The width of
the level-up system gives the player freedom because of all of the potential ways
in which the characters can become more powerful. Freedom and structure are
not mutually exclusive; FFVI balances them to create an open-ended second half
that still feels climactic in the end. I’d like to point out one last idea that the FFVI
designers included to make their level progression seem wider. It’s an obvious
feature, but one that probably doesn’t register as being a kind of level-up, at first.
On the next page is a picture of the airship before and after retrieval of all the
characters.

14 2. Examining the Balance of Plot, Exploration, and Combat

Because the player has to use 12 characters for the final dungeon, every character
acquired in the second half of the game acts as another reclaimed set of levels—
especially considering that the reclaimed character comes back with the party’s
current average character level. Thus, every character that enters the party
increases the ability of the player to make it through the final dungeon by a
significant amount. This is a kind of meta-level-up, but it does help to structure
the second half of the game in a way that’s meaningful beyond mere plot.

Wide and Narrow Levels 15

 Global Portfolio Optimization – the Black-Litterman
approach
How to size your bets – the Kelly criterion
Optimal investment – multiple assets
Risk parity
Risk factor investment
Hierarchical risk parity
Trading and managing portfolios with Zipline
Scheduling signal generation and trade execution
Implementing mean-variance portfolio optimization
Measuring backtest performance with pyfolio
Creating the returns and benchmark inputs
Getting pyfolio input from Alphalens
Getting pyfolio input from a Zipline backtest
Walk-forward testing – out-of-sample returns
Summary performance statistics
Drawdown periods and factor exposure
Modeling event risk
Summary
6. The Machine Learning Process
How machine learning from data works
The challenge – matching the algorithm to the task
Supervised learning – teaching by example
Unsupervised learning – uncovering useful patterns
Use cases – from risk management to text
processing
Cluster algorithms – seeking similar observations
Dimensionality reduction – compressing
information
Reinforcement learning – learning by trial and error
The machine learning workflow
Basic walkthrough – k-nearest neighbors
Framing the problem – from goals to metrics
Prediction versus inference
Regression – popular loss functions and error
metrics
 Classification – making sense of the confusion
matrix
Collecting and preparing the data
Exploring, extracting, and engineering features
Using information theory to evaluate features
Selecting an ML algorithm
Design and tune the model
The bias-variance trade-off
Underfitting versus overfitting – a visual example
How to manage the bias-variance trade-off
Learning curves
How to select a model using cross-validation
How to implement cross-validation in Python
KFold iterator
Leave-one-out CV
Leave-P-Out CV
ShuffleSplit
Challenges with cross-validation in finance
Time series cross-validation with scikit-learn
Purging, embargoing, and combinatorial CV
Parameter tuning with scikit-learn and Yellowbrick
Validation curves – plotting the impact of
hyperparameters
Learning curves – diagnosing the bias-variance
trade-off
Parameter tuning using GridSearchCV and pipeline
Summary
7. Linear Models – From Risk Factors to Return Forecasts
From inference to prediction
The baseline model – multiple linear regression
How to formulate the model
How to train the model
Ordinary least squares – how to fit a hyperplane to
the data
Maximum likelihood estimation
Gradient descent
 The Gauss–Markov theorem
How to conduct statistical inference
How to diagnose and remedy problems
Goodness of fit
Heteroskedasticity
Serial correlation
Multicollinearity
How to run linear regression in practice
OLS with statsmodels
Stochastic gradient descent with sklearn
How to build a linear factor model
From the CAPM to the Fama–French factor models
Obtaining the risk factors
Fama–Macbeth regression
Regularizing linear regression using shrinkage
How to hedge against overfitting
How ridge regression works
How lasso regression works
How to predict returns with linear regression
Preparing model features and forward returns
Creating the investment universe
Selecting and computing alpha factors using TA-
Lib
Adding lagged returns
Generating target forward returns
Dummy encoding of categorical variables
Linear OLS regression using statsmodels
Selecting the relevant universe
Estimating the vanilla OLS regression
Diagnostic statistics
Linear regression using scikit-learn
Selecting features and targets
Cross-validating the model
Evaluating the results – information coefficient and
RMSE
Ridge regression using scikit-learn
 Tuning the regularization parameters using cross-
validation
Cross-validation results and ridge coefficient paths
Top 10 coefficients
Lasso regression using sklearn
Cross-validating the lasso model
Evaluating the results – IC and lasso path
Comparing the quality of the predictive signals
Linear classification
The logistic regression model
The objective function
The logistic function
Maximum likelihood estimation
How to conduct inference with statsmodels
Predicting price movements with logistic regression
How to convert a regression into a classification
problem
Cross-validating the logistic regression
hyperparameters
Evaluating the results using AUC and IC
Summary
8. The ML4T Workflow – From Model to Strategy Backtesting
How to backtest an ML-driven strategy
Backtesting pitfalls and how to avoid them
Getting the data right
Look-ahead bias – use only point-in-time data
Survivorship bias – track your historical universe
Outlier control – do not exclude realistic extremes
Sample period – try to represent relevant future
scenarios
Getting the simulation right
Mark-to-market performance – track risks over
time
Transaction costs – assume a realistic trading
environment
 Timing of decisions – properly sequence signals
and trades
Getting the statistics right
The minimum backtest length and the deflated SR
Optimal stopping for backtests
How a backtesting engine works
Vectorized versus event-driven backtesting
Key implementation aspects
Data ingestion – format, frequency, and timing
Factor engineering – built-in factors versus
libraries
ML models, predictions, and signals
Trading rules and execution
Performance evaluation
backtrader – a flexible tool for local backtests
Key concepts of backtrader's Cerebro architecture
Data feeds, lines, and indicators
From data and signals to trades – strategy
Commissions instead of commission schemes
Making it all happen – Cerebro
How to use backtrader in practice
How to load price and other data
How to formulate the trading logic
How to configure the Cerebro instance
backtrader summary and next steps
Zipline – scalable backtesting by Quantopian
Calendars and the Pipeline for robust simulations
Bundles – point-in-time data with on-the-fly
adjustments
The Algorithm API – backtests on a schedule
Known issues
Ingesting your own bundles with minute data
Getting your data ready to be bundled
Writing your custom bundle ingest function
Registering your bundle
Creating and registering a custom TradingCalendar
 The Pipeline API – backtesting an ML signal
Enabling the DataFrameLoader for our Pipeline
Creating a pipeline with a custom ML factor
How to train a model during the backtest
Preparing the features – how to define pipeline
factors
How to design a custom ML factor
Tracking model performance during a backtest
Instead of how to use
Summary
9. Time-Series Models for Volatility Forecasts and Statistical
Arbitrage
Tools for diagnostics and feature extraction
How to decompose time-series patterns
Rolling window statistics and moving averages
How to measure autocorrelation
How to diagnose and achieve stationarity
Transforming a time series to achieve stationarity
Handling instead of how to handle
On unit roots and random walks
How to diagnose a unit root
How to remove unit roots and work with the
resulting series
Time-series transformations in practice
Univariate time-series models
How to build autoregressive models
How to identify the number of lags
How to diagnose model fit
How to build moving-average models
How to identify the number of lags
The relationship between the AR and MA models
How to build ARIMA models and extensions
How to model differenced series
How to identify the number of AR and MA terms
Adding features – ARMAX
Adding seasonal differencing – SARIMAX
 How to forecast macro fundamentals
How to use time-series models to forecast volatility
The ARCH model
Generalizing ARCH – the GARCH model
How to build a model that forecasts volatility
Multivariate time-series models
Systems of equations
The vector autoregressive (VAR) model
Using the VAR model for macro forecasts
Cointegration – time series with a shared trend
The Engle-Granger two-step method
The Johansen likelihood-ratio test
Statistical arbitrage with cointegration
How to select and trade comoving asset pairs
Pairs trading in practice
Distance-based heuristics to find cointegrated
pairs
How well do the heuristics predict significant
cointegration?
Preparing the strategy backtest
Precomputing the cointegration tests
Getting entry and exit trades
Backtesting the strategy using backtrader
Tracking pairs with a custom DataClass
Running and evaluating the strategy
Extensions – how to do better
Summary
10. Bayesian ML – Dynamic Sharpe Ratios and Pairs Trading
How Bayesian machine learning works
How to update assumptions from empirical evidence
Exact inference – maximum a posteriori estimation
How to select priors
How to keep inference simple – conjugate priors
Dynamic probability estimates of asset price
moves
Deterministic and stochastic approximate inference
 Markov chain MonteCarlo sampling
Variational inference and automatic differentiation
Probabilistic programming with PyMC3
Bayesian machine learning with Theano
The PyMC3 workflow – predicting a recession
The data – leading recession indicators
Model definition – Bayesian logistic regression
Exact MAP inference
Approximate inference – MCMC
Approximate inference – variational Bayes
Model diagnostics
How to generate predictions
Summary and key takeaways
Bayesian ML for trading
Bayesian Sharpe ratio for performance comparison
Defining a custom probability model
Comparing the performance of two return series
Bayesian rolling regression for pairs trading
Stochastic volatility models
Summary
11. Random Forests – A Long-Short Strategy for Japanese Stocks
Decision trees – learning rules from data
How trees learn and apply decision rules
Decision trees in practice
The data – monthly stock returns and features
Building a regression tree with time-series data
Building a classification tree
Visualizing a decision tree
Evaluating decision tree predictions
Overfitting and regularization
How to regularize a decision tree
Decision tree pruning
Hyperparameter tuning
Using GridsearchCV with a custom metric
How to inspect the tree structure
 Comparing regression and classification
performance
Diagnosing training set size with learning curves
Gaining insight from feature importance
Strengths and weaknesses of decision trees
Random forests – making trees more reliable
Why ensemble models perform better
Bootstrap aggregation
How bagging lowers model variance
Bagged decision trees
How to build a random forest
How to train and tune a random forest
Feature importance for random forests
Out-of-bag testing
Pros and cons of random forests
Long-short signals for Japanese stocks
The data – Japanese equities
The features – lagged returns and technical
indicators
The outcomes – forward returns for different
horizons
The ML4T workflow with LightGBM
From universe selection to hyperparameter tuning
Sampling tickers to speed up cross-validation
Defining lookback, lookahead, and roll-forward
periods
Hyperparameter tuning with LightGBM
Cross-validating signals over various horizons
Analyzing cross-validation performance
Ensembling forecasts – signal analysis using
Alphalens
The strategy – backtest with Zipline
Ingesting Japanese Equities into Zipline
Running an in- and out-of-sample strategy
backtest
The results – evaluation with pyfolio
 Summary
12. Boosting Your Trading Strategy
Getting started – adaptive boosting
The AdaBoost algorithm
Using AdaBoost to predict monthly price moves
Gradient boosting – ensembles for most tasks
How to train and tune GBM models
Ensemble size and early stopping
Shrinkage and learning rate
Subsampling and stochastic gradient boosting
How to use gradient boosting with sklearn
How to tune parameters with GridSearchCV
Parameter impact on test scores
How to test on the holdout set
Using XGBoost, LightGBM, and CatBoost
How algorithmic innovations boost performance
Second-order loss function approximation
Simplified split-finding algorithms
Depth-wise versus leaf-wise growth
GPU-based training
DART – dropout for additive regression trees
Treatment of categorical features
Additional features and optimizations
A long-short trading strategy with boosting
Generating signals with LightGBM and CatBoost
From Python to C++ – creating binary data
formats
How to tune hyperparameters
How to evaluate the results
Inside the black box – interpreting GBM results
Feature importance
Partial dependence plots
SHapley Additive exPlanations
Backtesting a strategy based on a boosting ensemble
Lessons learned and next steps
Boosting for an intraday strategy
 Engineering features for high-frequency data
Minute-frequency signals with LightGBM
Evaluating the trading signal quality
Summary
13. Data-Driven Risk Factors and Asset Allocation with Unsupervised
Learning
Dimensionality reduction
The curse of dimensionality
Linear dimensionality reduction
Principal component analysis
Independent component analysis
Manifold learning – nonlinear dimensionality reduction
t-distributed Stochastic Neighbor Embedding
Uniform Manifold Approximation and Projection
PCA for trading
Data-driven risk factors
Preparing the data – top 350 US stocks
Running PCA to identify the key return drivers
Eigenportfolios
Clustering
k-means clustering
Assigning observations to clusters
Evaluating cluster quality
Hierarchical clustering
Different strategies and dissimilarity measures
Visualization – dendrograms
Density-based clustering
DBSCAN
Hierarchical DBSCAN
Gaussian mixture models
Hierarchical clustering for optimal portfolios
How hierarchical risk parity works
Backtesting HRP using an ML trading strategy
Ensembling the gradient boosting model
predictions
Using PyPortfolioOpt to compute HRP weights
 Performance comparison with pyfolio
Summary
14. Text Data for Trading – Sentiment Analysis
ML with text data – from language to features
Key challenges of working with text data
The NLP workflow
Parsing and tokenizing text data – selecting the
vocabulary
Linguistic annotation – relationships among tokens
Semantic annotation – from entities to knowledge
graphs
Labeling – assigning outcomes for predictive
modeling
Applications
From text to tokens – the NLP pipeline
NLP pipeline with spaCy and textacy
Parsing, tokenizing, and annotating a sentence
Batch-processing documents
Sentence boundary detection
Named entity recognition
N-grams
spaCy's streaming API
Multi-language NLP
NLP with TextBlob
Stemming
Sentiment polarity and subjectivity
Counting tokens – the document-term matrix
The bag-of-words model
Creating the document-term matrix
Measuring the similarity of documents
Document-term matrix with scikit-learn
Using CountVectorizer
TfidfTransformer and TfidfVectorizer
Key lessons instead of lessons learned
NLP for trading
The naive Bayes classifier
Welcome to our website – a haven for those who love books
and value knowledge. We believe that reading opens endless
doors — to learn, to imagine, and to become better each day.
With that belief, we’ve created a vast and dynamic
collection of books, ranging from classic literature and
scholarly works to self-growth guides and charming
children’s [Link] website is not only a marketplace, but
a bridge connecting readers to the lasting essence of
culture and wisdom. Designed with care, it offers a graceful
interface, smart navigation, and a seamless experience to
help you find what truly matters. Combined with our special
promotions and home delivery, reading becomes easier and
more enjoyable than [Link] us on this journey of
learning, inspiration, and self-discovery — where every book
marks a new beginning.

[Link]
 Imaging of the Liver and Intra-hepatic Biliary
Tract: Volume 1: Imaging Techniques and Non-
tumoral Pathologies Emilio Quaia

Get PDF Instantly

Click the button above to access the document.

Generated for [Link]