The Development of an IoT based Filipino Sign Language

Interpretation System with Haptic Technology for Deaf-Mute/

Students

A Capstone Project presented to the

Faculty of the Department of
Information Technology

In Partial Fulfillment of the

Requirements for the Degree of
Bachelor of Science in Information Technology

By

Belisario, John Christian B.

Bustamante, Christianne Mae D.
Angeles, Catherine Ysabel A.
Montero, Carmel T.
 February 2024

CHAPTER I
INTRODUCTION
1.0 Introduction
In today’s digitally transformed world, the Internet of Things (IoT)
has become a groundbreaking technology that continues to transform
multiple industries, including the education industry. One of the major
obstacles facing deaf and speech-impaired students in the educational
setting is the lack of effective communication. To address this challenge,
this capstone project seeks to develop an innovative IoT-based system
utilizing the haptic technology which is specifically designed for deaf and
speech-impaired students. To achieve such a solution, leveraging cutting-
edge IoT and haptic technologies, this system aims to bridge the
communication gap between teachers and students, enabling a more
inclusive learning environment.

The research focuses only on the peer development on an

innovative and creative system the fully resonates on the power of the
Internet of Things (IOT)/ Arduino and the haptic technology to interpret
and recognize the Filipino Sign Language. The system is thoroughly
designed to recognize and translate gestures from the Filipino Sign
Language into an audible, or a displayed language, this facilitating
seamless communication.

The application of a haptic technology in the system, which allows

the users to interact and feel or interpret information through touch and
hand gestures, providing more an immersive experience and an intuitive
way of learning and understanding Filipino Sign Language. This study is
not only beneficial to the educational setting, where immersive and
interactive learning can significantly enhance the student’s
comprehension in both verbal and non-verbal communication.
 By fostering a deeper understanding of language structure and
expression, the use of haptic technology can also contribute to improved
interpersonal communication between sign language users and their
hearing counterparts. Moreover, given the prevalence of digital
technologies in contemporary society, integrating haptic capabilities into
educational platforms could help bridge the gap between traditional
classroom instruction and online learning environments. Ultimately, the
inclusive nature of this technology holds great promise in breaking down
barriers to accessibility in education and far more beyond that

1.2 Objective of the Study

1.2.1 General Objectives:

 To create a Sign Language Interpretation System using IoT

based technology that operates real time.
 To provide a translated sign language gestures into written and
audible language and delivering haptic feedback for aiding
communication and learning.
 To generalize the construction of a sign language interpretation
system with machine learning and deep learning algorithms.

1.2.2 Specific Objectives:

 To create and design a prototype glove device that can translate

sign language into a displayed text and audible notion and
integration of the haptic technology which is to provide sensory
response and feedback.
 Integrate a network infrastructure to transmit and deliver
interpreted data and facilitate real time information.
 Conducting a comprehensive user testing and evaluations on
teacher’s, deaf and speech impaired students and regular
students to measure the systems accuracy and dependability of
the data transmitted in real time.
1.3 Scope and Delimitation of the Study
1.3.1 Scope:
The study will focus only on the areas of:
 Design and Development: The study involves on the designing
and the development of an IoT-based Filipino Sign Language
Interpretation system. This includes both properties of hardware
and software components that are necessary for the system’s
operation.
 Users: The target of our system use is both private and public
schools withing the National Capital Region Area. The primary
target is the deaf/mute individuals, the secondary target is the
teachers and the regular students at specified target schools,
and this system can be beneficial for everyone interested in Sign
Language Interpretation. (Needs more time and can be change)
 Filipino Sign Language: The system only generalizes on the
Filipino Sign Language or FSL. It will translate the given
Filipino Sign gestures into a displayed output and in future
comes the audible output.
 Haptics: The system will utilize haptic technology; this is to
allows users to feel and understand information through the
sense of touch and non-verbal gestures.
 User Testing and Monitoring: The system's effectiveness will
be monitored thoroughly using user feedback and performance.

1.3.2 Limitation:
 Language Barrier: The system will only interpret Filipino Sign
Language (FSL). Other sign language such as ASL or the 150
sign languages are not involved in the version of the system.
 Hardware Limitations: The hardware parts of the system will
be limited to a degree because the IoT hardware components
are expensive, and the system will be limited to the capabilities
of hardware used.
  Geographical: The development and implementation and the
testing of the system will be limited to only the National Capital
Region. (May Change)
 Time: The given time to complete the system is 1 year only,
long-term effects and benefits of the sign language may not be
fully explored in the specific study.
 User Proficiency: The system will only depend on the
effectiveness of user proficiency and those who have familiarity
with the system.

1.4 Significance of the Study:

The development of an IoT based Filipino Sign Language

Interpretation System with Haptic Technology can play a vital role in the
rise of the realm of assistive and interactive technology. The system is
design to narrow down communication barriers and generalize the
impairments, in particularly with the deaf/mute community. By fully
resonating the power of Internet of Things (IoT) and Haptic Technology,
the system can interpret and translate FSL into audible and displayed
language. And for the integration of the haptic technology, the system
can enhance the User Experience by allowing users to feel and
understand information through touch and hand gestures and providing
the user with more immersive and interactive way of learning and
understanding the Art of Sign Language.

The systems significance greatly extends beyond the capabilities of

software development because it incorporates the hardware side of the
system. This represents an innovative and creative way of addressing
real-world phenomena and improve the lives of deaf/mute individuals.
And addressing potential of the system as a basis for a sudden change in
society.

For the students, the system can be a powerful tool for teaching
and learning for the deaf/mute students and as well as the normal
students. This can help them communicate with barriers and effectively
communicate with their friends and acquaintances. This can also be a
proponent for interactive learning and immersive activities.

For the Teachers/Professors, the system can assist the teachers to

interact with their deaf/mute students. As well as to serve a purpose of
teaching aid for Filipino Sign Language, thereby promoting the integrity
of the classroom.

For future researchers and developers, the study can provide a

systematic foundation for future research and development in fields such
as IT, Computer Science and Computer Engineering. The field of IoT is
limited to these sources. This study can be inspiring and allow future
students and developers to create more innovative solutions with the use
of IoT and Haptic Technology.
 CHAPTER II
REVIEW OF RELATED LITERATURE

The chapter indicates the ideas and concepts relevant to the

studies and it is discussed to provide the foundation of the proposed
study. An examination of the literature and the studies significant to the
sign language interpretation, haptics, is beneficials to identify the
appropriate methods and procedures to advancing in the proposed
research study and research in assistive communication technologies.

Filipino Sign Language

According to the study of Montefalcon, et al (2021), their study

presents the claims of the accuracy of the sign language which is 92.86%
of a data set of 30 gestures. The paper presents a deep learning model
that can interpret FSL itself. They have used convolutional neural
network model or CNN to take out features from frame by frame of the
video. The secondary model that they have used is the long short-term
memory model or LSTM network to capture a temporal sequence of
gesture in hand. In relevance to this, the study of Disini, eLegal PH
(2020), their study occurs on the mandates the use of domains such as
education, health, justice, and media until the government. Their study
provides articles on the Republic Act No. 11106 or the Filipino Sign
Language Act of 2018, which this declares the Filipino Sign Language or
FSL as the national sign language in relation to the mandate by the
republic act.

Wearable Glove

The study by the ScienceDaily (2020), their article covers the

features and benefits of the wearable tech glove such as flexibility,
durability and low cost. They have also discussed the challenges and
opportunities in the development in the field of Electronics. In relevance
to this study, Chin (2020), their study on the device that connects
wirelessly to an application and converts the gestures into audible and
displayed language, the ASL that they have made can translate gestures
into speech in real time. The device consists of a pair of gloves, sensors
that run the length of each finger and the system sensors pick up the
motions of each finger and translates them into individual letters, words
and even phrases.

Haptic Technology

According to the study of Giri, Maddahi and Zaerinia (2020), the

classifications of the haptics devices are based on the construction and
the functionality such as hand-held, wearable and grounded devices.
Their study also inscribes the current state of haptic technology and the
application of this in other fields. This also discusses major limitations
and challenges of haptics in prospects in this technology. Another study
occurred in relevance to this by Bortone, D’Angelo and De Rossi (2022),
shows that their paper analyzes the structures, functionality and
application and the performance of the haptic device. They categorized
the device based on the wearability level, such as gloves, finger-worn
devices, arm-worn devices, jackets and vests, belts and devices from
head to foot. They have also provided a survey of over 90 haptic devices
developed and described in papers published in 2018 to 2021. Another
study that is in relevance to this study was by Ramones, del-Rio-Guerra
(2023), that they have stated that haptic devices focus on the design,
implementation and the evaluation which provides immersive haptic
feedback on the user. They have also stated that in their paper, that they
presented the recent developments in haptics and their applications in
various fields such as medicine, education, entertainment and industry.

Iterative Model - Software

 The research paper title “Research on Various Software
Development Lifecycle Models” by Khan (2020) discusses various types
of traditional SDLC models, such as, waterfall model, v model and spiral
model. It also describes contemporary models like agile model and RAD
model. The paper highlights the importance of security procedures in
software development life cycle process. The iterative model is one of the
models discussed in this paper, which could be beneficial to the proposed
system as it allows for continuous improvement and iteration until the
final product is complete. In relevance to this study is another research
paper by Sangeeta, et al. (2021), it presents a new failure rate model
centred on iterative software development life cycle process. It aims to
integrate a new modulation factor for incorporating varying needs in
each phase of iterative software development process. It comprises
imperfect debugging with the possibility of fault introduction and
removal of multiple faults in an interval as iterative development of the
software proceeds. This model could be applicable to the proposed
system as it allows for changes and improvements in each iteration,
which could be useful when developing a complex system.

Long-Short Term Memory (LSTM)

The study by Aparna and Geetha (2020), their study proposes sign
language recognition using a convolutional neural network (CNN) and
long short-term memory (LSTM). Their architecture used CNN as a pre-
trained model for feature extraction and is passed to the LSTM for
capturing spatiotemporal information. While another LSTM is used to
increase accuracy. In relevance to this, the study of Rakun et. al (2018),
their study uses LSTM as their model for a sign language system for the
Indonesian language which uses inflectional words, in their study a new
method has developed for separate inflectional words into three feature
vector sets which reduce the amount of features sets used and takes an
entire sequence as input and it does not rely on pre-clustered per frame
data. This suits the system as the SIBI sequence data has a long-term
temporal dependency. With this, the 2-layer LSTM performed its best
with a 95.4% accuracy with root words. Another similar model is lower in
accuracy at 77% due to the difficulty in recognizing prefixes and suffixes
with inflectional words using the combined skeleton image feature sets.

Python

According to the study of Rayhan and Cross (2023), basing on their

research with the language, which has focus on the 3 main components,
application domains, methods and the challenges and opportunities of
their study. This also highlight some of the historical anatomy of Python
alongside if advantages and disadvantages and features with its
community and sources. The study is aimed to promote the use of Python
as a versatile and all-purpose tool for research and tech innovation.
PHPmyAdmin

According to the study by Kumar, et al. (2021), PHPMyAdmin is a

widely used tool for managing MySQL databases in computer
engineering. The researchers delve into the aspects of PHPMyAdmin
including its ability to execute SQL queries, support for multiple
languages and include tools for importing and exporting data. They also
highlight its flexibility through add ons and its open-source nature that
encourages community involvement in development and assistance.
Hardware Technology

The main objective of Sengupta, Mallick and Das (2019) study is to

transform hand gestures based on hardware technology using Arduino.
The hardware devices will translate sign language into speech where
Bluetooth-enabled cloth-driven gloves are being utilized. Five flexible
sensors and one accelerometer are included in the glove. With the use of
these gloves, the users will be able to perform a variety of hand gestures
which will then translate into appropriate statements. The use of Flexible
sensors is crucial in this project along with the accelerometer value of
the hands slant position relative to the ground, the resistance value
change that results from the sensor’s degree of curvature is also
monitored. Through Bluetooth communication, the data is sent to any
smart phone device and is further handled by a microcontroller module.
In relevance to this, the study of El-din and El-Ghany (2020) proposed a
model with a gloved equipped with 5 flex sensors that interface with a
control unit. It converts Arabic sign language and American sign
language to text and speech which is shown through graphic user
interface. Their method uses machine learning to develop an accessible
and user-friendly SL translation system. The system achieved a 95%
recognition rate with static gestures and up to 88% with dynamic
gestures.

Artificial Intelligence (AI)

According to the study of Ahire et. al (2022), their study uses image
processing, computer vision and neural network techniques to construct
a visual Indian sign language interpretation system. To translate sign
language motions into text, they created an AI-based ISL interpreter
where an application is used to recognize local motion or fingerspelling
in ISL and translates it into text and speech in 104 different languages.
The suggested concept uses convolutional neural networks and image
processing techniques to put the 99% validation accuracy model through
training. Additionally, the study of Papastratis et al. (2021) AI sign
language technologies is discussed which covers the methods for
collection of sign language data and recognition that represents the
techniques for identifying and translating sign language. It also covers
the benefits and difficulties of the various AI technologies and examines
the relationships between them.

RA 10173 – Data Privacy Act of 2012

The study of Foronda et al (2023), they have found that many

factors affect the implementation of the RA 10173 on one company like
ALECO namely the limited sources of awareness and education of the law
itself, hindrance of time and resources and mediocre priority of the
certain agenda, the study occurs and have found that most of the
benefactors and implementer have little to no knowledge of some
provisions, which is the role of the DPO or the Data Protection Officer
along with the proceeding compliance of the implementation. In
correlation with this, the study of Doce and Ching (2018), which they
entice on the data collected by them based on qualitative single holistic
study design, along with existing ones that data privacy, compliance, e-
governance which they identified factors affecting the performance of
SUCs or State and University Colleges. In this providing some efforts on
Universities and Colleges in the Philippines to comply with the Data
Privacy law.
 CHAPTER III
RESEARCH DESIGN AND METHODOLOGY

In this chapter, the System and the program flowchart, Data Flow
Diagram, Use Case Diagram, Software Architecture and other project
diagrams are indicated.

These diagrams are important so that the study is readable to

anyone.

Research Design

3.2.1 Data Gathering Technique

The data gathering technique for this study is a combination of both

properties of qualitative and quantitative data methods. The quantitative
data will be collected through surveys, experiments, and focus groups
this is to measure the usability of the IoT-based Filipino Sign Language
Interpretation System with Haptic Technology. The qualitative data will
be indicated by interviews and observations to explore the experiences
and perception of both deaf/mute and the teacher and regular students
who use and does not use the system.

Data Style Source Data Collection Data Analysis

Method Methods
Qualitative Deaf Mute Interviews and Thematic
student, regular Observations Analysis
Quantitative students and Surveys, Descriptive
teachers even Experiments, Statistics and
staffs and Focus Inferential
 Groups
Figure 1. Data Gathering Technique

(The part below can subject to change, please don't change it yet)

3.2.2 Setting of the Study

The setting of the study is in the district 1 Public Schools in Marikina

City, Philippines. The study will be conducted in the 5 of the 8 selected
public schools that have deaf/mute students and non-deaf/mute students
and teachers; the schools are:

 Barangka Elementary School/Barangka National High School

 San Roque Elementary School/San Roque National High School
 Leodegario Victorino Elementary School/Jesus Dela Peña High
School
 Sto Niño Elementary School/Sto Niño National High School
 Sta Elena Elementary School/Sta Elena High School

3.2.3 Subject of the Study

The study subject will be the deaf/mute students, regular students and
the teachers that are enrolled or have assigned regular classes of the
selected Public Schools. The study will use a purposive sampling
technique to select the participants of the following criteria:

 The deaf-mute students must be in the special education section

in elementary, ages 9 to 30, and have voice impairment or
hearing impairments.
 Normal students must be in grades 7-10 in high school, ages 13-
18, and must be a regular student at that specific school.
 Teacher must be licensed and experienced in regular class,
special education teachers are also involved in this study, and
must have familiarity with Filipino Sign Language (FSL).

The study aims to accommodate at least 250 students from all schools in
regular classes, the target for deaf/mute is at least 25 students due to the
shortage of special education courses in each school, and to interview at
least 30 teachers from all schools in the entire study.

Schools Number of Deaf/ Number of Number of

Mute Teachers Regular
Students
Barangka 5 8 50
Elementary
School/Barangk
a National High
School
San Roque 5 5 50
Elementary
School/San
Roque National
High School
Leodegario 5 6 50
Victorino
Elementary
School/Jesus
Dela Peña High
School
Sto Niño 5 6 50
Elementary
School/Sto Niño
National High
School
Sta Elena 5 5 50
Elementary
School/Sta
Elena High
School
Figure 2: Subjects of the Study

3.2.4 Sampling Technique

The sampling technique for this study is on the options of stratified

random sampling technique or cluster sampling technique, this is to
ensure that the participants of this study are representatives of
deaf/mute, regular and teachers of district 1 public schools in Marikina
City. The study will use the following below:

 The study will obtain the list of public schools in all Marikina City
and narrow it down to district 1 only that have possibly deaf/mute
 students and regular students that are currently enrolled in those
schools.
 The study will randomly select 5 out of the 8 from that list.
 The study will obtain the list of deaf or mute students and their
special education instructors each in the selected schools given by
the school administrators.
 The study will classify the deaf/mute according to their ages,
because special education is a separate section from the regular
section. And for the teachers their classification is based on years
of experience, subject area and modes of teaching.

Option 1:

Figure 3: Stratified Random Sampling

Option 2:
 Figure 4: Cluster Sampling

3.2.5 Statistical Treatment of Data:

The statistical treatment of data will use the following:

For the quantitative data, the study will use descriptive statistics to
summarize the data using measures of central tendency (mean, median,
and mode) and the measures of dispersion (standard deviation, range and
variance). This study will also use inferential statistics to test the
hypotheses and compare the groups using T-Test, Chi Square Test and
ANOVA.
 Figure 5: Quantitative Treatment of Data

For the qualitative data, the study will use the thematic analysis
treatment to identify the report, analyzing patterns and themes that
emerge on the data. The study will follow the six steps of the thematic
analysis proposed by Braun and Clarke (2006).

Figure 6: Qualitative Treatment of Data

 CHAPTER IV
REFERENCES

Chin, M. (2020b, July 1). Wearable-tech gloves translate sign language

into speech in real time. UCLA.
[Link]
speech

Disini, eLegal Philippines. (2020). eLegal Philippines. [Link]

primer-on-the-filipino-sign-language-act-of-2018/

Giri, G. S., Maddahi, Y., & Zareinia, K. (2021). An Application-Based

Review of Haptics Technology. Robotics, 10(1), 29.
[Link]

Haptic Devices: Wearability-Based Taxonomy and Literature Review.

(2022). IEEE Journals & Magazine | IEEE Xplore.
[Link]
Montefalcon, M. D., Padilla, J. R., & Rodríguez, R. (2021). Filipino Sign
Language Recognition using Deep Learning. FSLDL.
[Link]

Ramones, A. F., & Del Rio Guerra, M. S. (2023). Recent Developments in

Haptic Devices Designed for Hearing-Impaired People: A Literature
review. Sensors, 23(6), 2968. [Link]

Wearable-tech gloves translate sign language into speech in real time.

(2020b, June 20). ScienceDaily.
[Link]

Khan, N. A. (2020). Research on various software development lifecycle

models. In Advances in intelligent systems and computing (pp. 357–364).
[Link]

Sangeeta, Sitender, Sharma, K., & Bala, M. (2021). New failure rate
model for iterative software development life cycle process. Automated
Software Engineering, 28(2). [Link]

Aparna, C., & Geetha, M. K. (2020). CNN and Stacked LSTM Model for
Indian Sign Language
Recognition. In Communications in computer and information science
(pp. 126–134). [Link]

Rakun, E., Arymurthy, A. M., Stefanus, L. Y., Wicaksono, A. F., & Wisesa,
I. W. W. (2018b).

Recognition of sign language system for Indonesian language using Long

Short-Term Memory

neural networks. Advanced Science Letters, 24(2), 999–1004.

[Link]

Ahire, G. A., Ambavale, P. N., Panchal, S. Y., & Dr. Kale A. (2022). Indian
Sign Language Interpreter. fin_irjmets1652061639.pdf

Papastratis, I., Chatzikonstantinou, C., Konstantinidis, D., Dimitropoulos, K., & Daras, P
(2021). Artificial intelligence technologies for sign language. Sensors, 21(17), 5843.
[Link]
A cost-effective design and implementation of Arduino based sign language interpreter
(2019, March 1). IEEE Conference Publication | IEEE Xplore.
[Link]
Sign Language Interpreter System: An alternative system for machine learning. (2020,
October 24). IEEE Conference Publication | IEEE Xplore.
[Link]
Foronda, S. M., Javier, N., Vigonte, F., & Abante, M. V. (2023).
Implementation of Republic Act 10173 or the Data Privacy Act of 2012 in
Albay Electric Cooperative (ALECO). Social Science Research Network.
[Link]

Doce, P., & Ching, D. (2018, June). RA 10173 and its Challenges to Philippine State
Universities and Colleges’ Compliance Performance: The Case of Mindanao State University
- General Santos City. Research Gate. Retrieved from [Link]
Rayhan, A., & Gross, D. (2023, September). The Rise of Python: A Survey of Recent
Research. Researchgate. Retrieved from [Link]