Sign Language Detection Based On Sensor Using Neural Network Method

Sevia Indah Purnama; Mas Aly Afandi; Muhammad Nur Al-Majid

doi:10.52435/complete.v5i2.638

Authors

Sevia Indah Purnama Center of Excellence Healthcare and Wellbeing Technology, Universitas Telkom, Banyumas, Indonesia
Mas Aly Afandi Center of Excellence Healthcare and Wellbeing Technology, Universitas Telkom, Banyumas, Indonesia https://orcid.org/0000-0003-4821-8191
Muhammad Nur Al-Majid Electrical Engineering Study Program, Universitas Telkom, Banyumas, Indonesia

DOI:

https://doi.org/10.52435/complete.v5i2.638

Keywords:

Disabilities, Artificial Neural Network, Sign Language

Abstract

Sign language is a way of communication for people with hearing and speech impairments. Sign language is generally done using hand gestures to articulate sentences. Different countries use different sign languages, increasing the complexity of sign language recognition. Communication between non-disabled and disabled people is difficult because non-disabled people do not learn sign language. This research aims to develop a device that can translate sign language specific to telling body conditions so that people without disabilities can understand. The device uses sensors to determine hand movement patterns to recognize sign language. The sign language recognized in this study focuses on body conditions and diseases such as asthma, cough, dizziness, depression, and tonsils. The results of the test show that the Artificial Neural Network (ANN) model has good performance with 98% accuracy value, 98% precision value, 98% recall, and 98% F1-Score. The test was conducted after the model was implanted and used to perform sign language testing. The parameter value of the confusion matrix shows high results. It can be concluded that the model created can be used to translate sign language to express the condition of tonsils, cough, depression, dizziness, and asthma.

References

L. Andriyani, “Pemenuhan Hak Disabilitas Perkotaan Dalam Kesetaraan Akses Ekonomi: Pemenuhan Hak Disabilitas Perkotaan Dalam Kesetaraan …,” Sosio Inf. Kaji. Permasalahan Sos. …, vol. 8, no. 2, pp. 119–130, 2022.

R. D. Agustanti, B. Waluyo, and D. A. Ramadhani, “Upaya Peningkatan Pengetahuan Dan Pemahaman Hak-Hak Penyandang Disabilitas Di Bidang Ketenagakerjaan Atas Dasar Persamaan Hak,” JMM (Jurnal Masy. Mandiri), vol. 6, no. 5, pp. 3776–3789, 2022, doi: 10.31764/jmm.v6i5.10237.

A. Of et al., “Advokasi Pekerjaan Sosial Terhadap Diskriminasi Pada Kaum Penyandang Disabilitas Di Dunia Kerja,” J. Soc. Work Soc. Serv., vol. 3, no. 2, pp. 130–138, 2022.

A. Nugroho, R. Setiawan, A. Harris, and Beny, “Deteksi Bahasa Isyarat Bisindo Menggunakan Metode Machine Learning,” J. Process., vol. 18, no. 2, Nov. 2023, doi: 10.33998/processor.2023.18.2.1380.

R. H. Alfikri, M. S. Utomo, H. Februariyanti, and E. Nurwahyudi, “Pembangunan Aplikasi Penerjemah Bahasa Isyarat Dengan Metode Cnn Berbasis Android,” J. Teknoinfo, vol. 16, no. 2, p. 183, 2022, doi: 10.33365/jti.v16i2.1752.

J. Li et al., ""SignRing: Continuous American Sign Language Recognition Using IMU Rings and Virtual IMU Data,"" Proc. ACM Interactive, Mobile, Wearable Ubiquitous Technol., vol. 7, no. 3, pp. 1–29, Sep. 2023, doi: 10.1145/3610881.

S. A. Khomami and S. Shamekhi, ""Persian sign language recognition using IMU and surface EMG sensors,"" Measurement, vol. 168, p. 108471, Jan. 2021, doi: 10.1016/j.measurement.2020.108471.

Y. Gu, C. Zheng, M. Todoh, and F. Zha, ""American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions,"" Front. Neurosci., vol. 16, Jul. 2022, doi: 10.3389/fnins.2022.962141.

R. Setiawan, H. H. Triharminto, and M. Fahrurozi, “Gesture Control Menggunakan IMU MPU 6050 Metode Kalman Filter Sebagai Kendali Quadcopter,” Pros. Semin. Nas. Sains Teknol. dan Inov. Indones., vol. 3, no. November, pp. 411–422, Dec. 2021, doi: 10.54706/senastindo.v3.2021.133.

W. T. Wahyudi, K. Karyanto, and R. M. Antosia, “Rancang Bangun Alat Resistivitas Berbasis Arduino Menggunakan Modul ACS712 dan ADS1115,” J. Tek., pp. 1–7, 2021.

E. W. Pratama and A. Kiswantono, ""Electrical Analysis Using ESP-32 Module In Realtime,"" JEECS (Journal Electr. Eng. Comput. Sci., vol. 7, no. 2, pp. 1273–1284, 2023, doi: 10.54732/jeecs.v7i2.21.

R. Dastres and M. Soori, ""Artificial Neural Network Systems,"" Int. J. Imaging Robot., vol. 21, no. 2, pp. 13–25, 2021.

I. Akil, “Komparasi Fungsi Aktivasi Neural Network Pada Data Time Series,” Inti Nusa Mandiri, vol. 18, no. 1, pp. 78–83, 2023.

M. A. Hanin, R. Patmasari, and R. Y. Nur, ""Sistem Klasifikasi Penyakit Kulit Menggunakan Convolutional Neural Network ( Cnn ) Skin Disease Classification System Using Convolutional Neural Network ( Cnn ),"" e-Proceeding Eng., vol. 8, no. 1, pp. 273–281, 2021.

Alfandi Safira and F. N. Hasan, “Analisis Sentimen Masyarakat Terhadap Paylater Menggunakan Metode Naive Bayes Classifier,” Zo. J. Sist. Inf., vol. 5, no. 1, pp. 59–70, 2023, doi: 10.31849/zn.v5i1.12856.

Lukman Priyambodo et al., “Klasifikasi Kematangan Tanaman Hidroponik Pakcoy Menggunakan Metode SVM,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 6, no. 1, pp. 153–160, Feb. 2022, doi: 10.29207/resti.v6i1.3828.

A. Ridhovan and A. Suharso, “Penerapan Metode Residual Network (Resnet) Dalam Klasifikasi Penyakit Pada Daun Gandum,” JIPI (Jurnal Ilm. Penelit. dan Pembelajaran Inform., vol. 7, no. 1, pp. 58–65, 2022, doi: 10.29100/jipi.v7i1.2410.