Deafblind people face remarkable challenges in communicating, because of their severe disability. The only way to interact with other people is the usage of the tactile sign language, which consists in understanding the sign language putting their hands on the signer’s hands. But this approach works only when the signers are in the same place. The aim of this project is to reduce the gap between deafblind people and the other ones, giving them the capability to communicate remotely. By collecting images with two cameras, the signer’s body is tracked with a deep neural network. The extracted coordinates of the body parts (chest, shoulders, elbows, wrists, palms and fingers) are used to move one or more robotic arms. The deafblind person can put his hands on the robots to understand the message delivered by the person on the other side. The entire system is based on a cloud architecture.

Empowering Deafblind Communication Capabilities by Means of AI-Based Body Parts Tracking and Remotely Controlled Robotic Arm for Sign Language Speakers

Massari L.;Terruso G.;Filosa M.;Tonietti G.;Oddo C. M.;Fanucci L.
2020-01-01

Abstract

Deafblind people face remarkable challenges in communicating, because of their severe disability. The only way to interact with other people is the usage of the tactile sign language, which consists in understanding the sign language putting their hands on the signer’s hands. But this approach works only when the signers are in the same place. The aim of this project is to reduce the gap between deafblind people and the other ones, giving them the capability to communicate remotely. By collecting images with two cameras, the signer’s body is tracked with a deep neural network. The extracted coordinates of the body parts (chest, shoulders, elbows, wrists, palms and fingers) are used to move one or more robotic arms. The deafblind person can put his hands on the robots to understand the message delivered by the person on the other side. The entire system is based on a cloud architecture.
2020
978-3-030-37276-7
978-3-030-37277-4
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11382/540811
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
social impact