Sign to Sentence: Translating Sign Languages Gestures into Fluent Natural Language Using NLP techniques
Disciplines
Other Computer Sciences
Abstract (300 words maximum)
The communication gap between deaf and hearing individuals remains a major accessibility challenge due to the limited understanding of American Sign Language (ASL) among the general population. While computer vision models have made progress in recognizing ASL gestures, the generated English translations often lack fluency, coherence and grammatical accuracy. These issues stem from the structural differences between ASL and English, as well as limitations in the gloss-based translation approaches. We develop a gloss-free, NLP-driven post-processing module that refines first-pass English sentences generated by ASL recognition systems. The model is fine tuned on the How2Sign dataset to enhance linguistic structure, context, and readability in translated outputs. By focusing on semantic fluency, our model approaches to improve the naturalness of ASL to English translation.
Use of AI Disclaimer
no
Academic department under which the project should be listed
CCSE – Computer Science
Primary Investigator (PI) Name
Md Abdullah Al Hafiz Khan
Sign to Sentence: Translating Sign Languages Gestures into Fluent Natural Language Using NLP techniques
The communication gap between deaf and hearing individuals remains a major accessibility challenge due to the limited understanding of American Sign Language (ASL) among the general population. While computer vision models have made progress in recognizing ASL gestures, the generated English translations often lack fluency, coherence and grammatical accuracy. These issues stem from the structural differences between ASL and English, as well as limitations in the gloss-based translation approaches. We develop a gloss-free, NLP-driven post-processing module that refines first-pass English sentences generated by ASL recognition systems. The model is fine tuned on the How2Sign dataset to enhance linguistic structure, context, and readability in translated outputs. By focusing on semantic fluency, our model approaches to improve the naturalness of ASL to English translation.