news
NCSSM students create an app that transcribes fingerspelling with a phone camera
June 24, 2022
Last fall, Ganning Xu ’23, Jolie Duquene ’23, Eric Liu ’23, Jacob Van Meter ’23, and Keshav Varadarajan ’23 won the Congressional App Challenge for North Carolina’s fourth district. They have now continued their streak by placing second in the Lenovo’s Ready, Set, App! competition, an initiative of the North Carolina Business Committee for Education.
Their app, SLTranslate, uses machine learning technology to transcribe the fingerspelling alphabet in real time using the phone’s camera. Users can simply open the app, navigate to the translate screen, and point their camera toward the person signing. The app then displays the letter signed underneath the camera feed with 80% accuracy. All letters from one session are transcribed together in a running series just underneath the most recent letter for easy access.
The app is powered by a k-nearest neighbors (kNN) machine learning model, which takes the image displayed by the camera and matches it to the corresponding letter. It does so by first taking an initial guess, then by setting a timer for half a second. If the model still recognizes it as the same letter after the timer ends, it successfully displays the translation. The students trained this model by pairing it with a database of over 87,000 images of signed letters. Then, they packaged it into an app using JavaScript, React Native, and Expo.
Currently, SLTranslate only supports recognizing individual letters. In the near future they plan to teach it to recognize American Sign Language signs for words, as well as utilizing facial expressions and instituting their own learning management system with interactive practice for anyone who wants to learn to sign.
For most of the group, this was their first time ever developing an app. None of them sign ASL or knew how to fingerspell. So what drew them to designing an app, and specifically to this subject?
Ganning Xu said it was not complicated.
“We just sort of got together one day,” he said. All he and his group knew was that they wanted to make an app, so they began to look for a problem they felt they could solve. They noticed that most high schools do not offer instruction in ASL or fingerspelling.
“Sign language isn’t offered in most schools, but half a million people [use] it. We wanted to make sign language more accessible to people,” Ganning said.
“I think it’s really important that sign language becomes more accessible to people who want to learn it,” said Keshav Varadarajan.
Keshav was the one who got the group together. He previously entered the Congressional App Challenge by himself with Luna, a website which consolidates learning management software into one concise page. Students were able to log in with their various accounts across many different platforms such as Canvas and Blackboard and bring all of their assignments together.
“I already entered once,” he said. “I wanted to do it with Ganning, [Jolie], Eric, and Jacob, so I got them involved too.”
According to the press release by Congressman David Price, their win in the Congressional App Challenge allowed them a chance to show their mock-up video on the U.S. House of Representatives website and inside the U.S. Capitol.
For Lenovo’s Ready, Set, App! challenge, the students were required to develop an Android app to solve a problem within the community. Their second-place victory scored them any Lenovo product worth up to $200.
In the future, they plan to continue to enter coding competitions and to continue supporting SLTranslate. Read more about the project and see its most recent pitch video on their Devpost page here.