SignSpeak AI
Real-time sign language fingerspelling translation
About
Browser application that reads sign language fingerspelling from a webcam and turns it into text and speech in real time. It covers 61 classes across two alphabets, 29 in American Sign Language and 32 in Arabic Sign Language, using eight computer vision capabilities against a coursework minimum of four.
Two models sit behind it: a landmark MLP trained from scratch on hand keypoints, and a fine-tuned MobileNetV3. A voting buffer requires 8 of the last 10 frames to agree at 0.85 confidence before a letter is emitted, which is what stops the output flickering between neighbouring letters mid-sign.
The honest result is split. American Sign Language reached 99.30% on 1421 samples at 23.65 FPS on CPU with 42.28ms mean latency. The Arabic arm stayed a proof of concept at 40% on 10 samples, because MediaPipe fails to extract usable landmarks from the 32x32 grayscale source images, so most Arabic examples were dropped before training. An ablation removing the depth coordinate cost 4.05 points of accuracy.
This was a five-person team project. My contribution was the technical report and documentation, plus a shared share of the architecture, the vision pipeline, both training scripts, and integration.
Key Features
- 61 classes across two sign alphabets, 29 ASL and 32 ArSL
- Eight computer vision capabilities in one pipeline
- Custom-trained landmark MLP plus a fine-tuned MobileNetV3
- Voting buffer of 8 of 10 frames at 0.85 confidence to stabilise output
- 23.65 FPS on CPU, 42.28ms mean inference latency
- Text and speech output in the browser
Architecture
A Next.js frontend captures webcam frames and streams them to a FastAPI service. MediaPipe extracts hand landmarks, which feed the custom MLP classifier, while a fine-tuned MobileNetV3 handles image-based classification. Predictions pass through a temporal voting buffer before being committed to the output string, then to speech synthesis. Both models were trained and evaluated separately per alphabet.