BotSigner

BotSigner is a desktop app that recognizes American Sign Language signs from a webcam. You record a short clip of a single sign, and it shows its ten best guesses from a vocabulary of 250 signs.

I built it in 8th grade (spring 2023) around the model I entered in Kaggle's Google - Isolated Sign Language Recognition competition, where it placed 72nd of 1,165.

How it works

1. Landmarks. MediaPipe Holistic extracts face, hand, and pose landmarks from each webcam frame (engine/data.py).
2. Preprocessing. The model keeps the lip landmarks plus the landmarks of the dominant hand and arm. Clips are mirrored so the dominant hand is always on the same side, frames where that hand isn't visible are dropped, and every clip is resampled to 64 frames (PreprocessLayer in engine/model.py).
3. Model. Each landmark group (lips, hand, arm) gets its own learned embedding. The embeddings are combined with learned weights, and a two-block transformer encoder classifies the sequence into one of 250 signs. It's written in TensorFlow/Keras, and the trained weights are in models/model.h5.
4. App. A PyQt5 window shows the camera feed with the detected landmarks drawn on it and lists the top 10 predictions when you stop recording (app.py).

Running it

BotSigner was written on Windows with Python 3.10. tensorflow-addons requires TensorFlow 2.15 or earlier.

pip install "tensorflow<2.16" tensorflow-addons mediapipe opencv-python PyQt5 pandas pyarrow
python app.py

Credits

The preprocessing and model architecture build on public Kaggle notebooks by Mark Wijkhuizen and Henck23. For my submission I widened the embedding and transformer layers, trained the model, and built the app around it. The sidebar UI in sidebar_ui.py comes from CodeQuestions/PyQt5-Video-Book.