Extracting Keypoints and Training Sequence Models for Sign Language Translation

This article discusses the data pipeline and training process for a real-time sign language translation system. It covers keypoint extraction using MediaPipe, normalization techniques, and the use of Transformer Encoder models with CTC loss for continuous sign language recognition.

Details

The article describes a multi-stage pipeline for building a real-time sign language translation system. It starts by introducing the key public datasets used to train the models, including WLASL, RWTH-PHOENIX-2014T, and How2Sign. To extract meaningful features from the raw video data, the authors use Google's MediaPipe Holistic framework to detect and extract 3D keypoints for the hands, pose, and face. This reduces the input dimensionality from millions of pixels to a 1,662-dimensional vector per frame, making the data more manageable for the machine learning models. To ensure the model is robust to variations in camera positioning, the authors implement a shoulder-based normalization technique that translates all keypoints relative to the midpoint of the shoulders. For the temporal sequence model, the authors choose a Transformer Encoder architecture, which excels at modeling long-range dependencies in the sign language sequences. They train this model using Connectionist Temporal Classification (CTC) loss, which allows the model to handle continuous sign language without needing explicit segmentation of the video. The goal is to optimize the end-to-end pipeline for real-time inference on consumer hardware, rather than relying on high-end GPUs.