Toward Robust Classification of Non-Manual Signals in Korean Sign Language Under Extreme Class Imbalance

초록

Non-manual signals, such as facial expressions and eye gaze, play a crucial role in sign language communication, comparable to manual signals. However, previous studies in sign language processing have primarily focused on manual signals and often treat non-manual signals only as auxiliary cues for improving recognition or translation performance. Furthermore, most existing sign language datasets lack explicit annotations for non-manual signals due to the high cost of manual labeling, which limits systematic analysis and independent modeling of these signals. This limitation remains an ongoing challenge in sign language research. To address this gap, we conduct a systematic study on non-manual signal classification using the Korean-Korean Sign Language Parallel Corpus 2022 (Version 1.0), one of the few publicly available datasets that provides structured annotations for non-manual signals. However, the dataset suffers from severe class imbalance among annotated non-manual signal labels, making effective learning difficult-particularly for underrepresented classes. To address this problem, we propose an improved framework that systematically analyzes and classifies non-manual signals using long-tail learning techniques, including loss reweighting, resampling, and focal-based optimization. Experimental results show that the proposed approach improves the Macro-F1 score from 0.6065 to 0.7610 compared with the baseline model, achieving substantial gains for minority classes. We further demonstrate that the model can estimate the temporal occurrence of non-manual signals without explicit frame-level annotations, enabling soft-label generation for unlabeled data. These findings provide a practical direction for reducing annotation costs and supporting scalable analysis of non-manual signals in sign language understanding.

키워드