Expanding Intra-class Difference and Boosting Frame-level Classification for Continuous Sign Language Recognition

Continuous Sign Language Recognition (CSLR) is a typical sequence-to-sequence (seq2seq) task from sign language video frame sequence to gloss sequence. Due to the annotation bottleneck in CSLR, there are usually weakly supervised annotations in existing sign language datasets. Therefore, it is difficult to train a deep learning model for CSLR. To solve the above difficulty, we propose the concept of expanding intra-class difference by expanding original labels and inter-class similarity filtering. And we fuse contextual features and spatial features to boost frame-level classification. We validated the effectiveness of the proposed approaches on three popular CSLR datasets: Chinese Sign Language dataset (CSL), PHOENIX-2014, and PHOENIX-2014-T. Experimental results show that our framework is effective.