EASL

Abstract

Large language models have revolutionized sign language generation by automatically transforming text into high-quality sign language videos, providing accessible communication for the Deaf community. However, existing LLM-based approaches prioritize semantic accuracy while overlooking emotional expressions, resulting in outputs that lack naturalness and expressiveness.

We propose EASL (Emotion-Aware Sign Language), a multi-emotion-guided generation architecture for fine-grained emotional integration. We introduce emotion-semantic disentanglement modules with progressive training to separately extract semantic and affective features. During pose decoding, the emotional representations guide semantic interaction to generate sign poses with 7-class emotion confidence scores, enabling emotional expression recognition.

Experimental results demonstrate that EASL achieves pose accuracy superior to all compared baselines by integrating multi-emotion information and effectively adapts to diffusion models to generate expressive sign language videos.

Key Features

Methodology

Architecture Overview

EASL employs a dual-module architecture with two core components:

Figure 1: EASL architecture showing DESE and EGSID modules with three-phase progressive training

DESE (Disentangled Emotion-Semantic Encoder)

Uses gated attention mechanisms to separate semantic and emotional representations from input text, extracting:

• Semantic representations (H): Frame-wise features for sign pose generation
• Emotional representations (E): Emotion-specific features for expression control

EGSID (Emotion-Guided Semantic Interaction Decoder)

Leverages emotional features to guide semantic decoding through emotion-aware multi-head attention, generating:

• Pose sequences (P): High-quality keypoint sequences for sign language
• Emotion confidence scores (Ec): 7-class emotion probabilities per frame

Experimental Results

Visual Comparison

Figure 2: Comparison showing EASL's ability to generate emotionally expressive sign language poses

Training Analysis

We evaluate emotion-semantic disentanglement effectiveness using cosine similarity between learned representations and BERT embeddings. The figure below illustrates the progressive training process across three phases.

Figure 3: Emotion similarity convergence across training phases, demonstrating effective disentanglement of semantic and emotional representations

Main Contributions

1. A multi-emotion guided semantic disentanglement framework that decouples semantic and emotional features, effectively adapting to diffusion models for emotionally expressive sign language videos
2. An emotion-semantic disentanglement architecture with DESE for separating representations and EGSID for emotion-guided semantic interaction
3. A three-phase progressive training strategy with parameter freezing that achieves integration of emotional expression with sign language generation
4. State-of-the-art pose accuracy on benchmark datasets with effective multi-emotion integration

Code & Resources

Authors

Acknowledgements

This research is supported by the National Key R&D Program of China (Grant No. 2021YFF0901502).

Citation

@inproceedings{zhao2026easl,
  title={EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generation},
  author={Zhao, Yanchao and Zhu, Jihao and Liu, Yu and Chen, Weizhuo and Yang, Yuling and Peng, Kun},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2026}
}