Development of an AI System for Emotional Reactions of a Digital Avatar
Digital avatars often look like static talking heads — canned facial expressions and monotone voice break trust. We built an empathic reactive avatar with an emotion pipeline that analyzes user emotions in real-time from voice, facial expression, and text, and synchronously reflects them on the avatar. As a result, engagement increases by 28–35% (1.3–1.4 times) compared to avatars without emotional response. The system supports Unity, Unreal Engine, and browser-based WebRTC stack.
Why Multimodal Approach Isn't a Luxury but a Necessity?
A single channel (only video or only audio) yields up to 30% false positives. Multimodal emotion — three sensors with different noise types — reduces the frequency of uncanny valley to 5% and boosts perceived naturalness to >3.8/5. Bayesian fusion with priority video > audio > text minimizes the impact of interference.
How We Detect User Emotions
The Emotion Input Pipeline consists of three independent channels:
Voice channel: SpeechBrain / audeering/wav2vec2 for audio emotion detection. A 4-class model (neutral, positive, negative, stressed) trained on IEMOCAP achieves ~82% accuracy. An 8-class version (fear, anger, joy, etc.) reaches ~72%. Latency p99 — 200 ms.
Video channel: DeepFace / FER+ for facial expression recognition over WebRTC. MediaPipe FaceMesh extracts 478 keypoints, and the classifier determines the emotion. Frame rate — 30 fps.
Text channel: BERT-based sentiment analysis (CardiffNLP) — context-dependent tone analysis. For example, the phrase "this task is complex" is not marked as negative in a technical context.
How We Fuse Data from Different Channels
Emotion Fusion — a Bayesian algorithm with priority video > audio > text (when all channels are available). Temporal smoothing via exponential moving average with a 2–3 second window prevents jittery state switches. If one channel drops, the system continues working on the remaining ones — this ensures fault tolerance. For experiments and logging we use Weights & Biases, and models are versioned with MLflow. Bayesian fusion in a dual audio-video system (extensible with text) provides a stable result.
How the Avatar Expresses Emotions
Emotion Output transforms the recognized emotion into three streams:
Face: FACS-based blend shapes via emotion-to-AU mapping. For example, "joy" → AU6 (cheek raise) + AU12 (lip corner pull) + AU25 (lips part). Intensity scales proportionally to emotion strength. More on FACS.
Voice: ElevenLabs TTS with stability and similarity parameters — fine-tuning expressiveness of synthesis in real time. Timbre, tempo, and volume can be adapted to the emotional state.
Gestures and gaze: A library of pre-recorded gesture clips triggered by emotion state. Positive → open gestures, more eye contact; tension → reduced gesticulation, gaze aversion during conflicting content. Gestures are synchronized with speech via timestamps.
LLM emotions — optional generation of empathetic responses, e.g., via GPT-4o, to create a deeper connection with the user.
Comparison with Typical Solutions
Multimodal approach gives a significant boost in key metrics. Our system outperforms standard solutions in engagement by 1.3–1.4 times (28–35%).
| Parameter | Our System | Typical Avatar Without Emotions |
|---|---|---|
| User engagement | +28–35% (1.3–1.4x) | baseline |
| Naturalness (5-point) | >3.8 | ~2.5 |
| Uncanny valley frequency | <5% of interactions | ~15% |
| Modalities | 3 (audio, video, text) | 1–2 |
| Context adaptivity | yes | no |
What's Included in the Emotion Pipeline Work
When you order development, we provide:
- Architecture document: description of chosen models, fusion algorithm, integration points.
- API documentation: endpoints for detection, fusion, and animation; example requests/responses.
- Access to model registry: versioned models on MLflow, possibility for further fine-tuning.
- Integration package: SDK for Unity/Unreal Engine, WebRTC client, sample code in Python/JavaScript.
- Test report: A/B test results, metrics for latency p99, accuracy, naturalness.
- 3-month support: consultations, bug fixing, optimization for your infrastructure.
Development Stages
- Analytics and detection calibration — model selection for your data, threshold tuning (2–3 weeks).
- Fusion and emotion-to-AU mapping — implementation of Bayesian algorithm in PyTorch/TensorFlow, temporal smoothing setup (3–4 weeks).
- Integration with avatar and TTS — connection to Unity, Unreal Engine, or WebRTC; ElevenLabs calibration (3–4 weeks).
- Testing and user study — naturalness assessment, A/B test, edge case documentation (2–3 weeks).
Total duration — from 10 to 14 weeks depending on integration complexity and customization scope. The cost is calculated individually based on project audit. This approach is significantly more cost-effective than building an in-house solution from scratch.
Metrics and Edge Cases
| Metric | Value |
|---|---|
| Emotion detection accuracy (4 classes) | ~82% |
| Perceived naturalness (5-point scale) | >3.8/5 |
| User engagement (vs. non-emotional avatar) | +28–35% |
| Uncanny valley incidents | <5% of interactions |
Sarcasm, cultural differences in emotional expression, and mixed emotions reduce accuracy. For professional scenarios (psychotherapy, HR) we recommend human-in-the-loop: the system flags uncertain states for an operator.
We are ready to discuss your project. Get a consultation — we will analyze your use cases and propose an optimal turnkey solution. 5+ years of experience in AI, 15+ avatar projects. Contact us for a detailed commercial proposal.