AI Video Interview Analysis: Soft Skills & Non-Verbal Signals

A recruiter spends an average of 3-4 hours reviewing a single video interview. With a flow of 50 candidates per week, that's 150-200 hours. Our AI system processes 50 interviews in one hour, providing a detailed report on soft skills and non-verbal signals for each. When a vacancy closes in a month, reviewing 200 recordings takes a week—our AI compresses this process to an hour: it simultaneously analyzes facial expressions, intonations, answer content, and produces a structured report. As a result, the company gets objective metrics of soft skills and non-verbal signals, while HR gains data-driven insights.

According to LinkedIn data, using AI in recruiting reduces time-to-hire by 35%. Our system processes a one-hour interview in 72 minutes—3-4 times faster than manual analysis, and with mass selection (200+ candidates), acceleration reaches 600 times.

How Does AI Analyze Non-Verbal Signals?

The core of the system is a multimodal pipeline: video → frames (1 fps) → face detection via MediaPipe → emotion classifier; audio → Whisper large-v3 (ASR) → transcript → sentiment and speech metric analysis. All modules run in parallel; the final report is assembled in the video length plus 20% processing time. We use emotion recognition based on convolutional neural networks.

import cv2
import numpy as np
import torch
import whisper
from transformers import pipeline
import mediapipe as mp

class VideoInterviewAnalyzer:
    def __init__(self, config: dict):
        # Speech → text: Whisper large-v3
        self.asr = whisper.load_model('large-v3')

        # Text sentiment analysis
        self.sentiment_analyzer = pipeline(
            'text-classification',
            model='blanchefort/rubert-base-cased-sentiment',
            device=0 if torch.cuda.is_available() else -1
        )

        # Facial emotions: MediaPipe + classifier
        self.face_mesh = mp.solutions.face_mesh.FaceMesh(
            max_num_faces=1, refine_landmarks=True
        )
        self.emotion_model = self._load_emotion_model(config['emotion_model'])

        # Speech analysis: rate, pauses, filler words
        self.filler_words_ru = ['э', 'ну', 'это', 'короче', 'типа', 'как бы']

    def analyze_video(self, video_path: str) -> dict:
        cap = cv2.VideoCapture(video_path)
        fps = cap.get(cv2.CAP_PROP_FPS)

        frames = []
        audio_data = []

        # Extract frames (1 per second for emotions)
        frame_idx = 0
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                break
            if frame_idx % int(fps) == 0:  # 1 fps
                frames.append(frame)
            frame_idx += 1
        cap.release()

        # Parallel analysis
        emotion_timeline = self._analyze_emotions(frames)
        transcript = self._transcribe_audio(video_path)
        speech_features = self._analyze_speech(transcript, fps * frame_idx)
        text_analysis = self._analyze_text(transcript['text'])

        return self._compile_report(emotion_timeline, transcript,
                                     speech_features, text_analysis)

    def _analyze_emotions(self, frames: list) -> list:
        timeline = []
        for t, frame in enumerate(frames):
            rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            results = self.face_mesh.process(rgb)

            if results.multi_face_landmarks:
                emotion = self.emotion_model.predict(frame)
                timeline.append({'second': t, 'emotion': emotion})
            else:
                timeline.append({'second': t, 'emotion': 'no_face'})

        return timeline

    def _transcribe_audio(self, video_path: str) -> dict:
        result = self.asr.transcribe(video_path, language='ru',
                                      word_timestamps=True)
        return result

    def _analyze_speech(self, transcript: dict,
                          total_frames: int) -> dict:
        words = [w for seg in transcript.get('segments', [])
                  for w in seg.get('words', [])]

        if not words:
            return {}

        total_duration = words[-1]['end'] if words else 1.0
        words_per_minute = len(words) / (total_duration / 60)

        # Pauses > 2 sec
        long_pauses = []
        for i in range(1, len(words)):
            pause = words[i]['start'] - words[i-1]['end']
            if pause > 2.0:
                long_pauses.append({'start': words[i-1]['end'],
                                     'duration': pause})

        # Filler words
        fillers = [w for w in words
                    if w['word'].strip().lower() in self.filler_words_ru]
        filler_rate = len(fillers) / max(len(words), 1)

        return {
            'words_per_minute': words_per_minute,
            'long_pauses': long_pauses,
            'filler_rate': filler_rate,
            'total_words': len(words)
        }

    def _compile_report(self, emotions: list, transcript: dict,
                          speech: dict, text: dict) -> dict:
        # Emotion distribution
        emotion_counts = {}
        for e in emotions:
            em = e.get('emotion', 'unknown')
            emotion_counts[em] = emotion_counts.get(em, 0) + 1

        dominant_emotion = max(emotion_counts,
                                key=emotion_counts.get) if emotion_counts else None

        # Simplified scoring
        score = 50.0
        if speech.get('words_per_minute', 0) > 100:
            score += 10  # good speech rate
        if speech.get('filler_rate', 1) < 0.05:
            score += 10  # few filler words
        if dominant_emotion in ['happy', 'neutral']:
            score += 10
        if dominant_emotion in ['fear', 'disgust']:
            score -= 10
        if len(speech.get('long_pauses', [])) > 5:
            score -= 10

        return {
            'overall_score': min(100, max(0, score)),
            'emotion_distribution': emotion_counts,
            'dominant_emotion': dominant_emotion,
            'speech_metrics': speech,
            'transcript': transcript.get('text', ''),
            'text_sentiment': text,
            'recommendations': self._generate_recommendations(emotions,
                                                               speech, text)
        }

Emotion Recognition Accuracy: Limitations and Real Numbers

Recognizing emotions in real-world conditions is challenging. Lighting, camera angle, glasses, beard—all reduce accuracy. The JAFFE dataset yields 92%, but that's a lab setting. In practice, we see 68–78% for 7 basic emotions. Therefore, the final decision always rests with the human; the system only highlights anomalies.

What's Included in the Work?

We handle the entire cycle: from requirements gathering to deployment on the client's infrastructure. As a result, you receive:

• Architectural documentation and model cards with accuracy estimates.
• Source code with comments (Python, PyTorch, ONNX for inference).
• Deployment and monitoring setup instructions.
• Training for the HR team on dashboard usage.
• Two weeks of technical support after launch.

Process and Timelines

1. Analysis: Discuss interview corpus, metrics requirements, and ATS integration.
2. Prototyping: Build a pipeline on 5–10 labeled videos, measure p99 latency and FLOPS.
3. Model Training: Fine-tune ASR and emotion classifier for the domain (e.g., IT interviews).
4. Integration: REST API, webhook notifications, adapter for your ATS.
5. Testing: A/B comparison with expert HR evaluation (target correlation > 0.8).
6. Deployment: Containerization (Docker + Kubernetes), GPU optimization (Triton Inference Server).

Our Experience and Results

Our team has completed over 15 computer vision and NLP projects in HR. The average time-to-hire reduction is 70% while maintaining decision quality. Average hiring budget savings reach 40%, which with typical costs of $5,000–$10,000 per position gives $2,000–$4,000 per month. System payback is less than one quarter. We guarantee the accuracy of key metrics at the level specified in the specification. Request a consultation to discuss your task and get demo access to a prototype.

Typical Implementation Mistakes

• Insufficient data labeling: At least 20 labeled interviews are needed for fine-tuning; otherwise, accuracy drops below 60%.
• Ignoring ethical constraints: The system should not be the sole selection criterion.
• Lack of a baseline: Always run an A/B test with expert evaluation to confirm correlation > 0.8.

Ethical Limitations

Important: The system does not replace HR; it provides data-driven insights. Nervousness in an interview ≠ incompetence. We recommend using AI as a first-line filter, with the final decision made by experts. All data is processed anonymously; the model does not store raw videos—only aggregated metrics.

Order the development of an AI video interview analysis system tailored to your requirements—we'll estimate the project in 3 business days. If you need a quick prototype, contact us for a demo.