AI Call Scoring System for Automated Operator Evaluation

Real case from our practice: our client — a retail chain with 300 calls per day. The QA team managed to listen to only 15% (45 calls). The missing 85% contained script violations, drops in empathy, and missed sales opportunities. Losses — up to 8% of revenue monthly (average 500k rub). We designed an AI system based on GPT-4o and Whisper that provided 100% coverage, increased scoring accuracy to 92% (compared to a human assessor), and reduced turnaround time by 70%. Savings amounted to over 400k rub monthly, with a payback period of 2–3 months.

Call scoring system (by definition Wikipedia) assigns a numeric score to each call based on a standardized methodology, creates operator rankings, and detects patterns requiring corrective training. The evaluation is not binary (ok/not ok) but multidimensional: greeting, hold, farewell, empathy, problem understanding, solution accuracy, GDPR compliance, AHT efficiency, and FCR.

What problems does AI scoring actually solve?

1. Selective control. A human assessor evaluates 10–20% of calls, the rest are a black box. AI processes 100%: every call gets a full scorecard. 2. Bias. Fatigue, subjective perception, different interpretation of criteria. LLM consistently applies the same methodology to all calls. 3. Lack of trends. Manual evaluation doesn't provide aggregate metrics. We build dashboards with moving averages, weekly/monthly trends, and heatmaps by violation type.

How we do it: stack and case

For evaluation, we use GPT-4o with response_format=json_object. The prompt contains all criteria with weights, an instruction "rate from 0 to 10" and a requirement to explain each rating. We apply fine-tuning (LoRA) to adapt to the specifics of operator speech. For transcription, we use Whisper with a target WER below 10%. In complex cases, we incorporate RAG with a knowledge base of scripts and frequent questions.

Example scorecard — Pydantic model:

from pydantic import BaseModel from typing import Optional class CallScorecard(BaseModel): call_id: str operator_id: str duration_seconds: float # Compliance greeting_score: float # 0-10 hold_procedure_score: float # 0-10 farewell_score: float # 0-10 gdpr_compliance: float # 0-10 # Quality problem_understanding: float # 0-10 solution_accuracy: float # 0-10 empathy_score: float # 0-10 # Efficiency aht_relative: float # 0-10 (relative to target AHT) first_call_resolution: float # 0 or 10 # Sales/Upsell (if applicable) offer_made: Optional[float] = None offer_quality: Optional[float] = None @property def total_score(self) -> float: scores = [ self.greeting_score * 0.10, self.hold_procedure_score * 0.05, self.farewell_score * 0.05, self.gdpr_compliance * 0.10, self.problem_understanding * 0.20, self.solution_accuracy * 0.25, self.empathy_score * 0.15, self.aht_relative * 0.05, self.first_call_resolution * 0.05, ] return round(sum(scores), 1) 

Automatic scoring via LLM:

async def score_call_llm(transcript: dict) -> CallScorecard: full_dialog = format_dialog(transcript["turns"]) response = await client.chat.completions.create( model="gpt-4o", messages=[{ "role": "system", "content": """You are a service quality evaluation expert. Rate the call on each criterion from 0 to 10. Be objective, base your rating only on the text. Return JSON with fields: greeting_score, hold_procedure_score, farewell_score, gdpr_compliance, problem_understanding, solution_accuracy, empathy_score, first_call_resolution. For each field, add comment_FieldName with explanation.""" }, {"role": "user", "content": full_dialog[:6000]}], response_format={"type": "json_object"} ) data = json.loads(response.choices[0].message.content) return CallScorecard( call_id=transcript["call_id"], operator_id=transcript["operator_id"], duration_seconds=transcript["duration"], **{k: v for k, v in data.items() if not k.startswith("comment_")} ) 

To see how this works on your data, contact us—we'll send demo access.

How to set up a scorecard in 5 steps

1. Define call types (inbound/outbound, sales/support).
2. Select criteria from the library (15 base, custom can be added).
3. Assign weights to each criterion (sum to 1.0).
4. Set threshold values (e.g., AHT no more than 300 s).
5. Run a pilot on 100 calls and adjust the prompt based on results.

Comparison: AI vs human — who is more accurate?

Why does AI work more accurately than a human?

LLMs don't get tired, don't skip calls due to lack of time, and apply the same criteria to all dialogues. In our A/B test, AI showed 92% consistency with the control group (2 QA managers, 100 calls), while consistency between two assessors was 78%. This confirms that automated evaluation is not only faster but also more objective.

What's included in the work

• ETL architecture for call transcription (Whisper, Russian language recognition models).
• Integration with ACD/CRM — receiving calls and enriching with customer data.
• Scorecard of 15 criteria — weight customization for your scenario.
• LLM assistant — generating scorecard with explanations.
• Dashboard of ratings and trends — Grafana or React.
• Calibration and monitoring — automatic tracking of AI vs human correlation.

Process overview

1. Analytics and audit — review current evaluation criteria, scripts, call types.
2. Scorecard design — weights, AHT norms, FCR boundaries.
3. Integration and labeling — connect to telephony, gather historical recordings.
4. Training and calibration — run baseline, adjust prompt, fine-tune model if needed (LoRA).
5. A/B testing — compare AI with manual assessments.
6. Deployment and dashboards — deploy inference, connect alerts (Telegram, Slack).

Estimated timelines

Cost is calculated individually based on call volume, required latency, and need for fine-tuning. Request a consultation — we'll send a sample scorecard and estimate.

Calibration: how we maintain accuracy

Periodically we compare AI scores with manual QA manager scores. Target: Pearson correlation > 0.85. If correlation drops, we initiate recalibration on a fresh labeled dataset. The process includes:

• Collect new transcripts (100 calls)
• Labeling by two QA managers
• Training a LoRA adapter based on discrepancies
• A/B test the following week

Our team's experience — 5+ years in NLP and 30+ call center projects. We guarantee transparency: you get full access to prompts, weights, and evaluation logs. Request demo access to see results on your data.