Which Approach Should You Choose for Text Summarization?
When summarizing a news feed with a volume of 10,000 articles per day, developers face a dilemma: extraction does not produce coherent text, while abstraction risks hallucinations. The solution is to combine methods and tune them to your data.
Key Challenges and Solutions
A major challenge is hallucinations in abstractive models. In legal documents, the cost of an error can reach millions, so we prefer a hybrid: extraction for safe fact extraction combined with fine-tuned T5 for the final text. Another issue is the limited context window. Documents exceeding 128k tokens must be chunked with 10% overlap and hierarchical summarization applied: first by sections, then a summary. Performance is also critical: under a load of 2000 req/s, latency p99 should not exceed 500 ms. We achieve this through vLLM, ONNX Runtime, and INT8 quantization. We also incorporate RAG for grounding summaries in external knowledge.
Our Approach: Stack and Case Study
In one project for a corporate news aggregator, we configured a hybrid system. The first pass was extractive sentence selection via TextRank with a similarity threshold of 0.7. The second was generating a summary from the selected sentences using IlyaGusev/rut5-base-absum, fine-tuned with LoRA on the company's news. Inference was accelerated via ONNX Runtime with dynamic quantization — FLOPS rose to 150 TFLOPS on a single A100. Result: ROUGE-1 0.52, BERTScore 0.68, latency p99 at 320 ms. This reduced reading time by 60%, saving the client approximately $15,000 annually in analyst hours.
Reasons for Errors in Abstractive Summarization
Models like GPT-4o are prone to hallucination when context is vague or contradictory. To minimize this, we use few-shot prompts with examples from your domain and add a constraint: "Do not use facts not present in the text." For critical scenarios, chain-of-thought is effective — the model first extracts facts and then formulates a summary. It is also important to consider that summarization quality directly depends on the representativeness of training data: fine-tuning on 10,000 documents from your domain can improve BERTScore by 0.05. In our tests, a hybrid scheme with a fine-tuned T5 outperforms pure extraction on ROUGE-1 by a factor of 1.3.
Effective Mitigation of Hallucinations
Hallucinations are reduced by fine-tuning on target data, using low-rank LoRA, and post-processing with fact verification. In the prompt, include strict instructions not to add information. For critical domains (legal, medical), extractive approaches are recommended.Choosing Between Extractive and Abstractive Summarization
| Scenario | Recommendation |
|---|---|
| News texts, speed important | TextRank or rut5-base-absum |
| Legal/medical documents | Extractive (no hallucinations) |
| Business reports, quality important | GPT-4o with Map-Reduce |
| High load (>100 req/s) | Distilled T5 + ONNX |
Comparison of Approaches by Criteria
| Criteria | Extractive | Abstractive |
|---|---|---|
| Hallucinations | None | Risk exists |
| Text coherence | Low | High |
| Data requirements | None | Requires fine-tuning |
| Inference speed | High | Medium (with ONNX – high) |
| Latency p99 | <50 ms | <500 ms (optimized) |
Process
- Requirements analysis: content type, token volume, success metrics (ROUGE, BERTScore, latency).
- Architecture selection: extractive, abstractive, or hybrid.
- Training/fine-tuning: LoRA fine-tuning for abstractive model.
- Integration: REST/gRPC API based on Docker image.
- Testing: ROUGE/BERTScore + A/B test on live data.
- Deployment: vLLM, ONNX, monitoring via Weights & Biases.
Estimated Timelines and Cost
Basic extractive summarization is set up in 3 working days and starts from $2,000. Abstractive with fine-tuning and production-ready integration — from 2 to 4 weeks, starting at $5,000. The cost is calculated individually: contact us — we will assess your project.
What's Included
- Architectural documentation (model card, pipeline diagram).
- Model code in a Docker image with ONNX runtime.
- Integration test (sample API call).
- Access to MLflow monitoring.
- MLOps pipeline setup.
- Team training (2-hour webinar with case studies).
- 3-month support guarantee.
Optimizing Latency for Production
An optimized ONNX model runs 2-3 times faster than base PyTorch. For high loads, we use vLLM with continuous batching — this reduces latency p99 to 300 ms at 1000 req/s. Order a pilot project — we will conduct an A/B test on your data.
Our team has over 5 years of experience, completed 5+ summarization projects, and served 10+ enterprise clients. We have been on the market for over 5 years and guarantee results that meet your metrics. Get a consultation — write to us.