Implementing Prompt Engineering for an AI System
Deployed an LLM in a support chatbot and got 40% hallucinations? A familiar situation. One of our clients, a fintech startup, spent a month developing a chatbot for credit consultations. A junior wrote the prompt in an hour—result: 40% of responses contained incorrect data on interest rates and terms. In two weeks, we designed a multi-level prompt with anti-hallucination instructions, chain-of-verification, and few-shot examples. Hallucinations dropped to 3%, accuracy increased from 55% to 94%, and p95 latency halved. In 5 years of working with AI systems, we have developed an approach that delivers predictable p95 latency and >95% accuracy under production loads.
Prompt Engineering is the discipline of designing input data for LLMs to obtain predictable, high-quality results. It includes structuring queries, managing context, selecting techniques (CoT, Few-Shot, ReAct), tuning parameters, and iterative calibration.
Why Standard Prompts Fail in Production
A common mistake: prompts written by a developer in ten minutes. They give an acceptable answer on 2–3 manual tests but fail under real load. Main issues:
- Hallucination—the model makes up things not in context. Without anti-hallucination instructions, up to 30% of responses contain false facts.
- Context sensitivity—a small wording change completely alters the response. Temperature and top-p not tied to the task type create irreproducibility.
- Lack of error handling—when data is missing, the model does not say "I don't know" but invents an answer.
We eliminate these problems through structured templates, verification, and A/B tests.
How to Reduce the Hallucination Percentage in LLMs?
Key technique: chain-of-verification—the model first generates an answer, then checks each claim against the context. We add a system prompt that prohibits inventing and introduces a confidence threshold. Additionally, we use few-shot examples with edge cases. In production, this reduces the hallucination rate from 20% to 2–3%. According to OpenAI recommendations, structured prompts cut hallucinations by 30–50%.
ANTI_HALLUCINATION_ADDENDUM = """ IMPORTANT: Only answer based on the provided context. If the information is not in the context, say: "I have no data on this question." Do not guess or make assumptions. If confidence is < 80%, indicate the degree of uncertainty. """ async def answer_with_verification(question: str, context: str) -> dict: answer = query_llm( f"Context:\n{context}\n\nQuestion: {question}", system=f"You are an analyst. {ANTI_HALLUCINATION_ADDENDUM}", ) verification = query_llm( f"Original answer: {answer}\n\nQuestion: Are all statements in the answer confirmed by the context? Answer as JSON: {{\"verified\": bool, \"unsupported_claims\": [...]}}", temperature=0, ) return {"answer": answer, "verification": json.loads(verification)} How We Design Industrial Prompts
The process starts not with code but with analyzing expectations: what the prompt should do, boundary cases, how to measure quality. We collect 100–200 representative queries and label the ideal answer. Only then do we write the first baseline.
Role Model and Context
The system prompt is built using the scheme: "You are [role]. Task: [goal]. Context: [conditions, data]. Rules: [what is allowed, what is not]. Output format: [explicit format]." For RAG, we add a source pointer. Example:
SYSTEM_PROMPT = """You are {role}. Task: {task_description} Rules: {rules} Output format: {output_format}""" A/B Testing and Metrics
We test each prompt variant on an eval set using LLM-as-judge. Compare by accuracy, completeness, style. Example test class:
class PromptABTest: def __init__(self, variants: dict[str, str]): self.variants = variants self.results = {name: [] for name in variants} def run_test(self, test_inputs: list[str], judge_prompt: str) -> dict: for input_text in test_inputs: outputs = {} for name, prompt in self.variants.items(): output = query_llm(input_text, system=prompt) outputs[name] = output comparison = query_llm( f"""Compare two responses to the query: \"{input_text}\" Variant A: {outputs[list(outputs.keys())[0]]} Variant B: {outputs[list(outputs.keys())[1]]} {judge_prompt} Return JSON: {{\"winner\": \"A\"|\"B\"|\"tie\", \"reason\": \"...\"}}""", temperature=0, ) result = json.loads(comparison) winner = result["winner"] if winner != "tie": winning_name = list(self.variants.keys())[0 if winner == "A" else 1] self.results[winning_name].append(1) return {name: sum(wins) / len(test_inputs) for name, wins in self.results.items()} Step-by-Step Prompt Calibration Guide
- Define the goal and metrics: accuracy, completeness, latency, hallucination rate.
- Collect an eval set: 100–200 real queries with reference answers.
- Create a baseline: a simple prompt with role model and basic rules.
- Iteratively improve: for each problem, add instructions, few-shot examples, checks.
- A/B testing: compare variant vs control on shadow traffic, pick the best.
Comparison of Prompt Engineering Techniques
| Technique | Application | Effect |
|---|---|---|
| Few-Shot | Add 3–5 examples to the prompt | Improves accuracy by 15–25% |
| Chain-of-Thought | Step-by-step reasoning | Boosts complex task quality by 30% |
| Self-Consistency | Multiple generations + voting | Reduces variance, increases reliability |
| Chain-of-Verification | Fact-checking after the answer | Cuts hallucination rate by 5–10 times |
What's Included in the Work
After calibration, you receive:
- Prompt templates in JSON/YAML with comments.
- An eval set of 200+ examples with reference answers.
- A metrics report: accuracy, recall, hallucination rate, p99 latency.
- A/B testing on shadow traffic (optional).
- Team training: a 2–4 hour workshop on maintaining and refining prompts.
- Stability guarantee: if quality drops after deployment, we adjust the prompt free of charge within a month.
Approach Comparison: One-shot vs Iterative
| Characteristic | One-shot (manual) | Iterative (with eval) | With A/B testing |
|---|---|---|---|
| Time to production | 1 day | 3–5 days | 5–10 days |
| Accuracy on test set | 30–50% | 60–80% | 90–95% |
| Latency stability | Low (p99 grows) | Medium | High (fixed p99) |
| Hallucination risk | High (>15%) | Medium (5–10%) | Low (<3%) |
An iterative approach with metrics is 2–3 times more effective than one-off writing—confirmed on 50+ projects.
Timeline and Results
- Basic prompt for a specific use case: 1–3 days.
- A/B testing with an eval set: 3–5 days.
- Production prompt with verification and documentation: 7–10 business days.
Cost is calculated individually, depending on the task complexity and number of prompts. A typical project ranges from $1500 to $4000. Savings from fixing hallucinations can reach $10,000 per month. Estimation takes 1 day: you describe the use case, we analyze and propose a plan.
Want stable LLM operation? Contact us to analyze your current prompt and suggest improvements within 1 day. Order a prompt audit and get a detailed report with metrics and recommendations. Get a consultation on your prompt today.