Automated Prompt Regression Testing with Promptfoo Integration

You updated the prompt to support a new response format, and a week later the model started hallucinating on simple queries — accuracy dropped by 30%. Each prompt change can silently break scenarios that worked for years. In one project, changing a single word in a prompt reduced accuracy by 30% and required a week of manual testing to catch the regression. We use Promptfoo to build automatic regression testing in a couple of days and forget about such surprises forever. Promptfoo lets you run testing 5x faster than manual and provides objective metrics like LLM-rubric, ROUGE-N, and contains-any instead of subjective evaluation. It is an open-source CLI application with dozens of assertion types that integrates into any CI/CD pipeline.

Our engineers have executed 40+ LLM integration projects, and in all cases Promptfoo reduced regression detection time from a week to a few minutes. We configure tests for your specific scenarios: check for required phrases, absence of forbidden words, compare model responses, and measure similarity to a reference via embeddings. As a result, you get a transparent process where every PR is checked automatically, and prompt quality is controlled by metrics.

What problems we solve

• Regressions after prompt changes: a new condition can break other cases. Promptfoo runs hundreds of tests in seconds.
• Model instability: the provider updated the model, and it started producing a different format. Tests catch this before deployment.
• Subjective evaluation: instead of "seems to answer well" — objective metrics (contains, LLM-rubric, ROUGE).
• Lack of CI/CD for prompts: development without quality control — like committing without unit tests.
• RAG scenarios: verifying correctness of retrieval and relevance of answers based on vector search.
• QA for AI: ensuring the model does not produce incorrect or unsafe responses in production.

Why prompt regression testing counts

LLMs are probabilistic systems. One word in a prompt can change the entire semantics of the response. Without automated tests, you release changes blindly. Promptfoo provides guarantee that old scenarios still work and new ones don't break existing logic. This is especially important in production systems with thousands of requests per day. According to the official documentation, Promptfoo is an open-source tool for evaluating and testing LLM prompts, used in projects handling up to 10,000 daily requests.

How we configure Promptfoo

We start with an audit of current prompts: collect all used templates, highlight key scenarios, and write test cases with assertions. A typical config:

providers:
  - openai:gpt-4o
  - openai:gpt-4o-mini
  - anthropic:claude-3-5-sonnet-20241022

prompts:
  - "Summarize the following text in 3 sentences: {{text}}"
  - "Create a concise 3-sentence summary of: {{text}}"

tests:
  - vars:
      text: "Long article about machine learning..."
    assert:
      - type: contains
        value: "machine learning"
      - type: llm-rubric
        value: "The summary is accurate and covers the main points"
      - type: javascript
        value: "output.split('.').length >= 3"

  - vars:
      text: "Another test document..."
    assert:
      - type: not-contains
        value: "I cannot"
      - type: rouge-n
        value: reference_summary
        threshold: 0.5

We don't use all assertions indiscriminately, only those actually needed. We add custom checks via JavaScript: for example, that the response contains exactly 3 points in markdown. This yields 100% confidence in output format.

Integrating Promptfoo into CI/CD

Add one command to your workflow:

promptfoo eval --ci

GitHub Actions example:

- name: Run prompt tests
  run: npx promptfoo eval --ci

If a test fails, CI exits with an error — PR not merged. We set up automatic report viewing via promptfoo view so the team can see diffs between prompt versions.

What's included in our service

• Audit of current prompts and identification of critical scenarios.
• Writing 10–50 test cases with assertions tailored to your metrics.
• Promptfoo configuration and provider setup.
• CI/CD integration (GitHub Actions, GitLab CI) with notifications.
• Complete documentation and team training on adding new tests and interpreting results.
• Access to our private dashboard with test history and pass rates.
• Support for 30 days after deployment, including prompt troubleshooting.

Process

1. Prompt analysis — collect current templates, identify critical cases (up to 2 hours).
2. Test design — write 10–50 test cases with assertions for your metrics.
3. Implementation — create Promptfoo configuration, set up providers and variables.
4. Testing — run eval, fix failing tests, clarify requirements.
5. Deploy to CI/CD — add step to pipeline, configure notifications.

Timeline and cost

Work takes from 3 to 7 days depending on the number of prompts and complexity of assertions. Basic setup starts at $500; complex projects may reach $3,000. Cost is calculated individually after the audit. Get a free consultation — we'll evaluate your project in one day. Typical savings: automating regression testing reduces manual QA costs by 80% and cuts time-to-deploy by 75%.

Checklist: what to check in prompt tests

Promptfoo vs manual testing

import promptfoo

results = promptfoo.evaluate({
    "prompts": ["Classify sentiment: {{text}}"],
    "providers": ["openai:gpt-4o-mini"],
    "tests": [
        {
            "vars": {"text": "Great product!"},
            "assert": [{"type": "contains", "value": "positive"}]
        }
    ]
})

print(f"Pass rate: {results.stats.successes}/{results.stats.total}")

Our engineers with 5+ years of experience in AI/ML have executed over 40 LLM integration projects. We guarantee test coverage will be maintained and expanded. Promptfoo is the most flexible tool for prompt regression testing. Request a consultation — we'll show how Promptfoo fits into your workflow and estimate scope in one day. Turnkey approach: from test cases to CI/CD, with documentation and training. Get a free prompt audit — contact us!