LLM Prompt Version Control and A/B Testing

Prompts Are Code

Many teams hardcode their prompts. There is no way to track who changed what, when, or why—and when performance drops, it is hard to find the cause.

Git-Based Prompt Registry

CODE

prompts/
├── customer_service/
│   ├── refund_handler.v1.yaml
│ ├── refund_handler.v2.yaml ←
│ └── refund_handler.v3.yaml ←
└── content_generation/
    └── blog_writer.v4.yaml

YAML

# refund_handler.v2.yaml
version: "2.0.1"
model: "claude-sonnet-4-6"
temperature: 0.1
system: |
 {{company_name}} CS .
 : {{refund_policy}}
metadata:
  author: "team-cx"
  eval_score: 0.87
 changelog: " , "

Python

class PromptRegistry:
    def __init__(self, dir: str = "prompts"):
        self.dir = Path(dir)

    def get(self, name: str, version: str = "latest") -> dict:
        if version == "latest":
            files = sorted(self.dir.glob(f"**/{name}.v*.yaml"))
            return yaml.safe_load(files[-1].read_text())
        return yaml.safe_load((self.dir / f"{name}.{version}.yaml").read_text())

A/B Testing: Traffic Splitting

Python

import hashlib

class PromptABTest:
    def __init__(self, control: str, experiment: str, split: float = 0.1):
        self.control = control
        self.experiment = experiment
        self.split = split

    def get_version(self, user_id: str) -> str:
 # user_id ( UX)
        h = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
        return self.experiment if (h % 100) < (self.split * 100) else self.control

LLM-as-Judge Automated Evaluation

Python

JUDGE = """
 : {msg}
 reply: {resp}

1~5 after JSON :
- empathy:
- accuracy:
- resolution:
- overall:

{"empathy": N, "accuracy": N, "resolution": N, "overall": N}
"""

async def llm_judge(msg: str, resp: str) -> dict:
    result = await llm.generate(JUDGE.format(msg=msg, resp=resp), temperature=0)
    return json.loads(result)

Statistical Significance Testing

Python

from scipy import stats

def analyze_ab(ctrl_scores: list, exp_scores: list) -> dict:
    _, p = stats.ttest_ind(ctrl_scores, exp_scores)
    ctrl_m = sum(ctrl_scores) / len(ctrl_scores)
    exp_m = sum(exp_scores) / len(exp_scores)
    return {
        "control_mean": ctrl_m,
        "experiment_mean": exp_m,
        "improvement": f"{(exp_m - ctrl_m) / ctrl_m * 100:+.1f}%",
        "p_value": p,
        "significant": p < 0.05,
        "winner": "experiment" if exp_m > ctrl_m and p < 0.05 else "control"
    }

CI/CD Integration

YAML

# .github/workflows/prompt-eval.yml
on:
  push:
    paths: ["prompts/**/*.yaml"]
jobs:
  eval:
    steps:
      - run: python scripts/validate_prompts.py
      - run: python scripts/run_evals.py --dataset golden --threshold 0.82
      - run: python scripts/register_ab_test.py --version ${{ github.sha }}

With this three-part series, the full prompt engineering lifecycle is complete (learning techniques → advanced patterns → systematic operations).

Common Ways to Misread A/B Results, and How to Check

Prompt A/B tests go wrong more often in interpretation than in design.

Causes

Randomizing per request instead of per user, so one user sees both versions
Checking daily and stopping once it looks significant ("peeking"), which inflates false positives
LLM-as-Judge bias toward longer answers or a particular position (the first one shown)

How to check

Confirm in the assignment log that each user saw only one version.

SQL

SELECT user_id, count(DISTINCT prompt_version) AS v
FROM llm_requests WHERE experiment = 'summary-v3'
GROUP BY user_id HAVING count(DISTINCT prompt_version) > 1;

Fix sample size and stop time before starting, and don't use results for decisions before then.
Judge bias check: evaluate the same pair twice with the order swapped and measure how often the verdict flips. If it's high, average both orders or calibrate with a human-rated sample.

Fix

Assign versions deterministically by hashed user ID (e.g., int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100 < 50). Don't use Python's built-in hash(): string hashes differ per process (PYTHONHASHSEED), so assignments would change across workers and restarts.
Compare cost (tokens) and latency alongside quality; if the quality difference is small, pick the cheaper one.

Prevention

Store the experiment definition (hypothesis, metrics, sample size, stop condition) in the same repo as the prompts.

#Prompt engineering #LLM

✦ ✦ ✦

Sources and verification

Sources and checks vary by article. Only articles with a recorded check date, scope and sources below were checked, and only within that scope. For other articles, verify against the current docs for your version before applying anything, and email us about errors. Editorial policy · email us

This article has no recorded human check (date and scope) yet.

Published · May 21, 2026

Topic-related official docs (auto-linked · not per-claim sources)Git documentation ↗

How we write and review →Editorial policy →

Was this helpful?

Comments

Be the first to comment.