The score went down when we expected it to go up.
At first, we had an easy explanation: AI output varies, the user inputs were different, and perhaps this run was simply unlucky. That explanation felt reasonable—until we ran the evaluation again.
Most of the quality criteria were scoring eight or nine. One was not. Human voice returned 5, 5, 5 and 6 across four runs. The repetition changed the conversation. What looked like random variation had started to look like a product signal.
That was the moment the eval became useful. It did not tell us exactly what to fix, but it told us where to look.
Could the AI run the investigation too?
We were already building inside an AI coding harness—Codex, Claude Code or a similar agent environment. So we tried moving the evaluation work into that same harness.
The agent created a varied set of realistic samples, ran each one through the actual product, judged the outputs against an agreed rubric and kept the scores with the reasons behind them. It could then inspect the prompt, shared templates, examples, application code and output processing before running the tests again.
This sounds wonderfully automatic. It can also fail.
An AI judge may favour its own style. Generated test cases may be too easy. An agent may improve the score without improving the experience—especially if it sees every test in advance. If we simply tell AI to test AI and trust the number, we have built a confidence machine rather than a quality system.
So we gave the automation boundaries. The rubric came from people. Some test cases were held back. The judge had to explain its score using evidence from the output. We reviewed samples ourselves, particularly the failures and the suspiciously perfect results.
Within those boundaries, the approach became surprisingly effective.
What the repeated runs revealed
Instead of broadly asking the model to “sound more human”, the harness could run controlled comparisons. Keep the code unchanged and vary the prompt. Keep the prompt unchanged and vary the input. Remove a shared template, then restore it. Compare the scores and the actual words side by side.
Sometimes the prompt will be responsible. Sometimes the fault will sit in code, missing context, test data or a formatting rule applied after generation. The first idea may not work, and the second may make another criterion worse. But each run leaves evidence, making the next change less speculative.
The goal is not one impressive score. It is a dependable pattern across enough representative cases. For one product, that might mean at least 90 successful runs out of 100, with a higher bar for safety, security or compliance. The number is not universal; the evidence and the risk matter more.
What began as a disappointing score became a better way of working. The AI harness handled the repetition: generate, run, judge, compare and investigate. We remained responsible for the intent, the quality bar and the decision to release.
AI did not remove judgement from the process. It gave us enough evidence to use our judgement well.