AI evaluation tells you whether a system works on the questions nobody rehearsed. On our projects it is the first deliverable rather than the last.
Running it first turns every later decision into a measurement instead of an argument.
What AI Evaluation Actually Produces
A test set made of your cases. Fifty to three hundred real examples from your own material, each with an agreed correct answer, written down by somebody who knows the domain. Assembling this is most of the work and it is the part that cannot be bought.
A score that names the decision. Accurate at what, against which cases, and what happens on the rest. Where the system routes a claim, the number that matters is how often it routes wrongly and what that costs.
A test suite you can run. A script in your repository that runs the whole set and prints the result, so anyone on your team can check after a prompt change or a model upgrade.
A baseline to beat. What the keyword search already manages, and what the current manual process gets wrong. Where the existing approach is close enough, that finding arrives in week one at our cost rather than month five at yours.
Why This Comes First
Measuring before building means the number is information rather than bad news about something that already exists.
It tells you whether retrieval or the prompt is at fault, whether a smaller and cheaper model would do, and where the remaining effort should go. Those three answers routinely save more than the evaluation costs.
It also settles the running-cost question honestly, because a model two points better at eleven times the price per call is a trade you can only see once both are measured.
What a Test Suite Catches
Silent regression. A prompt is edited to fix one complaint and three other behaviours change with it.
Provider updates. Models change underneath a stable version string, so a system that was fine on Monday answers differently on Thursday.
Input drift. The documents change, the vocabulary moves, and retrieval quietly starts missing.
All three surface the same way once the suite exists: the number moved, on a date you can point at.
What It Costs
A first AI evaluation on a system that already exists is usually one to two weeks, most of it spent building the test set with your people rather than writing code.
Run as the opening phase of a new project it is shorter, because the cases are gathered anyway while requirements are written. The suite is yours, in your repository, and it runs without us.
Tell us what the system is supposed to get right and we will tell you how to know whether it does.
Related Services
Retrieval systems are measured on whether the cited passage actually supports the answer, which is RAG development.
Agents are measured on whether the tool call was the right one, which is AI agents. Where the model is a classical one scored against a holdout, that is machine learning.