An AI model evaluation answers two questions at once in Ireland: whether the system works, and whether you can show somebody that you checked.
Under the EU AI Act the second one has teeth, and both are served by the same piece of work.
What an AI Model Evaluation Has to Contain
A test set drawn from your own cases. Fifty to three hundred real examples with an agreed correct answer for each, written down by somebody who knows the domain.
A score expressed as the decision. Where the system declines an application, the number that matters is how often it declines wrongly, who that affects, and what the appeal path is.
A record of what was tested and when. Which model version, which prompt, which cases, and what the result was. This is the part a regulator asks for.
A test suite anyone can run. A script in your repository that reproduces the whole thing, because reproducibility is what separates a record from an anecdote. Public benchmarks measure how a model handles somebody else’s problem, which is why the cases here are all yours.
Why the EU Angle Changes the Order of Work
Under the AI Act, obligations attach to a system’s risk classification, and the higher tiers require documented testing, logging and human oversight.
Running the AI model evaluation first means the record accumulates as a by-product of the work, at very little extra cost. Building first means reconstructing decisions nobody wrote down.
We build the evidence and your advisers decide what it has to prove, which is the division of labour that keeps both sides useful.
Residency, Decided Before the Test Set
Where the evaluation set contains personal data, it stays inside the European Union. That usually means running the suite against a model deployed on infrastructure you control, which is on-premise AI.
Worth settling at the start, because it constrains which models can be compared, and the test set is the expensive part to rebuild.
What a Test Suite Catches
Silent regression. A prompt is changed to fix one complaint and three other behaviours move with it.
Provider updates. Models change underneath a stable version string.
Input drift. The documents change, the vocabulary moves, and retrieval starts missing.
All three surface the same way: the number moved, on a date you can point at.
What It Costs
One to two weeks for a first evaluation of an existing system, most of it spent assembling the test set with your people rather than writing code. Shorter when it opens a new project.
The suite is yours and it runs without us. Tell us what the system must get right and we will tell you how to prove that it does.
Related Services
Retrieval systems are scored on whether the cited passage genuinely supports the answer, which is RAG development.
Where the model is a classical one scored against a holdout, that is machine learning. Where nothing may leave your infrastructure, start at on-premise AI.