Looking after AI after launch

Support begins on the day the agent or model enters daily work; building and launching it is a separate scope. We take over a system that is already running, built by us or by another contractor, and make sure it answers as well as it did on the day it was accepted.

We check answer quality on samples from current work: at an agreed interval we take a portion of real queries, and your subject-matter department rates the answers on a simple scale. A drop below the level of the acceptance day is a signal that something has changed: the documents, the price list, a regulation or the model itself. This drifting of answers away from reality is called model drift; the sample lets you spot it before a customer does. The provider you buy model access from releases new versions and retires old ones; we test a new version on the same sample before switching to it.

The document base, the instructions for the model (the prompt, the text that tells the model how to answer) and the model version each carry a version number and an effective date. Every change is approved by a designated person on your side, usually the process owner, after a check on the sample. If your system falls under the EU regulation on artificial intelligence (AI Act), we write its requirements into the acceptance criteria for changes.

The cost of using the model depends on the amount of text processed, counted in tokens: fragments of words that the provider bills for. We set limits, split the cost by department and type of query, and report it every month. When it grows faster than the number of cases, we look for the cause: overly long queries, repetition or use outside the intended purpose.

The AI decision log has to answer a customer complaint or an auditor's question without a programmer. For every answer we record the date, the input data, the source documents, the model and instruction versions, and the person who approved it. Employee reports go to one place, each gets a reply and a status, and recurring ones are turned into a change for approval.

1. We take over the system. We check which documents the agent uses, which model version is running, what is recorded and who answers reports. Result: a description of the starting state with a list of gaps.
2. We set the quality measure. With your subject-matter department we choose the first sample and the rating scale that will tell us the model is getting things wrong. Result: a baseline for every later review.
3. We run reviews and changes. At the agreed interval we rate the sample, check the cost, answer reports and prepare changes for your approval. Result: a monthly report and the change register.
4. We hand over or stay on. We teach your team to run the reviews on its own, or we continue the support under the contract. Result: support in the hands of people who know its history.

1. Who rates the sample of answers, how often, and how will we learn of a drop below the acceptance-day level?
2. How will the contractor learn that our terms and conditions or price list have changed, and who enters the change into the document base?
3. What happens when the provider retires the model version we run on, and who tests the new one before the switch?
4. Who on our side approves changes to the model instructions and to the document base?
5. In what form will we see the model usage cost each month, and what happens when the limit is exceeded?
6. How do we find an answer from several months ago in the decision log, and how long is the log kept?
7. Where does an employee report a wrong answer, who replies, and how do we know how many reports are waiting?

The scope of support depends on what the agent does and what one of its mistakes leads to, so the first step is an analysis of the process and of what has been recorded since launch.