The pilot worked. The artificial intelligence model answers staff questions, shortens ticket handling or does a first assessment of applications, and the team has shown the board a result that justifies the spend. Then internal audit, the legal department or a client arrives and asks: where does this answer come from, where is the data, who is responsible for it. The pilot has no answer, so the model never reaches production, the day-to-day running of the process. The money for building it has been spent, and people are still handling the process.
Why a pilot is easy and entering the process is hard
A pilot answers one question: can the model do it? Production answers another: who bears the consequences when the model gets it wrong? In a pilot the model works on selected data, alongside the process, and its answers are seen by a handful of people who know it is a trial. In production the answer goes to a client, into the accounts or into a decision on an application, and the company and its board answer for the consequences.
The cost of that difference can be named. An answer without a source has to be checked by hand, and then the company pays twice: for the model and for the checking. Data that has left the company comes back as a lawyer's question: what left, to whom and when. A decision that cannot be reconstructed becomes, in a dispute with a client, the company's word against the client's. Auditors and lawyers ask about things a pilot never needed, and this is where the project stalls.
Three conditions before the model enters the process
All three have to be settled before the build. In a finished system each of them is a separate project, with its own cost and deadline.
Every answer has a source
Each answer from the model shows where it comes from: a contract number, a version of the terms and conditions, an entry in the system. The employee who passes the answer on sees the source and can check whether the model read it correctly. A model that answers from general knowledge, with no source, is fit for gathering ideas. A process in which the answer has consequences for a client takes a model that answers solely from the company's document base. That base holds what the responsible business department has approved. When the terms and conditions change, it is the document base that needs updating. The model stays the same.
It is known where the data is and who sees it
Three questions to the model provider, the company the model comes from, settle this condition. Where does the model run: in a cloud in the European Union or on the company's own servers? Who has access to the queries and answers: the company's staff, the model provider, its subcontractors? What does the provider record: are queries stored, for how long, and are they used to train its models? The answers belong in the contract with the provider, in writing. Some data, such as clients' personal data or data covered by confidentiality, never reaches the model at all. The system removes or replaces it before the query is sent. That decision comes before the choice of provider, because it determines who is even in the running.
A decision can be reversed and evidenced
A model in a process makes or suggests decisions that a client, an auditor or a regulator will ask about months later. Each needs a record: what data the model received, what it answered, who approved it and when. Before launch comes a risk assessment: what happens when the model answers wrongly, whom it affects and what it will cost. Where a decision has consequences for a client (a refusal, a fee, a change of terms), the process keeps a human in the loop, an employee who approves the model's answer before it reaches the client. Add to that an off switch: an agreed way of returning the process to running without the model, and a named person entitled to use it. The EU regulation on artificial intelligence imposes obligations that depend on a system's risk, and the decision record, the risk assessment and the human in the loop are what an auditor will check first.
What this means for the contract and the budget
The three conditions go into the contract with the system's contractor as acceptance criteria, with a description of how each will be checked. The evidence: a sample of answers with their sources shown, the model provider's written confirmation of where it runs and what it records, a sample of recorded decisions, and the report of a shutdown test. A contractor who will not write these criteria into the contract is telling you that it will deliver a pilot.
The budget has two parts, and a pilot shows only the first. The second is maintenance after launch. Someone assesses the model's answers on a running sample. The document base needs updating when terms or regulations change. The provider releases new versions of the model, which have to be checked before switching over. The decision record grows, and someone reviews it. The annual price of maintenance should sit in the offer next to the price of the build. Without that line, after the first change in regulations the model answers according to the old version and nobody knows.
Members of the team have led AI implementations in regulated organisations. The list below gathers questions the board can ask before acceptance without reading the documentation.
Checklist: eight questions before acceptance
- Does every answer from the model point to the document or system entry it comes from, and can an employee check it?
- Where does the model run and where does the data sit: in a cloud in the European Union or on our own servers?
- What does the model provider record from our queries, how long does it keep them and does it train its models on them?
- Which data never reaches the model at all, and who has checked that?
- Does every decision have a record: what data the model received, what it answered, who approved it and when?
- Was a risk assessment made before launch, and who signed it?
- Who is entitled to switch the model off, and how long does it take for the process to return to manual operation?
- What does a year of maintaining the model after launch cost, and who carries it out?
The answers depend on the process the model is to enter: what decisions are made in it, on what data and with what consequences for the client. That is why the first step is an analysis of that process, before anyone chooses a model.