AI adoption ยท Improvement
Keep AI useful as your business changes
Something that worked in month two stops working in month ten, usually without anybody noticing. Sources get reorganised, the people who were trained move on, the metered cost grows, and models change underneath the workflow. Ongoing work means one named owner, a monthly page with numbers on it, and a decision each month to expand, adjust or stop.
This page is as useful if you bought AI somewhere else. The measurement and the ownership question do not depend on who configured it.
Ownership and maintenance
What does keeping it working involve?
Six things. None of them is dramatic, which is exactly why they get dropped.
- Named ownership
- One person inside your company answers for the result. Not the IT lead, unless the constraint is IT's. Shared ownership is the same as none, and it shows up three months later as nobody having looked.
- Quality evaluation
- A sample checked on a fixed rubric, at a fixed interval, against the same measure as the baseline. Output that nobody checks drifts quietly, and the first evidence of drift is usually a client noticing.
- Change control
- Models, connectors, permissions and the instructions people rely on all change underneath you. Knowing what changed and when is how you explain a result that moved without anyone touching the workflow.
- Maintenance
- Connections break, sources get reorganised, and a document store gets restructured by somebody who did not know an assistant was reading it. This is the unglamorous line that keeps the rest working.
- Training refresh
- New joiners, and the people who learned it in month one and never revisited it. Products change enough in a year that last year's training quietly stops being true.
- Running costs
- Licences, the metered consumption line underneath them, and the reviewer time. Tracked monthly, because the metered line is the one that moves without anyone deciding.
The monthly page
What does the monthly page look like?
Thirteen fields, reviewed by the named owner. This is the artefact that turns a business case into evidence, and it is the fifth test of an AI-native company.
| Field | What goes in it |
|---|---|
| Objective | The constraint this was bought to remove, in one sentence. |
| Route | Which of the four routes the benefit travels, and who committed to it. |
| Baseline | The measured before state, with its date and sample size. |
| Target | The number that would make this worth continuing. |
| Observed | This month's result, measured the same way as the baseline. |
| Adoption | The share of the intended population active this month. |
| Quality | The guardrail measure, on the same sample. Never a speed figure on its own. |
| Exceptions | How often a human had to intervene, and what for. |
| Cost to date | Cumulative, with cash and internal time shown separately. |
| Realised benefit | What has actually landed, against the route named above. |
| Evidence source | Where the numbers came from, so somebody else can check them. |
| Owner | The named person who answers for it. |
| Decision | Expand, adjust or stop, with the date. |
The fields are what matter; the figures are yours. The method behind them is set out in full on the business case page.
Three separate claims
Why keep use, capacity and value apart?
Because they are three different claims and only the third one reaches the accounts. A monthly report that runs them together will always look better than the business is doing.
Use
People opened it and did something with it. Necessary, and on its own it proves nothing about the result.
Capacity
Hours came back to a team. Real, and still not money. It stays capacity until a route is named.
Realised value
Cash moved, by one of four routes: serve more, do not hire, stop buying, avoid a loss. There is no fifth.
Start with the monthly scorecard
What should we improve next?
The scorecard points at one of three, and stopping is as real an answer as the other two.
Improve the existing setup
Most months this is the answer. Look at the instructions people rely on, whether the sources are current and correctly permissioned, who has been trained since the last review, and whether the named owner is still the right person.
Investigate a recurring workflow
When the same workflow keeps appearing in the exception column, and it runs at volume, on a schedule, or has to write back into a core system, it has outgrown an assistant. That makes it worth examining as a scoped automation, not proof that a build is necessary or that it would pay.
Explore process discoveryReassess or stop
If the observed result has not cleared the target and there is no specific change left to make, go back to the objective and the business case. A review that has never produced this outcome is not a review.
Re-run the business case
Whichever it is, the advantage at this point is a measured baseline rather than an estimate, which makes the second business case considerably easier than the first.
Discuss your next step
Common questions about keeping AI working
How do we know whether AI is still working six months in?
By measuring the same thing you measured at the start, on the same kind of sample, and putting it in front of the person who owns the objective. The question is hard to answer when no before state was recorded and nobody owns the after. Keep it to one page so it gets read in a management meeting rather than filed.
What goes wrong between month three and month twelve?
Four things, and they are quiet. The people who were trained move on and nobody trains their replacements. A source gets reorganised and the assistant starts answering from something out of date. The metered consumption line grows because a new workflow rides on the same platform. And the person who championed it gets promoted into something else. None of these announces itself.
Why keep usage, capacity and value in separate columns?
Because collapsing them is how a report ends up claiming a saving that never reaches the accounts. Usage says people opened it. Capacity says hours came back to a team. Value says cash moved, and it only moves by one of four routes: the same people serve more work that exists to be served, a planned hire is cancelled in writing, a spend line is switched off, or a costly event becomes less likely. Reporting capacity as though it were cash is the error the whole business case method exists to prevent.
What does an expand, adjust or stop decision actually look like?
It is taken against thresholds agreed before the pilot rather than after the results arrived. Expand means the numbers cleared the target and the next group or the next workflow is named. Adjust means something specific was wrong, usually the sources, the instructions or who it was given to, and there is one change to make before the next review. Stop means it did not clear and will not, and the licences come off the bill. All three are real outcomes, and a review that has never produced the third is not a review.
How do you choose what to automate next?
From the exception count. When the same workflow keeps turning up as the thing a person had to intervene on, at volume, on a schedule, or writing into a core system, it has outgrown an assistant and become a build. That is the point to look at process discovery under AI Automation, with the advantage that you now have a measured baseline instead of an estimate.
We bought AI somewhere else. Can you support it?
Usually. The measurement, the ownership question and the monthly decision are the same whoever configured it, and the first month is normally spent finding out what it can currently reach and who is actually using it. Tell us what was bought, for whom, and what it was meant to improve. Scope and cadence are agreed against what we find rather than against a standard package.
Cannot say whether your AI is working?
Tell us what was bought, who it was for and what it was meant to improve. The first thing we do is find out what it can currently reach and who is actually using it.
The first call covers what you want to improve, the tools you already run, and whether a scoped paid discovery is the right next step.