Post 2 of 8 in the series on rebuilding a bank’s software development factory for the agentic era.

The AI token bill : This is the post I would have wanted to read eighteen months ago, so I am putting it second rather than burying it in the middle of a design series.

Here is what happened at XYZ Financial Services, told as a sequence, because the sequence is the argument.

Fourteen months

It began with encouragement. The management board, reasonably and in good faith, told the IT organisation to use AI massively to raise productivity and get more into production. That message was repeated at a town hall, written into the annual objectives, and echoed by every layer beneath it. There was no friction in the way. Licences were procured centrally, access was granted broadly, and nobody wanted to be the person who slowed down the strategic priority.

Adoption did what the message asked of it. Seat activation went from a few hundred to nearly the whole engineering population inside two quarters.

Then usage changed shape, which is the part that mattered and the part nobody was watching. Engineers moved from inline completion, which is a cheap assistive interaction, to agent runs, which are long, iterative, and consume context by the hundred thousand tokens. Squads discovered they could point a fleet at a migration and walk away. Someone in the Italian site ran a repository-wide refactoring experiment over a weekend because it was Friday and it seemed worth trying.

Then the invoice arrived. XYZ’s AI spend went from a predictable per-seat licence line that finance barely noticed to a metered consumption line that grew by a factor I will not pretend to reconstruct precisely, and in the worst month the total cost of running IT was higher than it had been before any of this started.

The productivity gains were real. They were also invisible in the same ledger, because they showed up as engineering hours not spent and features arriving earlier, while the cost showed up as a run-rate increase in a cost centre with a name and an owner.

A crisis room was formed.

Why the curve broke

Five things happened at once, and the remedies are different for each, so it is worth separating them.

The pricing model changed shape underneath the budget. A per-seat licence is linear in headcount and forecastable a year out, which is how IT budgets get built. Agentic consumption is metered and effectively unbounded. It scales with ambition, with context size, with how many iterations a task takes, and with how many runs happen in parallel. XYZ moved from the first cost model to the second without anyone re-forecasting, because the transition happened inside a tool people were already licensed for.

There was no unit of account. Nobody could say what a change cost. Without that number there is no way to tell spend that bought something from spend that did not, so the only lever available when the invoice lands is a total cap, applied to a distribution nobody has measured.

The directive had an input metric and no output metric. “Use AI massively” is measured by usage, so usage is what people produced. An organisation asked to maximise an input will maximise the input, and the metrics that made the adoption look successful, seat activation and message volume and suggestions accepted, were exactly the ones that could not distinguish valuable work from expensive noise.

Failed runs cost the same as successful ones, and weak context makes runs fail. This is the observation I would put in front of a governance body first. An agent working against a repository with no context file, a stale specification, and no reachable test environment will read more, guess more, iterate more, and fail more often than an agent working against a curated corpus. Every one of those extra iterations is billed. Token spend is therefore a lagging indicator of context quality, and a large part of what looked like an AI cost problem at XYZ was documentation debt wearing a different invoice. Cutting the budget does not fix that.

And parallelism removed the natural brake. A human engineer who tries something expensive gets tired. A fleet does not. One badly configured migration lane can spend a quarter’s allocation in a weekend, and at XYZ something close to that happened, with nobody acting improperly at any point.

What the crisis room got right

Two things, and I want to be fair about this because crisis rooms have a poor reputation.

It did not revoke access. That would have produced the failure mode I have watched elsewhere: the sanctioned path becomes slow or gated, the unsanctioned path is a browser tab, and source code leaves the perimeter in a way nobody can measure. Shadow usage does not show up in the metrics that measure sanctioned usage, which is the entire problem with it.

And it treated the problem as one of governance and unit economics rather than of technology, which is the correct diagnosis.

The trap it avoided is the flat cap. An organisation-wide monthly ceiling, applied uniformly, enforced by cutting off access when it is reached. That converts a cost problem into a delivery problem in the last week of every month, it penalises the teams doing expensive valuable work equally with the teams running experiments, and it teaches everyone that the capability is unreliable. Which is the fastest way to lose the adoption you spent a year building.

The six streams

The answer was a governance body whose job is to rationalise the use of the AI budget, organised into six streams with owners. I will take each in turn, because a stream with an owner and no definition of done becomes a standing meeting.

FinOps and governance owns the money: the consumption model, the allocation to domains, chargeback or showback, forecasting, and arbitration of the AI budget. Finished looks like a cost per merged change, by risk tier and by domain, that the head of the factory can defend in a budget meeting, plus alerting on anomalous consumption within hours rather than at month end. The trap is that FinOps becomes an approval queue for individual runs, which reintroduces exactly the coordination cost the whole architecture exists to remove. Budget governance belongs at the level of allocations and guardrails, not transactions.

Platform SDLC and GitHub Copilot owns the primary harness and the delivery toolchain: enterprise configuration, organisation policy, content exclusion, identity, agent lane definitions, and integration with Azure DevOps. Finished looks like per-lane metering, per-team visibility, and a kill switch that has actually been tested. The trap is treating this as a licensing and rollout function rather than as a platform product whose main customer is an agent fleet.

Methodology, covering the business analysts, the factory, the business, and the project delivery office, owns how work is specified and how it flows. This is the stream with the largest effect on cost and the least obvious connection to it, for the reason above: a well-specified change costs a fraction of a badly specified one, because the agent stops guessing. The trap is that this stream gets staffed as a documentation exercise and produces templates nobody executes against.

AI platforms, meaning Prisme.ai and comparable enterprise platforms, owns the non-coding AI estate: business-facing assistants and workflow tools serving operations, client service, and the back office. Be careful here, because conflating this with the factory is a governance error I have watched cause real trouble. A coding harness serves engineers building software under change control. A business-facing platform serves operational processes, may touch client data directly, and raises its own AI Act classification questions that have nothing to do with the factory. Share the budget and the provider concentration analysis. Keep the control designs separate. Finished, for this stream, is an inventory of AI use cases outside the factory with an owner, a data classification, and a risk classification for each. That inventory is always longer than anyone expects, and a third of it was built by a business team without telling IT.

Training and change management owns capability and behaviour: the AI literacy curriculum required by Article 4 of the AI Act, the practical craft of specifying and reviewing, and the works council relationship. The trap is the e-learning module. A forty minute video with a quiz produces a compliance record and no capability, and the difference shows up in the token bill, because people who cannot specify burn more of it.

Technology watch owns the question of what changed this quarter: model releases, harness features, framework shifts, pricing changes. The trap is that it becomes a newsletter. A watch function that does not run evaluations is producing opinion, and opinion about model quality is worth very little when the question is whether a specific model version is good enough for a specific class of change on your own code.

The number that actually fixes this

Everything above resolves into one figure that XYZ did not have and now must: the fully loaded cost of a merged, reviewed change, segmented by risk tier and by domain.

It is computable if your delivery records are in order, because the flow record knows which agent runs belonged to which change and the metering knows what each run cost. It turns a budget conversation into an engineering conversation, because it can be compared against the thing it replaced, which is the fully loaded hourly cost of the engineers who used to do that work. And it decomposes usefully. A domain with a high cost per change has a weak context corpus, an unpaved path, or a review queue so slow that work gets re-run. Every one of those is a platform investment with a value attached.

The second number, and the one I would put next to it: the share of consumption spent on runs that produced nothing. Abandoned runs, guardrail blocks, runs that failed on missing context. I would expect that share to start somewhere north of a quarter and to fall as the context corpus fills in. It is waste in the precise sense that it bought nothing, and unlike total spend it is something the platform team can act on.

The third, for the budget holder: consumption against allocation per domain, with anomaly alerting set in hours. Not a cap that terminates access. A signal that reaches a named person while the weekend refactoring experiment is still running.

Two things I would refuse to report

Seat activation and usage volume. These were the metrics that made the original adoption look like a success while the cost curve broke underneath it, and they will be no more informative on the way down than they were on the way up.

Estimated hours saved. Every vendor business case is built on this number and it is unfalsifiable by construction, because it compares what happened to a counterfactual nobody observed. Worse, in a bank it will be computed by the people whose budget depends on it. If you want to know whether the programme delivered value, the honest measures are throughput at constant quality, escaped defect rate, and cost per change. All observable, none requiring anyone to imagine an alternative history.

I would extend the same refusal to percentage of code written by AI. It is unmeasurable in any principled way once a human has edited a suggestion, and it measures an input rather than an outcome even if you could measure it. Someone will request it anyway, probably for a slide about digital transformation, and the right answer is a short written explanation of why the number does not exist.

The part the governance body will like least

Charter it with an end date.

Eighteen to twenty-four months, stated at formation, at which point the streams fold into their permanent homes. FinOps into the IT financial function. The harness into the platform group. Methodology into the factory. Training into HR and IT governance. Technology watch into the platform group’s evaluation capability.

A body created in a crisis that is still meeting three years later has become a coordination layer, and the whole argument of this series is that coordination layers which exist to move information between people are the thing an agentic factory can finally replace. It would be a poor outcome if the programme that builds that capability left behind a permanent committee as its monument.

Next post takes Block’s organisational model apart element by element and sorts it against MaRisk, DORA and the AI Act. What transfers, what transfers only after surgery, and what does not survive the trip at all.