Testing an AI analyst on a real incident

TimberWorks runs on its own now. A PrestaShop storefront takes orders around the clock, Matomo watches the customers, Loki and Prometheus collect the logs and metrics, an Apache Camel layer carries master data, and an ERP sits behind all of it, serving CSV files over SFTP. These are the ordinary components of a small company's information system, and they are real products rather than mock-ups. The lab has stopped being a construction site, which means we can finally run experiments on it instead of just building it.

Meanwhile the whole industry is pointed at AI agents and what they can pull off. We wanted to know, half out of curiosity and half out of self-interest, whether one of them could find its way around TimberWorks.

There is a great deal of noise about agents resolving incidents, and it seemed like a good moment to sit down and ask the questions that interest us:

• Can an AI really find the root cause of a problem in a complex architecture?
• Is the information system itself suited to an agent?
• Is the famous harness hard to build, and does it genuinely change performance?
• What actually raises the odds of reaching the root cause?

And plenty of others we will get to another day.

So we designed an incident that is not obvious. It is more of a business failure than a technical one, it costs the company real revenue, and it has a single defined cause.

The incident

TimberWorks ships to two countries, the United States and Canada. Its ERP holds a short list of who carries what and where: 3 lines, one of them Canada. We deleted that line.

Seconds later the integration did what it exists to do. It read the list, found 2 services where there had been 3, and made the shop agree.

A customer in Ontario now reaches the delivery step and finds it empty. Nothing to choose from, on a page that cannot be completed without choosing. They cannot buy, and they cannot be told why, because the shop was asked which carriers serve Canada and answered, correctly, none.

Nothing errored. No service went down, no exception was raised, no error line was written, and all 5 operational dashboards stayed green. American checkout was untouched, so orders kept arriving and revenue kept rising. The only record of the change is a single INFO line written by the integration, two systems upstream of the symptom, and it drops out of any ordinary log window within hours.

The evidence is not hidden. It is scattered, partly perishable, and nothing in the company raises its hand. In the 6 minutes before we started watching, 16 Canadian visitors reached the storefront and none of them bought anything.

Method

We ran three analysts against the fault, and each of them had the same read access to the company's APIs and nothing more.

• angel is the baseline: a fixed set of tools for reaching the API data.
• philip gets no tools at all, only written documentation of the systems and a shell, and has to build its own way in.
• eve has philip's desk and differs only in the harness driving it.

philip against eve isolates the harness. angel differs from both in tooling and harness at once, so a gap there cannot be pinned on either. Going in, we assumed this was the axis that would decide things: better tools, better answers.

Why not simply hand them the tools? The moment a tool helps too much you are measuring the tool, and the point is to test the information system rather than the agent.

Beyond a briefing that names the company and the four systems an analyst can reach, the shop, the analytics, the logs and the ERP feed, the prompt is the whole of what it is told:

"We've had complaints from customers in the last 5 minutes. It seems we have a problem — please investigate."

It names no system, no market and no hypothesis.

We crossed three factors against each other:

• the analyst, angel, philip or eve
• the reasoning effort it is allowed, low, medium or high
• the model behind it, either gpt-5.6-luna or gpt-5.6-terra

That gives 18 combinations, and we ran each of them 9 times, because a single run tells you almost nothing about an agent while 9 tell you whether a difference is real. The campaign came to 162 investigations.

We graded on three things: that the problem is specific to Canada, that Canada has no shipping coverage, and that it came from the ERP feed. An answer counts when it states the first two. The third is recorded but not required, because that evidence expires. Where the bar sits turns out not to matter much: demand one of the three or demand all three and the result moves by 2.5 points. A model graded every answer as it arrived, then we read all of them by hand, blind to its grades. The two agree on 97.5% of runs.

Then we put the carrier back, rebuilt the shop, let it settle, and sent the same ticket 54 more times with nothing to find. An analyst that reports a plausible fault whenever asked scores well on an incident and is worthless in production.

Results

96 of the 162 investigations found it.

We were wrong about which axis would matter. Swapping the model moves nothing we can demonstrate. What moves almost everything is the reasoning effort, and that happens to be the only thing in the whole experiment we assigned. Every other difference between runs is a choice a run made, or something the clock did to it.

There is a floor under the flat model axis, and this campaign cannot say where. Both of our models sit well above the point where an investigation stops being possible at all. Put the same ticket to a much smaller one and the work does not degrade gradually, it stops happening. The axis looks flat because we only sampled the part of it that works.

The gap between the analysts is the result we thought we were buying, and it is the one that does not survive. philip ran later, when the campaign was already doing worse, and the same effort label buys it up to 2.8 times as much reasoning as angel. Correct for both and what is left is too small to call real. We cannot separate what an analyst was handed from how much it was allowed to think. So one of the questions we opened with does get an answer, just not a usable one.

The failures were not what we pictured either. Almost every wrong answer that named a cause named something real: a settlement worker that really does hold bank-wire orders for five minutes before marking them paid, API errors that genuinely were logged, a few of them ours. A weak investigation does not reach for fiction. It reaches for a true fact that explains nothing, which is much harder to catch, because everything in the report checks out.

What separated the runs

If the analyst did not decide the outcome, something else had to.

Four things clear the noise, and only the first was ours to assign. The other three are all about where an analyst went, and when. Reading the system of record is worth nearly as much as the reasoning setting itself, and among the runs that got there at all, the ones that went early did substantially better.

The figure cannot take it further than that. Nobody told those runs where to look, so an analyst may well go to the ERP feed because it is already on the right track, not the other way round. Order and outcome are tangled together here, and only an experiment that fixes the opening move will pull them apart.

What effort buys, and what it costs

That word, effort, has been carrying a lot of weight so far. Here is what it actually bought:

The last column is what one answer cost: everything all 54 runs spent, the ones that found nothing included, divided by the number that found the fault.

The mechanism is not subtle. The setting buys tool calls, and calls are what the bill is made of.

Fit a linear model to that cloud and the slope is the number to take away: about 20,000 billed tokens per tool call. None of that is the cost of thinking, which is under half a percent of the total. It is the conversation being re-sent every time the analyst reaches into the company. The bill measures how often it reached, not how hard it thought, and the three settings sit at successive points along one slope rather than on slopes of their own.

Because it is the same context arriving again, 91% to 94% of it is served from cache, and that share rises with the setting. Whatever a token costs you, the re-sent part is the cheap part.

Now put the outcome back onto the same space.

The answers are up and to the right: a median of 0.96 million tokens against 0.42 million for the runs that found nothing. But the two groups overlap through the whole middle of the range, and the biggest spenders are not the most successful. Inside the high setting, runs above the median call count score 85% and those below score 93%. A run that is getting somewhere keeps pulling, and a run that has given up stops. Spending records whether the trail was still warm as much as it explains why.

Which makes the trade different from how it looks per run. Run for run, high costs nearly three times the tokens of low. Per diagnosis it costs the same: about 1.6 million tokens buys an answer at either end of the scale. What the setting changes is not the price of an answer but how many of your runs produce one, 17 of 54 against 48 of 54. The low setting only looks cheaper: it buys the same rate at a worse ratio, and everything spent on its 37 empty runs bought nothing.

The control

This was meant to be the boring half, and it was the most surprising thing the campaign produced.

8 runs found real defects nobody had planted and we did not know were there: a delivery promise of 8 business days where the ERP says 5, because 2 feed rows share one carrier identifier; 2 products reporting 0 sellable against 183 and 256 units in stock; and every one of 198 orders taking the Cross-Border carrier while Ground, the domestic one, was never selected.

That last one we now think is ours rather than the company's. Nothing in the shop distinguishes a customer preferring a carrier from our simulator picking one, and it is almost certainly the simulator. The analyst was right about the data and wrong about the business, which is the same failure mode as the settlement worker, arriving from the opposite direction.

Effort turns up here again, and what it buys is real findings rather than more guesses. Inventions fall the same way, though at 18 runs a side that direction is worth replicating rather than believing. Either way, more thinking did not make the analyst wilder. It made it turn up things about a company we thought we knew.

What the company would have to change

Both campaigns kept pointing back at the same handful of things, and none of them were about the analysts.

Every change to the business should leave a record, and this one barely did. The integration wrote a line at INFO level, indistinguishable from a thousand others and gone from any ordinary window within hours. What was needed was an account of what changed, when, and from what to what. The company altered its own shipping offer and kept no ledger of having done it.

None of that is an AI problem. It is ordinary practice that plenty of information systems, this one included, do not follow.

Then there are the logs themselves, and how much of them is signal. Ours are quiet enough that a determined analyst can read them end to end, and even so the settlement worker collected most of the wrong answers and 12% of the failures went chasing errors the experiment had itself generated. Now put the same investigation in front of a company whose services crash regularly, emit continuously, or take orders at a hundred times this volume. The evidence does not get harder to reach. It gets harder to find among everything else that is also true.

The ERP is the counter-example, and it is why reading it predicted the outcome better than anything else a run chose to do. Master data lives in files, edited through a pull request, so every change carries an author, a timestamp and a diff alongside the state it produced.

Against all that, the surprise is how little the company had to be adapted. No instrument was built for this fault and nothing any analyst was handed pointed at shipping. They found their way in through the APIs the systems already expose. A fairly ordinary information system turned out to be more legible than we expected. Had we shipped a tool that checked shipping coverage by country, the runs would have passed and we would have learned only that we had guessed our own incident correctly.

Which leaves a different open question than the one we started with. Not whether an agent can work in a company like this, but how to make it reliable and cheap. Detection has to get steadier, and it has to improve on two axes that pull apart: accuracy, and tokens burned. This campaign bought its accuracy at 20,000 tokens a call and a great many calls. An MCP layer is the obvious next thing to try, though standardizing access is not the same as needing it less often.

Limits and next steps

This is one fault, one topology, one prompt, one company, and the runs shared a live environment that they were also changing: the success rate fell from 68% in the first quarter of the campaign to 48% in the last while the planted fault never moved. These numbers describe this campaign rather than a rate.

Our own instruments needed watching too. We assumed a healthy company for the control and it turned out to hold three real defects, and the automatic grader that matched us on the incident reported the effort result backwards on the control. Everything above was read by hand.

The fault itself was also easier than we designed it to be. An answer had to name three things and runs stated all three or none, because one table carried the whole chain once an analyst thought to look at shipping by zone. The next one should be a fault whose evidence genuinely lives in different systems.

Two experiments follow. Reading the ERP feed early is worth 31 points among the runs with the least room to think and nothing among those with the most, which suggests order can substitute for budget; fixing the opening move by instruction would settle it, and it is cheap. And every run so far has held read access to everything, which is not an employee. It is a superuser with a job title. Whether least privilege makes an incident unsolvable is the question this lab was built to ask.

What Archipel Labs is for, the code, or get in touch.