What it costs to believe the machine has it covered

In one long night, agents built six working parts of a system for us. Every one of them passed its own checks. Most of what was wrong with them passed too.

Our position: AI has become very good at making things. It has not become better at telling you whether the thing is any good. In a person those two abilities tend to arrive together. In a machine they do not — and the cost of believing otherwise is paid in the gap between something looking finished and somebody finding out it isn't.

It will do work that was already done

Twice in one night, we asked for something to be built, and it was built — well, quickly, and for no reason. It already existed. Someone had finished it days earlier.

The second time was worse than wasted effort. The replacement was not quite the same as the original. The original had three safety rules in it that the new version did not, because the person who wrote the original had thought about the problem for longer. Nobody noticed from the outside. Both versions looked finished. Both passed their checks.

The lesson is not "machines are forgetful". It is that enthusiasm to build is not the same as knowing whether to build, and only one of those got cheaper.

It will prove itself right using its own assumptions

One part had to read a public information feed. An old note in our own files described the format of that feed precisely. The work was done from the note. The test — the check that proves the work is correct — was also written from the note.

So the two agreed. The check passed. And against the real feed, the thing returned nothing at all, silently, because "nothing" is a perfectly valid answer. The format had changed years ago and the note had not.

This is the most ordinary failure in automated work and the hardest to see: when the same mind writes the work and the test of the work, the test proves they agree with each other, not that either is true. A person reviewing their own essay has the same problem. We just stopped expecting the problem to be there, because a green tick looks like an outside opinion.

It will tell you a number that does not exist

Earlier in this project, a tool wrote a daily summary in plain English. One of its summaries reported a specific loss, to the rupee, on a specific date. The figure was invented. Not exaggerated — invented. Nothing in any file said it.

It is worth sitting with how that reads to a human being. A confident sentence with a precise number in it is the single most trusted form in business writing. We read the specificity as evidence. That instinct is correct about people, who find specificity effortful, and wrong about machines, which find it free.

What we do now is dull and works: the tool may only use a number it can point to in a file, and if it cannot find one it must write unknown. It is allowed to be useless. It is not allowed to be confident.

It will build something that looks exactly like success

This is the expensive one.

We added a cautious little rule: when something important has just been officially announced, widen the system's uncertainty. Sensible. It was deliberately switched to "watch only" — it records what it would have done and changes nothing.

Within minutes of running against the real world, we had our answer. Official bodies publish constantly. On a quiet Sunday evening, in a ninety-minute window, the rule found 44 qualifying announcements. It would have fired on essentially every cycle, forever.

A rule that always fires is not a rule. It is a constant with a story attached. Had it been switched on, nothing would have broken, no alarm would have sounded, and a dashboard somewhere would have shown a feature working — for months. The failure that costs you most is not the one that crashes. It is the one that looks like it is working.

It cannot tell you when something is actually working

We spent four years' worth of data testing ideas in an area where people are extremely confident: automated trading.

We wrote down each question, and what would count as success, before looking at the answer. Then we checked each once. A famous, well-documented pattern lost money once realistic costs were charged. Five different ways of "stepping aside when things look risky" all did worse than doing nothing — one of them significantly worse. A clever adaptive system beat a simple fixed one, until we asked what a dumber version would have scored, and the dumber version matched it. Four ways of reading market direction scored between 49% and 52%, which is a coin.

One thing survived, and it is modest: the market reliably overestimates how big a coming move will be. Not which way. How big.

An AI asked to build any of those ideas would have built all of them, competently. That was never the hard part. The hard part is the thing that says no, and it is not a model — it is arithmetic plus the willingness to be disappointed.

How little it takes to be sure of nothing

Here is a number worth carrying around. If you try something twenty times and it works fourteen — a 70% hit rate, which feels decisive — the honest range for the true rate is roughly 46% to 88%. That range includes a coin flip.

Twenty is more evidence than most confident claims rest on. Most people, and most dashboards, declare a thing works long before that.

So what does it actually cost

Not money, mostly. It costs the time you spend believing.

Every example above would have survived. The duplicate would have run. The silent reader would have returned nothing for weeks. The invented number would have been quoted in a meeting. The rule that always fires would have shown green on a chart. None of it would have announced itself. You would have found out later, from an outcome, and then had to work backwards through everything you had built on top.

That is the bill: not a crash, but a quiet stretch of time in which you were confident and wrong, and built further on it.

What we do instead

Four things, none clever.

1. Make it touch the real thing early. Not a sample, not a description — the actual source, once, before anything is written around it. Almost every mistake above died on first contact with reality and would have survived any amount of further checking.
2. Write the rules before the work. The things that caught us were written weeks earlier by someone with no stake in that night's output. A constraint written in advance cannot be talked round and does not get tired at two in the morning. The ones written afterwards always somehow permit what you just did.
3. Make it say "unknown". Anything that reports must be able to admit it does not know, and must find that easier than guessing.
4. Decide what would count as success before you look. Then look once. This is the one people skip, because it mostly returns an answer you did not want.

We run a company where agents do most of the work, so none of this is an argument against using them. It is an argument about where the remaining human job actually sits. It is not building any more. It is deciding what is true — and being willing to hear that the answer is no.

This describes our own internal work and a private system that holds no customer money and cannot place an order. The figures are from our own records and pre-registered studies on public data. It is an account of how we work, not advice.

Also read: Brief Nº5 — Everything you are told to check is public. None of it is in one place. →

← All notes