It Worked Once

A surgeon who succeeds ninety percent of the time is a miracle in a slide deck and a malpractice suit in a hospital. Same number. Opposite verdict. The gap between the two is not the surgeon’s skill. It is the difference between watching a thing happen and living with how often it does.

We keep those two straight everywhere except software. A ninety percent success rate would get a pilot grounded, a bank shut down, a bridge condemned. Show the same rate in an AI demo and the room leans in. It worked. Ship it.

It worked once.

That is the whole trap. A demo proves that a thing is possible. It proves the ceiling exists. It says nothing about the floor, and production lives on the floor.

Possibility Is Not Reliability

Watching a model do something once tells you the good outcome is reachable. That is real information. It is also one bit of it. “Can this happen” and “how often does this happen” are different questions with different answers, and a demo only ever answers the first.

People collapse the two constantly. They see a clean run and their brain files it under works, permanently, the way it would file a fact. But reliability is not a fact you can observe in a single sighting. It is a rate. You cannot see a rate happen. You can only measure it, over many runs, most of which are boring and some of which you would rather not look at.

The demo shows you the run worth looking at. That is the problem.

The Sample Was Chosen

A demo is not a random draw from production. It is the best draw its maker could find, under conditions its maker controlled.

The input was hand-picked. The prompt was tuned by someone who already knew what good output looked like. The run that fell over happened yesterday, off camera, and got cut. Even an honest demo is selection, because you build a demo by running the thing until it works and then showing that time. The dishonest ones just stack the deck harder. Either way, what you are handed is the ceiling a second time, dressed up as the average.

So when the demo lands and the instinct says ship it, notice what actually happened. You did not observe that the system is reliable. You observed that reliability is possible, on a curated input, with an expert steering. Production will hand it the inputs nobody curated, with nobody steering, forever.

Reliability Multiplies

Now the part the demo hides on purpose, because it cannot survive being shown.

Give an agent a task that takes ten steps. Say each step is ninety percent reliable, which for real work is generous. The odds it gets the whole chain right are not ninety percent. They are about thirty-five. Reliability multiplies down a chain. It does not average.

This is the entire reason a short demo glows and a long task collapses. Every step you add is another roll, and the rolls compound against you. The demo was three steps in a clean room. The real job is forty steps in the rain. Same model, same per-step competence, and the outcome went from magic to coin flip because the task got longer, which is the one thing production always does.

Watch for the demo that is impressive and short. The length is not incidental. It is the reliability budget being spent where you can see it work.

The Question With No Answer

“Does it work” has no answer, because it is the wrong question. Break it into the two it is hiding. How often. And what happens the times it doesn’t.

The second one matters more and gets asked less. An agent that fails loudly, that stops and says it could not do the thing, is something you can build around. You catch it, you retry, you route a human in. An agent that fails the way these actually fail, by producing something plausible, confident, and wrong, and handing it over without a flag, is a different animal entirely.

A ten percent failure rate you can see is an annoyance. A ten percent failure rate you cannot see is a time bomb with a friendly face. The failures do not announce themselves. They look exactly like the successes, right up until the one that costs you something. This is why verification is the whole game and generation is the cheap part. You are not trying to make the model produce output. It does that already. You are trying to be able to tell which of its outputs you can trust, which is the harder problem and the one nobody demos.

Production Is the Distribution

So the real work was never getting it to work once. You did that in the first hour. That is the part that demos, and it is over.

The work is dragging the floor up and pulling the variance in, until the bad draws are rare enough and cheap enough to live with. Bounding what the agent is allowed to attempt. Checking the output before anything downstream trusts it. Structuring the task so there are fewer places for a step to go sideways. None of it demos well, because reliability is invisible when it is working. A system that quietly does the right thing on every input looks like nothing is happening. That blankness is the tell that it is good, and the reason nobody in the room claps.

It is the same reason a ninety percent employee gets managed out. Their capability was never in question. They can clearly do the job. They just cannot be counted on to, and counted-on is the entire value of an employee. Capability is table stakes. Consistency is the product.

Two People Holding Samples

Watch any argument about whether agents are ready and you will see it. One person has a great run they will not stop describing. The other has a disaster they will not let go of. Both are telling the truth. Both things happened.

And the argument goes nowhere, every time, because a great anecdote and a terrible anecdote are the same kind of object. Neither is a rate. You cannot settle a question about a distribution by trading its outliers back and forth across a table. The optimist is describing the ceiling. The skeptic is describing the floor. They are looking at the same system and neither has measured it.

The only move that ends the argument is the unglamorous one. Run the real task, on real inputs, enough times to get a number, and look at what the failures cost. Almost nobody does this. It is slow, it produces a chart instead of a moment, and the demo already felt like proof.

A demo was never the deliverable. It tells you the thing is possible, which is worth knowing and easy to mistake for more than it is. Whether it is reliable is a different question, a harder one, and the only one that was ever going to decide whether you can use it.