Enterprise computing rests on an implicit contract that is so old it has become invisible: given the same input, a system produces the same output. Every layer of institutional infrastructure built over the last fifty years — testing regimes, change management, audit trails, reconciliation processes, SOX controls, incident response — presupposes this contract. When a batch job produces a wrong number, you can replay it, bisect it, and find the defective line. The defect is stable. It will be there when you go looking for it.
Large language models break this contract. Not as a bug, but as a property of what they are. The same input can produce different outputs; semantically equivalent inputs can produce materially different behavior; and the failure modes are not stable enough to bisect. The industry has mostly responded to this by debating which model is best. That is the wrong debate. Model choice is a procurement decision that will be revisited every six months. The durable decision — the one that will still be shaping your organization in five years — is architectural: where, precisely, do you place the boundary between the deterministic and stochastic parts of your systems?
This essay is an attempt to treat that question with the seriousness it deserves, including the parts that are genuinely unresolved.
The compounding problem
Start with arithmetic that is unglamorous but decisive.
Suppose an agent executes a multi-step task, and suppose — generously — that each step succeeds 99% of the time. The probability that a 20-step chain completes without error is 0.99²⁰ ≈ 82%. At 50 steps, it is roughly 61%. At a more realistic 95% per-step reliability, a 20-step chain succeeds about 36% of the time.
Three things follow from this, and none of them depend on which model you use.
First, long autonomous chains are an architecture smell unless something breaks the exponential decay. The only things that break it are verification and recovery: steps whose outputs can be checked, and failures that can be caught and retried without corrupting state. An agent architecture is really a claim about where verification happens.
Second, per-step reliability improvements are enormously leveraged. Moving from 95% to 99% per step doesn't improve a 20-step chain by four points — it improves it from 36% to 82%. This is why frontier labs are correct that model capability matters, and why enterprises are also correct that it isn't sufficient. Both parties are looking at the same exponent from different sides.
Third — and this is the part most analyses omit — the arithmetic cuts the other way for verification-rich environments. If each step's output is independently checkable and failed steps can be retried, chain length stops being the binding constraint. The binding constraint becomes the cost and latency of verification. This reframing matters more than any capability announcement of the last two years: the economically tractable applications of agents are not determined by task difficulty, but by verification asymmetry — the gap between the cost of doing a task and the cost of checking it.
Code is the canonical example, and it explains why software engineering became the first serious agent market. A generated function is cheap to verify: it compiles or it doesn't, tests pass or they don't, and the type system catches a class of errors at zero marginal cost. Contrast this with, say, a generated strategic analysis, where verification requires a competent human reading the whole thing — verification costs approximately as much as generation. The first domain supports long agent chains. The second does not, regardless of how good the model is, because errors accumulate faster than anyone is checking for them.
A practical corollary for anyone building a roadmap: rank candidate processes not by how expensive they are to perform, but by how cheaply their outputs can be verified. This produces a different, and better, portfolio than the "automate our most expensive workflows" heuristic that dominates board decks.
Workflows, agents, and the honest middle
The word "agent" is doing too much work in enterprise conversations, so it is worth being precise. There is a spectrum with two well-understood endpoints.
At one end: deterministic orchestration with stochastic components. The control flow is conventional code — branches, retries, queues, state machines. LLM calls sit inside that structure as functions: classify this document, extract these fields, draft this response for review. The system's topology is fixed at design time. Anthropic's engineering guidance calls these workflows, and the label is apt because everything the enterprise already knows about workflow systems — idempotency, dead-letter queues, replay, audit — transfers directly.
At the other end: model-directed execution, where the model decides what to do next, which tools to invoke, and when the task is done. The topology is determined at runtime. This is what "agent" properly means, and it is a categorically different engineering artifact, because the thing you must now test is not a path through a graph but a policy over an open-ended action space.
The uncomfortable truth is that most enterprise value captured to date lives at the workflow end, while most enterprise attention lives at the agent end. This is not because agents don't work. It is because the workflow end lets you spend fifty years of accumulated reliability engineering, while the agent end forces you to invent replacements for it. The right question for any given process is not "can an agent do this?" — increasingly, yes — but "what does failure cost, and what machinery exists to catch it?"
That machinery has a name in every mature engineering discipline: containment. For model-directed systems, containment decomposes along two axes that should be scored explicitly for every action an agent can take: reversibility (can this action be undone — a draft, a staged change, a sandboxed execution — or is it irrevocable — a payment sent, an email delivered, a record deleted?) and blast radius (does a failure affect one record, one customer, or the ledger?). Actions that are reversible with a small blast radius can be delegated with light oversight. Actions that are irreversible with a large blast radius should require deterministic gates — not because models are untrustworthy in some moral sense, but because no statistical system, human or machine, should sit unmediated on an irreversible high-blast-radius action. Banks internalized this long ago: it is why wire transfers have maker-checker controls even though the humans involved are competent. Agent architecture is not a new discipline. It is dual-control, rediscovered under conditions where one of the parties is a distribution.
Testing is dead; long live evaluation
Deterministic systems are validated by testing: enumerate the cases that matter, assert exact outputs, run the suite on every change. This does not survive contact with stochastic components, and the failure is subtle enough that many organizations discover it in production. A test that passes tells you one sample from a distribution was acceptable. It tells you almost nothing about the distribution.
The replacement discipline is evaluation: measuring the distribution of outcomes over representative input populations, with graded rather than binary scoring, tracked over time. This sounds like testing with extra steps. It is not — it differs in three ways that have organizational consequences.
Evaluation is statistical, so it requires input populations that actually represent production traffic, which means someone must own the pipeline that samples, sanitizes, and labels production data. In most enterprises no such role exists, and eval sets quietly rot into unrepresentativeness.
Evaluation is continuous, because the system changes underneath you even when your code doesn't. Model version updates, prompt drift, upstream data shifts — any of these can move the output distribution without a single line of your code changing. An eval suite that runs at release time is a smoke detector installed in the wrong room.
And evaluation is expensive to maintain in a way test suites are not, because graded scoring at scale generally means either paid human review or model-graded evaluation — and the latter imports the reliability question one level up. Model-graded evals are useful and widely used; they are also a correlated-error risk, since the grader and the graded often share blind spots. The honest position is that model-graded evaluation is a cost optimization to be validated against human judgment on a sampled basis, not a substitute for it.
Anyone budgeting an AI program should treat evaluation infrastructure as a first-class line item — comparable to what test infrastructure costs a serious software organization, which is to say: substantial, permanent, and the first thing that degrades under schedule pressure.
The observability story runs parallel. Logs answer "what happened"; for stochastic systems you need traces that answer "what did the model see, what did it decide, and what would it likely have done otherwise" — full prompt/context capture, tool-call sequences, and the sampled counterfactuals that make post-hoc analysis possible. Enterprises that treat this as optional telemetry will find, at their first serious incident, that they have built a system whose failures cannot be investigated. For a regulated institution, an uninvestigatable failure is not a technical debt item. It is a supervisory finding.
Governance is control engineering, not documentation
Financial institutions have an underappreciated head start here. The Federal Reserve's SR 11-7 guidance on model risk management — written in 2011 for credit and market risk models — already contains the correct skeleton: models are approximations whose errors have consequences; they require independent validation, ongoing monitoring, inventories, and defined use boundaries. A bank CIO does not need a new governance philosophy for LLMs. They need to extend an existing one to models that are larger, more general, and — critically —behaviorally underspecified: nobody, including their creators, can fully enumerate what they will do outside the evaluated envelope.
That last property changes governance mechanics more than governance principles. A policy document that says "the model shall not execute trades above threshold X" is worth exactly as much as the runtime enforcement behind it. Effective AI governance is therefore mostly systems engineering: permission systems that scope what an agent can do rather than instructions about what it should do; deterministic policy gates on the irreversible/high-blast-radius quadrant; kill switches that degrade gracefully to human workflows; and audit trails rich enough to reconstruct decisions. The document describes the mechanism. The mechanism is the governance.
This has an implication that vendors are not incentivized to state plainly: instruction-following is not a security boundary. A model told to stay within limits is a model that will usually stay within limits. "Usually" is an acceptable property for a drafting assistant and an unacceptable one for a payment authorization path. The architecture, not the prompt, must carry the guarantee — the same principle, incidentally, that separates input validation from asking users nicely.
The strongest argument against everything above
Intellectual honesty requires steelmanning the position that most threatens this essay's thesis, so here it is.
Rich Sutton's "bitter lesson" observes that in AI research, elaborate human-engineered structure is repeatedly outperformed, then obsoleted, by general methods riding computational scale. A version of that argument applies to enterprise AI architecture: much of the scaffolding described above — decomposition into verifiable steps, checkpoints, constrained action spaces — exists to compensate for models that cannot yet sustain long coherent execution. Per-step reliability has been improving; coherence horizons have lengthened materially year over year. If that continues, some of today's carefully engineered containment will look like the elaborate feature engineering that preceded deep learning: rational at the time, wasted in retrospect. The frontier labs believe some version of this. They are not obviously wrong.
Two considerations cut back, and they define where the argument actually lands.
First, the compounding arithmetic never goes away; it relocates. Better models don't eliminate the exponent — they let you spend it on longer, more valuable chains. Autonomy expands to consume reliability gains, the same way traffic expands to fill highways. The verification question ("how do we know this ran correctly?") therefore survives every capability improvement, because it is asked anew at each longer horizon. And institutions do not adopt at the frontier of what is possible; they adopt at the frontier of what is accountable — a line that regulation, fiduciary duty, and reasonable caution all hold well behind the capability frontier, indefinitely.
Second, and more practically: even if aggressive capability forecasts hold, organizations still need architecture for the transition, which is measured in years and will be traversed while running systems that real money and real customers depend on. The rational posture is to build scaffolding while treating it as partially disposable — biasing toward containment mechanisms that remain valuable as models improve (evaluation infrastructure, permissioned tool access, audit trails, verification harnesses) and away from mechanisms that merely compensate for current weakness (fine-grained task decomposition, elaborate prompt pipelines, rigid step-by-step choreography). The first category is infrastructure. The second is workaround. Budget accordingly, and expect to write off the workarounds.
What actually compounds
Strip away the parts of this essay that may be obsoleted by the next model generation, and a residue remains that will not be, because it is not about models at all.
Organizations that deploy these systems well are forced to make their work legible: to specify what processes actually are, what "correct" means for each, what data the process consumes and emits, and who is accountable when it goes wrong. That specification work — process contracts, data contracts, ground-truth definitions, verification criteria — is painful, unglamorous, and almost entirely absent from enterprises today, where the real process lives in the heads of tenured employees and diverges freely from the documented one. It is also the single strongest predictor of whether an AI deployment produces value or an expensive incident, because a stochastic system pointed at an unspecified process automates the ambiguity along with the work.
This is why "which LLM is best?" is the wrong question in a deeper sense than vendor neutrality. Models are becoming the most fungible component in the stack — swappable, benchmarkable, priced increasingly like a commodity with a quality gradient. What is not fungible is an institution's accumulated verification infrastructure, its evaluation assets built from its own production data, its legible process definitions, and an operating model that knows which decisions to delegate to a distribution and which to gate behind determinism. Those assets compound. They survive model migrations. And they cannot be purchased from any vendor, including us, because they are made of organizational decisions rather than software.
The enterprises that understand this are not asking which model to buy. They are redesigning work under a new assumption: that cognition is becoming a system component with a cost curve, a latency, an error distribution — and an exponent that must be engineered around. The boundary between deterministic and stochastic execution is the load-bearing wall of that redesign. Place it deliberately.
Rios Applied AI designs enterprise operating models for production AI systems. This essay reflects our current views; several of its open questions — coherence horizons, eval amortization, the durability of containment scaffolding — are empirical, and we expect to be wrong about some of them.