Field NoteDemonstration

The Diagnostic had to pass its own test.

Before we asked anyone to trust our methodology on their workflows, we ran it on our own tool — and the tool failed its first review.

Rios Applied AI · July 2026 · ~6 min read

A studio with no client case studies owes its visitors a different kind of proof. Ours is this: before we ask anyone to trust our methodology on their workflows, we ran it on our own tool — and the tool failed its first review.

The Workflow Diagnostic is the free instrument on this site: sixteen questions, about four minutes, and you get a preliminary read on one workflow — legibility grade, verification class, risk class, a recommended maximum autonomy tier. The obvious way to build it in 2026 is also the fastest: hand the visitor's answers to a language model and let it write an assessment. A weekend of work, and the demo would look identical to what we shipped.

We didn't build it that way, and the reason is the point of this note:the Diagnostic is itself a workflow that delegates work to an AI system, which means it has to score acceptably on its own rubric. So before writing code, we assessed the tool we were about to build, using the same classifications it would apply to visitors' workflows.

The assessment was not flattering.

Consider what the naive version actually is. The task — "read a stranger's description of their workflow and render a judgment about automation risk" — is judgment-dense work: reasonable experts can disagree about a verdict, which makes it W4 on our complexity scale. Verifying any single output means an expert reading the whole thing, so verification costs roughly what generation costs: V3. And the blast radius of a wrong answer is not contained. A shallow or confidently wrong assessment, rendered under our name, to a managing partner evaluating whether to hire us — that's a reputational action with no undo. Run those classifications through the same autonomy lookup we'd apply to a client's workflow and the answer comes back: a system like this does not get to operate autonomously in public. Our own tables said don't build it.

What our tables permit is narrower: move everything load-bearing across the determinism boundary. So the shipped architecture splits the work in two. Every classification — the grades, the classes, the tier, the recommended pattern — is computed by deterministic, versioned, unit-tested code from structured answers. Same answers, same result, forever; you can open the browser's developer tools and read the rubric running. The architecture reserves exactly one place a model could ever sit — below the boundary, where its failure would be survivable, drafting the narrative memo that explains classifications it cannot alter. In v1 we left that slot empty: even the memo is assembled deterministically from templates keyed to the computed results, so no model runs anywhere in the tool. If we ever fill that slot, the methodology page changes first, as a published event. The memo is convenience; the verdict is code — and in v1, so is the memo.

Writing the rubric as code turned out to be the expensive part — and the instructive one. Prose tolerates ambiguity that software refuses. Our own taxonomy line between "reversible with effort" and "partially irreversible" had lived comfortably in essays for months; a lookup table demanded we decide, once, which side specific cases fall on, and the deliberation took longer than implementing the entire scoring engine. We also hit a case where two answers can honestly contradict each other — a visitor reports that errors are caught by a simple check, then reports that checking an output is harder than producing it. The tempting fix was to let the model reconcile the contradiction quietly. We chose the opposite: the engine flags the inconsistency to the visitor and explains which answer it privileged. An honest inconsistency note builds more trust than silent resolution — and silently resolving contradictions is precisely the failure mode we warn clients about.

Two limitations, stated plainly because this note is proof only if it includes them. First, the Diagnostic measures self-report, and self-report measures the documented process — the version of the workflow that lives in the owner's head. Our own methodology says the answer usually changes when you observe the practiced process; the gap between the two is legibility debt, and no questionnaire can see it. That is not a modest disclaimer, it is a structural boundary of the instrument, and it is why the paid Assessment exists. Second, a sixteen-question rubric cannot assign the extreme classes honestly — it will never output W1 or W5, and we capped what it claims accordingly.

The counterargument deserves its hearing: a frontier model reading sixteen answers would probably produce a reasonable assessment most of the time, and a more fluent one. "Most of the time" is exactly the property at issue. A drafting assistant can be usually right; an instrument that renders verdicts under a firm's name, to strangers, with no reviewer in the loop, cannot — not because models are weak, but because that combination of judgment-density, verification cost, and irreversibility fails the delegation test no matter whose model you use. When the classifications change — when verification becomes mechanical, when a human reviews each output — the boundary can move. That is the whole method: the boundary follows the classes, not the technology.

The Diagnostic is a small system. But it is, as far as we can tell, the only workflow-assessment tool that can show you the assessment of itself — and the assessment said: keep the model away from the verdict. We took our own advice. That, more than anything the tool outputs, is what we'd offer as proof.

Try it yourself: the Workflow Diagnostic. How it decides:the methodology exhibit.