Claude Fable 5 launched on June 9, and by the end of the week the US government had ordered its access suspended for foreign nationals, prompting Anthropic to pull public access to the model entirely while it sorts out what it calls a misunderstanding. By the time you read this, the model may be back, or it may not. That's its own story, and not the one I want to tell.
Here's the part that outlasts any single launch. Every time a new model ships, we all perform the same ritual. One prompt, one game clone, one verdict. The model in the headline changes. The ritual doesn't. And the ritual is the wrong way to decide whether a model matters for your organization.
The better method has been sitting in your issue tracker the whole time. More on that below.
The one-shot ritual
Every model launch produces the same genre of content. Build me a Flappy Bird clone. Build me a task manager. Build me a personal website. One prompt, screen recording, hot take. Sean Goedecke described this pattern well in his piece on why it takes months to know if a new model is good: most people fall back on vibe checks, pet questions, and artistic prompts like Simon Willison's famous request for an SVG of a pelican riding a bicycle. Fun, memorable, and as Goedecke notes, disconnected from the skills real work depends on.
The one-shot build has three problems as an evaluation method.
It tests the wrong distribution. A game clone is greenfield, zero-to-one work. Enterprise software development is overwhelmingly brownfield: bug fixes, migrations, regulatory changes, and features bolted onto systems written years ago by people who have since left. If your team spends 80% of its time modifying existing code, an evaluation built entirely on net-new toy apps tells you almost nothing.
It has a sample size of one. Models are non-deterministic. Kili Technology's evaluation guide cites research on enterprise agentic AI showing reliability dropping from 60% on a single run to 25% when measured across eight consecutive runs. An agent that nails your one-shot demo may fail three out of four times in sustained operation. One run is an anecdote, not a measurement.
It rewards the wrong outcome. A one-shot build is judged on whether something appears on screen. Real work is judged on whether the change is correct, complete, and safe to merge. Those are different bars. The Vibe Code Bench paper from earlier this year made the gap concrete: across 16 frontier models building complete web applications from specifications, the best model reached 61.8% accuracy. Zero-to-one app building is genuinely unsolved. It's also not the job.
What the launch evidence actually says
Here's the irony. The strongest evidence for Fable 5 has nothing to do with one-shot builds. It's brownfield evidence.
The flagship example in Anthropic's announcement is a migration. Stripe reported that Fable 5 performed a codebase-wide migration in a 50-million-line Ruby codebase in a single day, work Anthropic says would have taken a full team over two months. That's not a demo. That's the least glamorous category of work in enterprise software, done at a scale that changes project economics.
The benchmark story points the same direction. Anthropic reports 80.3% on SWE-Bench Pro, which tests resolving real issues in real repositories, and the top score among frontier models on Cognition's FrontierCode, which requires solutions that meet production codebase standards. Anthropic's own framing is explicit: the longer and harder the task, the larger Fable 5's lead. Their guidance to teams is to evaluate it on previously unsolved problems, not short snippets and easy issues.
Independent results add useful texture. CodeRabbit ran Fable 5 through its code review benchmark and found it close to but slightly behind Opus 4.8 on actionable precision, with a tradeoff they summarized as more depth but not always a clean finish. They also made a point worth keeping: at $10 per million input tokens and $50 per million output, you should evaluate this model on cost per solved task, not token price.
My own experience during the few days the model was available matches the brownfield story. I ran Fable 5 on the kind of work enterprises actually have: builds against an existing codebase, plus a steady diet of bug fixes. The first-shot correctness was the best I've seen from any model generation, including GPT-5.5 running in Codex, which was my previous benchmark for this kind of work.
But notice what that claim is. It's one practitioner, one backlog, a few days. An anecdote with a job title attached is still an anecdote. The reason I trust my read more than a launch-day hot take isn't the conclusion. It's the method. Which brings us to the point.
The hardest part of evaluating a model is building a test set that reflects your real work. Except you don't have to build it. You already have one. It's called your backlog, and the closed items in it come pre-labeled with correct answers.
Here's the method I've been using, and what I'd recommend to any engineering leader being asked "should we adopt this?"
Pull 10 to 15 recently closed items from your real backlog. Mix them: a couple of gnarly bug fixes, a small feature, a refactor that touched multiple files, one ambiguous ticket where the requirements needed interpretation. For each one, you already know what the correct fix was, because your team shipped it. That's a labeled evaluation set, and it cost you nothing to create.
Define success before the run. This is the discipline from last week's edition on loop engineering, applied one floor up. A loop needs a verifiable stop condition. An evaluation needs a verifiable success condition. Tests pass, behavior matches the shipped fix, no regressions in the surrounding module. Written down before you start, not judged by feel afterward.
Run each task more than once. Given the reliability gap between single-run and repeated-run performance, three runs per task is a reasonable floor. You're not just measuring whether the model can do the work. You're measuring whether it does the work dependably, because dependability is what determines whether you can change your delivery process around it.
Measure cost per solved task. Count the tokens, count the retries, count the human review time. A model that solves 9 of 10 tasks at a higher per-run cost can still be dramatically cheaper than one that solves 6 of 10 at a lower one.
My own test set came from a complex web application and its bug queue, because that's where my recent work has been. Nothing exotic: defects with known fixes, features built against an existing codebase, the occasional ticket where the requirements needed interpretation. Every closed item had a known correct answer and a known cost in human hours.
If your shop runs pension administration systems, your equivalents might be a vesting logic bug or a plan amendment that changed an early retirement reduction factor. The material doesn't matter. What matters is that it's your material, with answers your team already validated.
What your non-engineering colleagues will notice
The same evaluation logic extends past code, and this matters because the launch numbers suggest the knowledge-work gains are nearly as large as the coding ones.
Anthropic reports the top score of any model on Hebbia's Finance Benchmark for senior-level analytical reasoning, with the biggest gains in document-based reasoning and chart and table interpretation. On GDPval-AA, a measure of economically valuable professional work, Fable 5 scores 1932 against 1890 for Opus 4.8 and 1769 for GPT-5.5. Anthropic also reports that analytics platform Hex saw the first model to clear 90% on its long-running analytics benchmark, with everyday spreadsheet work finishing 25 to 30% faster.
For a pension fund, that capability set maps onto a specific pile of documents: actuarial valuation reports, plan texts, member statements, investment performance decks, regulatory filings. The work of pulling a number out of a chart buried on page 47 of a PDF and reconciling it against a spreadsheet is exactly what these benchmarks measure.
And the evaluation method is identical. Don't ask the model a clever finance trivia question. Take an analysis your team completed last quarter, where the conclusions were validated and the numbers are settled. Hand the model the same source documents and compare. Your finished work is the answer key.
The discipline keeps moving upstream
This is the third time the same lesson has shown up in this series. Harness engineering taught us that agent productivity comes from the environment you build around the model. Loop engineering taught us that autonomy is only as good as the verifiable stop condition you define before the loop starts. Evaluation is the same muscle again: the value of your judgment about a model depends entirely on the rigor you put in before you run it.
The pattern across all three is that the model keeps getting better and the human discipline keeps moving up a floor. We've gone from writing code, to writing specs, to building harnesses, to designing loops, and now to designing the evaluations that tell us when to trust all of the above.
So skip the game clone. Pull ten closed tickets, write down what done means, run each one three times, and count the cost per solved task. Fable 5 looks to me like a real step change on exactly the work enterprises care about. But you shouldn't take my word for it, and you definitely shouldn't take a screen recording's. Your backlog already knows the answer. Go ask it.