Riley Brown likes Claude Fable 5.2 for editing video, building apps and making 3D films. He also remembers accidentally spending $12,000 in one month on AI API calls at a previous startup. When he asks Suhail Doshi how to tell when the expensive model earns its keep, Doshi's answer is refreshingly small: take five things you already do, give several models the same inputs, and see where you can tell the difference.
Keep the premium model for jobs where it changes what you can ship. For the rest, test a lower-cost option against your actual acceptance criteria. The number to watch isn't the price of a token. It's the cost of getting a result you can use.
Watch Riley Brown and Suhail Doshi
Source: Riley Brown's November 21, 2026 interview and its transcript. Doshi co-founded Mixpanel and founded Particle, which sells inference for lower-cost models. Brown's bill and Doshi's design tests are their own accounts, not independently audited results. Rates were checked on November 22.
What the 30x Figure Actually Means
In the interview, Doshi says GPT Astra and DeepSeek V4.1 Flash gave him similar results on some design tasks at sharply different costs. He describes Fable's output as somewhat better in his informal comparisons. The interview doesn't provide a controlled sample or scoring scale, so the title's 30x isn't a universal Claude markup or an independently measured saving.
| Direct API example | Uncached input | Output |
|---|---|---|
| Claude Fable 5.2 | $8.00 | $40.00 |
| DeepSeek V4.1 Flash | $0.15 off-peak / $0.30 peak | $0.60 off-peak / $1.20 peak |
USD per million tokens, based on provider rate cards. This table excludes cached input, tool fees, alternative providers and premium service tiers. It does not reconstruct Doshi's actual runs.
A rate card also doesn't tell you how many tokens a job will consume. Doshi's company offers an example: Particle reports 28.4% to 52.2% fewer output tokens for Particle-1 than its DeepSeek V4.1 Flash base across four evaluations. That's a vendor-reported result on those workloads. It doesn't establish the same savings for video editing, writing or your own tools.
The Five-Task Scorecard
An eval is a repeatable test with a definition of what counts as success. Like Simon Willison's recurring SVG drawing test, you want a task you can rerun when a new model comes out. Unlike a public benchmark, your test should reward the things you'd notice on an ordinary workday. Use this scorecard as a starting point, then swap in five jobs you actually do.
| Task | Pass check | What a human should judge |
|---|---|---|
| Talking-head edit | The MP4 plays in sync, preserves the best take and contains no clipped words. | Do the cuts help the story? Count minutes spent repairing them. |
| Script or newsletter | It follows the brief and adds no unsupported facts. | Would you publish it in your own voice? How much rewriting did it take? |
| Recurring admin | It extracts the correct deadlines and owners from the test inputs without sending anything. | Did it flag what needs your attention rather than repeat the whole inbox? |
| Small app change | The target interaction works and the relevant regression tests pass. | Can someone maintain the patch without unpicking unrelated changes? |
| Short 3D scene | The editable scene opens and a test render meets the brief without missing assets. | Does the camera movement tell the story? Is the finished shot worth the render time? |
Download the Excel Scorecard
Download the five-task scorecard (.xlsx)
Log one row per attempt, including retries. The workbook calculates pass rate, cost per usable result, and total and average human cleanup minutes. Failed attempts still count toward spend and cleanup; the sample row is excluded. Filter by model or setup to compare the same five tasks.
- Freeze the inputs and pass checks. Use the same brief, sample files, tools and retry allowance. Record each model version, effort and provider. If you change both the model and the toolchain, label it as a workflow comparison.
- Grade blind where you can. Rename outputs A and B before judging. Mark pass or fail before scoring taste from 1 to 5. A beautiful export with broken audio still fails.
- Log every attempt. Include tool and model charges, elapsed time, retries and minutes of hands-on repair. Divide the total model-and-tool spend by the number of accepted results, and compare human time separately.
- Repeat what matters. Five tasks are a first pass, not a statistically robust ranking. Retest close calls on fresh examples before changing a recurring workflow.
Doshi suggests turning the history of a good edit into reusable test criteria. Your exported cut and notes about what you changed are enough. Use approved material and remove private footage, account details and client information before sharing examples with another service.
Design a blind five-task model eval
Why Always-On Agents Change the Bill
A single cheap answer isn't the right unit for an agent that runs on every new ticket or every hour. More runs, longer contexts, tool calls and retries can raise the total bill even when the unit rate falls. The interview's forecasts about subsidies and consumer pricing are predictions. For budgeting today, distinguish included subscription usage, API credits and separately metered usage instead of treating them as interchangeable.
Once a routine job reliably passes on a cheaper model, use that as the default. Escalate if a check fails, if the job is unusually ambiguous, or if it's an important final creative pass. Brown and Doshi discuss OpenRouter as a place to compare alternatives. Record the provider as well as the model, and check data-handling terms before sending work files. Cap spend and retries for scheduled jobs, and keep a person in the loop before anything gets published or sent.
Here's why a cheap model can lose: in an illustrative 100-run test, $12 of spend for 80 usable results is $0.15 each. A $30 run budget that yields 96 usable results costs about $0.31 each. If the second option consistently saves several minutes of cleanup on each accepted job, the extra spend may be worth it. If no one can tell the difference, keep the cheaper route.
Headless Editing and the GTA Benchmark
Brown says he gets better results on some edits when the agent works directly with files through FFmpeg instead of controlling an editor. For 3D work, Blender supports rendering without a graphical interface. These are examples of changing the tools a model can use, not proof that every model edits better without a UI. Headless work also makes it easier to lose track of the latest export. Protect the originals, use versioned outputs, and ask for a short edit log and a preview you can watch.
Doshi's proposed benchmark starts with the original Grand Theft Auto and gets harder over time, with GTA VI as the distant target. It's a proposal, not a published leaderboard or a claim that an agent rebuilt these games. Its useful insight: a screenshot can't tell you whether a game is playable. With an original or rights-cleared mini-game, grade controls, whether a mission can actually be completed, and whether the game still works after playing for a while.
What to Learn When More Work Can Be Done With Code
Doshi's claim that software engineering is dead is his view of how the job is changing, not a workforce finding. He suggests learning how tokens and caches affect cost, how models are trained, and how to evaluate them instead of just following release announcements. For a creator, the equivalent is knowing which parts of a story to keep, what to verify in the final file, and how to recover when the tooling goes wrong.
Doshi points to Paul Graham's How to Do Great Work when talking about curiosity. He sees an arm drawing a picture as a starting point for experimenting with robotics, not a sign that real-world reliability is solved. His closing question is more useful than a forecast about robotics or jobs: what would you try to make now that used to need a team? Try one finished scene, one playable level or one useful tool. Then pay for the model that helps you finish it.
Video Chapters
| Time | Topic |
|---|---|
| 00:00 | Intro |
| 01:11 | First impressions of the new AI models |
| 03:28 | The GTA benchmark: From AI demos to real games |
| 08:04 | AI pricing, subsidies and always-on agents |
| 27:10 | When to switch to cheaper open models |
| 30:07 | Building your own evals for real work |
| 33:55 | Every work task is becoming a coding task |
| 42:08 | Adapting your career and following curiosity |
| 51:12 | Scaling your ambition: Work, creativity and robotics |