AI Model Reviews

Is Claude Worth Paying 30x For? Suhail Doshi's Five-Task Test

Riley Brown likes Claude Fable 5.2 for editing video, building apps and making 3D films. He also remembers accidentally spending $12,000 in one month on AI API calls at a previous startup. When he asks Suhail Doshi how to tell when the expensive model earns its keep, Doshi's answer is refreshingly small: take five things you already do, give several models the same inputs, and see where you can tell the difference.

Keep the premium model for jobs where it changes what you can ship. For the rest, test a lower-cost option against your actual acceptance criteria. The number to watch isn't the price of a token. It's the cost of getting a result you can use.

Watch Riley Brown and Suhail Doshi

Source: Riley Brown's November 21, 2026 interview and its transcript. Doshi co-founded Mixpanel and founded Particle, which sells inference for lower-cost models. Brown's bill and Doshi's design tests are their own accounts, not independently audited results. Rates were checked on November 22.

What the 30x Figure Actually Means

In the interview, Doshi says GPT Astra and DeepSeek V4.1 Flash gave him similar results on some design tasks at sharply different costs. He describes Fable's output as somewhat better in his informal comparisons. The interview doesn't provide a controlled sample or scoring scale, so the title's 30x isn't a universal Claude markup or an independently measured saving.

Direct API exampleUncached inputOutput
Claude Fable 5.2$8.00$40.00
DeepSeek V4.1 Flash$0.15 off-peak / $0.30 peak$0.60 off-peak / $1.20 peak

USD per million tokens, based on provider rate cards. This table excludes cached input, tool fees, alternative providers and premium service tiers. It does not reconstruct Doshi's actual runs.

A rate card also doesn't tell you how many tokens a job will consume. Doshi's company offers an example: Particle reports 28.4% to 52.2% fewer output tokens for Particle-1 than its DeepSeek V4.1 Flash base across four evaluations. That's a vendor-reported result on those workloads. It doesn't establish the same savings for video editing, writing or your own tools.

The Five-Task Scorecard

An eval is a repeatable test with a definition of what counts as success. Like Simon Willison's recurring SVG drawing test, you want a task you can rerun when a new model comes out. Unlike a public benchmark, your test should reward the things you'd notice on an ordinary workday. Use this scorecard as a starting point, then swap in five jobs you actually do.

TaskPass checkWhat a human should judge
Talking-head editThe MP4 plays in sync, preserves the best take and contains no clipped words.Do the cuts help the story? Count minutes spent repairing them.
Script or newsletterIt follows the brief and adds no unsupported facts.Would you publish it in your own voice? How much rewriting did it take?
Recurring adminIt extracts the correct deadlines and owners from the test inputs without sending anything.Did it flag what needs your attention rather than repeat the whole inbox?
Small app changeThe target interaction works and the relevant regression tests pass.Can someone maintain the patch without unpicking unrelated changes?
Short 3D sceneThe editable scene opens and a test render meets the brief without missing assets.Does the camera movement tell the story? Is the finished shot worth the render time?

Download the Excel Scorecard

Download the five-task scorecard (.xlsx)

Log one row per attempt, including retries. The workbook calculates pass rate, cost per usable result, and total and average human cleanup minutes. Failed attempts still count toward spend and cleanup; the sample row is excluded. Filter by model or setup to compare the same five tasks.

  1. Freeze the inputs and pass checks. Use the same brief, sample files, tools and retry allowance. Record each model version, effort and provider. If you change both the model and the toolchain, label it as a workflow comparison.
  2. Grade blind where you can. Rename outputs A and B before judging. Mark pass or fail before scoring taste from 1 to 5. A beautiful export with broken audio still fails.
  3. Log every attempt. Include tool and model charges, elapsed time, retries and minutes of hands-on repair. Divide the total model-and-tool spend by the number of accepted results, and compare human time separately.
  4. Repeat what matters. Five tasks are a first pass, not a statistically robust ranking. Retest close calls on fresh examples before changing a recurring workflow.

Doshi suggests turning the history of a good edit into reusable test criteria. Your exported cut and notes about what you changed are enough. Use approved material and remove private footage, account details and client information before sharing examples with another service.

JQ AI SYSTEMS starter prompt

Design a blind five-task model eval

Ready to copy

Why Always-On Agents Change the Bill

A single cheap answer isn't the right unit for an agent that runs on every new ticket or every hour. More runs, longer contexts, tool calls and retries can raise the total bill even when the unit rate falls. The interview's forecasts about subsidies and consumer pricing are predictions. For budgeting today, distinguish included subscription usage, API credits and separately metered usage instead of treating them as interchangeable.

Once a routine job reliably passes on a cheaper model, use that as the default. Escalate if a check fails, if the job is unusually ambiguous, or if it's an important final creative pass. Brown and Doshi discuss OpenRouter as a place to compare alternatives. Record the provider as well as the model, and check data-handling terms before sending work files. Cap spend and retries for scheduled jobs, and keep a person in the loop before anything gets published or sent.

Here's why a cheap model can lose: in an illustrative 100-run test, $12 of spend for 80 usable results is $0.15 each. A $30 run budget that yields 96 usable results costs about $0.31 each. If the second option consistently saves several minutes of cleanup on each accepted job, the extra spend may be worth it. If no one can tell the difference, keep the cheaper route.

Headless Editing and the GTA Benchmark

Brown says he gets better results on some edits when the agent works directly with files through FFmpeg instead of controlling an editor. For 3D work, Blender supports rendering without a graphical interface. These are examples of changing the tools a model can use, not proof that every model edits better without a UI. Headless work also makes it easier to lose track of the latest export. Protect the originals, use versioned outputs, and ask for a short edit log and a preview you can watch.

Doshi's proposed benchmark starts with the original Grand Theft Auto and gets harder over time, with GTA VI as the distant target. It's a proposal, not a published leaderboard or a claim that an agent rebuilt these games. Its useful insight: a screenshot can't tell you whether a game is playable. With an original or rights-cleared mini-game, grade controls, whether a mission can actually be completed, and whether the game still works after playing for a while.

What to Learn When More Work Can Be Done With Code

Doshi's claim that software engineering is dead is his view of how the job is changing, not a workforce finding. He suggests learning how tokens and caches affect cost, how models are trained, and how to evaluate them instead of just following release announcements. For a creator, the equivalent is knowing which parts of a story to keep, what to verify in the final file, and how to recover when the tooling goes wrong.

Doshi points to Paul Graham's How to Do Great Work when talking about curiosity. He sees an arm drawing a picture as a starting point for experimenting with robotics, not a sign that real-world reliability is solved. His closing question is more useful than a forecast about robotics or jobs: what would you try to make now that used to need a team? Try one finished scene, one playable level or one useful tool. Then pay for the model that helps you finish it.

Video Chapters

TimeTopic
00:00Intro
01:11First impressions of the new AI models
03:28The GTA benchmark: From AI demos to real games
08:04AI pricing, subsidies and always-on agents
27:10When to switch to cheaper open models
30:07Building your own evals for real work
33:55Every work task is becoming a coding task
42:08Adapting your career and following curiosity
51:12Scaling your ambition: Work, creativity and robotics
Share
X LinkedIn Reddit
Build Yours

Want a system
like this one?

Book a free 30-minute call. We map your situation, identify the highest-impact automation, and figure out if we are a fit.

Book Free 30-min Call