Four Claude runs, one songwriting task, and the biggest gap in the whole benchmark came from a single model measured against itself twice. This is what a model-efficiency test looks like before it has earned the right to name a winner.
I ran an identical songwriting brief through four back-to-back generations on Suno, across three Claude models in one sitting: Opus twice, Sonnet once, Haiku once. Same required structure, same repeated hook line, same Suno style tags, same execution steps.
Two runs of the exact same model, on the exact same task, produced model-controlled token counts of 52,062 and 273,147. That is a 5.25x spread, from a single model measured against itself twice. If a model can disagree with itself by that much, a single run per model can do no better than point at a winner.
One disclosure before the numbers: I built the measurement harness myself, inside the same session I was measuring, running Anthropic's own models. Nobody at Anthropic reviewed or funded any of this, and I went in already curious whether Sonnet would come out ahead. Read what follows as a workbench log.
The naive way to compare models is to add up every token in the transcript and call the smallest number the winner. That number is close to meaningless here, and the reason is prompt caching.
Claude's API charges cache-read tokens, the context you resend, at a tenth of the base input price, and charges cache-write tokens, the context getting newly cached on the default five-minute tier, at one and a quarter times the base price (Anthropic, 2026). Across all four runs, over 99 percent of every token counted was cache traffic: context getting resent or re-cached from turn to turn. Output, the part the model actually controls, was under 1 percent of the total. A raw token count mostly measures how long the conversation had gotten by the time a given run started.
That is the same shape I found auditing three months of real sessions in the last post in this series, where 86 percent of the bill turned out to be resent context. A different measurement landed on the same underlying fact: context is the expense, and output is the part everyone thinks they are paying for.
So I built a narrower number for this test: model-controlled tokens, defined as output tokens plus 1.25 times cache-write tokens. It drops cache-read entirely, since that is context history the model neither generated nor chose. What is left is closer to "how much did this model actually produce to get the job done," which is the question a token-efficiency comparison is supposed to answer.
On wall clock, Sonnet finished the whole songwriting-plus-Suno workflow in 148.8 seconds, 29 percent faster than Haiku's 209.6 and 44 percent faster than Opus's slower run at 265.4. Sonnet also used the fewest tool calls of any run, 12 against Haiku's 26, while logging the fewest tool errors, one against Opus's six on its slower pass. On the raw numbers, Sonnet looks like the efficient, predictable option.
But the token-cost metric complicates that story. Opus's better run used 52,062 model-controlled tokens, about 11 percent fewer than Sonnet's 58,471. The same model that spiked to 273,147 tokens on its first pass also produced the single cheapest run in the whole set. Speed and token cost picked different favorites, and the model with the widest range beat everyone once.
Haiku's story is a mechanism. It got things wrong about as often as the others, two tool errors against Sonnet's one, and it reached the same result through 55 API calls instead of Sonnet's 21, breaking the same handful of actions into smaller steps. That is a strategy difference in how a smaller model sequences tool use, and it is the direct reason its wall clock trailed Sonnet despite comparable output token counts.
| Run | Model-controlled tokens | Wall clock | Tool errors | Cache reads |
|---|---|---|---|---|
| 1 · Opus | 273,147 | 265.4s | 6 | 2.0M |
| 2 · Opus | 52,062 | n/a | n/a | 3.2M |
| 3 · Sonnet | 58,471 | 148.8s | 1 | 3.3M |
| 4 · Haiku | 82,996 | 209.6s | 2 | 4.1M |
The two n/a cells are the harness grading its own homework: the cheapest run kept the thinnest paper trail, which is one more thing the next round fixes.
Two gaps keep this from being a real answer.
First, all four runs happened in one continuing session, so cache-read tokens climbed with every run regardless of model: 2.0 million, then 3.2, then 3.3, then 4.1. Model-controlled tokens sidesteps that specific inflation since it ignores cache-read, but a subtler cost rides in through the write side. Cached context expires after five idle minutes, and the first call after an expiry re-writes the whole conversation at the premium rate, billed to whichever run it lands in. A single expiry on a session this size drops six figures into the metric through pure scheduling luck. Opus's 273,147 first pass, which started cold off the harness build and logged six tool errors, is exactly where that fingerprint would land, and the harness as built cannot yet separate a cold cache from a genuinely wasteful run.
The session leaked into the environment too. Suno's workspace filled up with prior tracks as the sitting went on, so later runs were clicking through a busier page than earlier ones. That is an environment difference stacked on top of the accounting one, and it landed hardest on Haiku, which ran last.
Second, and more basic: n of 1 per model, n of 2 for Opus, is an anecdote with a spreadsheet attached. The Opus swing proves that directly. A single run could have caught any of the three models on an unusually clean or unusually messy pass, and there is no way to tell which from inside one data point.
Someone could reasonably ask why publish this at all if it can't confirm anything. The answer is that the harness and the metric are the actual finding here, and they are worth having in hand before the sample size gets fixed. A method that can't survive a model disagreeing with itself by 5x has no business declaring a winner regardless of how many times you run it.
Sonnet has the early lead on speed and consistency, and Opus has the early lead on token cost when it is having a good run. Neither claim survives on one data point, and I am making neither one yet.
The fix is specific and small: run each model three times, each run in its own fresh session with an empty Suno workspace, under this same locked protocol, with the harness logging the gap between calls and the size of every cache write so a cold cache can no longer impersonate a wasteful model. That removes all three confounds at once and turns single points into a median worth trusting. That is the next post in this series. If Sonnet's lead survives three clean sessions, I will say so plainly. If it does not, I will say that too.
The Artificial Advantage: the frameworks behind everything here, written for professionals, not programmers.