Cutting our agent's harness cost 10x

Like everyone building on LLMs right now, our AI costs grew the way Hemingway said you go bankrupt: gradually, and then suddenly. Most of the bill made sense to us. We make AI video ads, so the video models that generate the footage were always going to be expensive. What we were not prepared for was the harness: the LLM that decides what footage to generate. By the time we priced it out, that alone was roughly 40% of our AI cost of goods, and it ran entirely on Anthropic models, Sonnet and Opus.

The path there was a bet we would make again. When we started the video agent, the Claude Agent SDK was the best harness we could find, Claude's models led the long-running tool work we cared about, and our prototypes were landing. Portability looked like insurance against a problem we did not have, so we chose momentum and built directly on Claude. The product scaled around that choice: eight custom tools became a hundred, and the agent picked up vision, a million-token context window, and subagents. A library choice had become product architecture, and its cost scaled with every ad we made.

The obvious response is to swap in a cheaper model, and that part is easy. The hard part is proof: knowing whether the cheaper model can run the same system without quietly making the product worse. No public benchmark could tell us that, so we had to build the evidence ourselves. That work is most of this article.

Why we couldn't just swap

A simple comparison of token prices could not provide that evidence. A candidate could cost one-tenth as much and still be more expensive if it stalled halfway through a plan, chose the wrong video provider, or produced ads that users asked us to redo. The more dangerous failures were quieter. A model could move exact dialogue into the wrong field, split one multi-speaker scene into separate clips, or leave a storyboard subtly incomplete. The system would keep running and return something. Only the ad would be worse.

So a lower token price made a model a candidate, not a replacement. A replacement had to preserve two different things: harness reliability and creative quality. We could test the first with code. The second required people to look at the finished ads. We arranged the work as three gates, each more expensive than the last, and a candidate advanced only when the previous result justified taking the next risk.

These were the candidates and the public numbers we started from, updated to the rates published on August 17, 2026. Prices are US dollars per million tokens, and the benchmark column is vendor-reported SWE-bench Pro, not our product eval. One rate moved while we worked: when we started, Sonnet's $2/$10 introductory pricing was scheduled to end on August 31 with a 50% per-token increase, and Anthropic made it permanent on August 10.

The public benchmark is useful for orientation, not ranking. The vendors used different scaffolds and inference settings, and a software-engineering benchmark says little about multi-speaker dialogue placement, vision inputs, or a 100-tool creative harness. The near-tie between Sonnet and Luna justified running the expensive test; it did not answer it. Caching and long-context pricing also differ enough per provider that we used the rate card only for screening and rebuilt real costs from our actual token mix.

Reaching the other models was as easy as it should be. The Claude Agent SDK does not expose a general model-provider interface, but it honors ANTHROPIC_BASE_URL, and Anthropic documents the mechanism for LLM gateways: anything that speaks the Anthropic Messages protocol can sit behind it. When a provider offered a usable Anthropic-compatible endpoint, as DeepSeek, MiniMax, and Muse did, we connected directly and kept the path short. For everything else we ran LiteLLM as a self-hosted sidecar: the same v1/messages format in, any provider out, selectable per request, so the main loop and a specialist subagent never had to share a model.

That made our system model-portable, not the SDK model-agnostic. A route can make a model reachable; it cannot tell you whether the model will use your tools correctly, recover from errors, follow your skills, and finish a long workflow. API compatibility was the doorway. The gates came next.

Gate 1: Could it operate the harness?

Small capability probes came first, against both the raw provider API and the Agent SDK path: tool choice, caching, vision, context depth, structured output, streaming. A logging proxy between the harness and the provider earned its keep immediately. When Muse rejected our very first request, the proxy let us find the offending schema among 107 tools instead of guessing: one tool described its Meta API parameters as JSON that could nest to any depth, which Anthropic's endpoint accepts and Muse refuses. We rewrote it to describe the three levels our product actually uses.

The gate itself was our deterministic planning eval. It runs the production agent with the production system prompt, skills, and roughly 100 tools, but blocks the tools that generate or render media, because this stage tests the plan, not the footage. A small LLM plays a cooperative customer who mostly approves and says “continue.” When the run ends, plain code inspects the final state and the tool-call log against 97 concrete yes-or-no checks across seven scenarios; no model judges the score. One scenario asks for a vertical street-interview ad with two speakers in one generated clip, and its checks verify exactly one scene, a multi-speaker-capable provider, the user's exact dialogue kept in structured fields, duration stored in the duration field instead of leaking “8s” into prompt text, and no media generated.

This gate is where our strongest open-weight candidate failed the interview. MiniMax M3 led the open-weight models on vendor-reported SWE-bench Pro and had one of the lowest credible rates on our shortlist. In our hardest scenario it tried roughly 20 times to submit a plan, and every attempt was rejected: first an empty plan, then scalar fields and nested objects turned into arrays. The validation errors named the accepted values, but the model never repaired the shape; eventually it blamed our serializer and gave up. Even with the timeout tripled, it passed 5 of the scenario's 14 checks. The failure was wrong, not slow.

Haiku 4.5 made the result more interesting. It was smaller, cheaper, and weaker on the broad intelligence rankings, and it also produced invalid tool inputs. In one run, though, it recovered from a dozen validation errors and passed 93% of the same scenario's checks. Repeated runs then exposed its own weakness: it sometimes finished the storyboard and ended its turn without storing the scene element. Argument construction, error recovery, and chain completion turned out to be three separate capabilities. No single ranking captured any of them, and our experiment could not say why the models differed; training, size, provider implementation, and error shape all varied together.

The ecosystem suggests these gaps will not stay this large. The Anthropic Messages format has quietly become a distribution channel: DeepSeek, MiniMax, Moonshot, and Z.ai all ship Anthropic-compatible endpoints and document Claude Code integration by name, MiniMax recommends that route as the primary interface to its model, and new model releases now advertise how they behave inside Claude Code the way they once advertised benchmark scores. Whatever explains the differences we measured, other vendors are already treating the Claude Agent SDK as a target to tune for — which is exactly what a de facto standard looks like.

A benchmark is the model announcing “I know kung fu.” The eval is the two-word reply: show me.

The failed calls forced one honest question: had we built schemas that were too complicated? The plan object is deeply nested because it encodes real conditional product state, and Sonnet populated it on the first attempt, so we refused to flatten the stored contract. Instead we changed the interface the model saw: a flatter, ordinary-looking facade that we convert back into the canonical structure and revalidate against the original schema before anything can touch state. The model got a shape it could handle; the product kept its invariants.

Here are the full-suite results we can stand behind. They are single runs from different dates and integration stages, so they are qualification evidence rather than a leaderboard; the defensible claim is that Luna cleared the production gate and was not worse than the incumbent run we had.

Two methodological lessons mattered more than we expected:

• Separate slow from wrong. A 300-second timeout made one model look broken; at 900 seconds it scored nearly perfectly. Another failure persisted at either timeout and was real.

• Repeat the hard scenarios. Muse passed the same case in only 67% of repeated runs. One run could have made it look either perfect or unusable.

Gate 2: Was the ad still good?

The deterministic suite deliberately stops short of creative judgment. It can prove that dialogue survived into the right field; it cannot tell whether the finished street interview feels believable. Candidates that cleared Gate 1 moved to full staging builds, where we watched the complete interaction and inspected the final ad. Then came replays: real customer briefs re-run in fresh internal sessions with the same inputs and a different harness model, which gave us comparable finished ads instead of synthetic traces.

The replays showed the models completing the same assignment with different creative instincts. In a street-interview brief for a meal-logging app, Sonnet treated the format as a collection of perspectives while Luna turned it into a compact one-respondent exchange.

Sonnet: grocery-store entrance, two respondents. “What's the most annoying part about tracking what you eat?”

Luna: city sidewalk, one respondent. “What's the hardest part of eating better?”

Sonnet asked a product-specific question, cut between two respondents inside the opening scene, and used four respondent voices across the full plan; Luna asked a broader wellness question and used two voices in total.

An ashwagandha podcast brief split the same way: Sonnet invented a guest with long recording days and an afternoon crash across six scenes, while Luna compressed the idea into a four-scene editorial argument. Both briefs produced usable ads. The models simply had different ideas about what would make them persuasive.

Sonnet's podcast opening scene

Luna's podcast opening scene

The pattern held across the broader cohort. Sonnet wrote like a screenwriter: its visual prompts were about 22% longer and used camera language nearly twice as often. Luna wrote more like a production spec, emphasizing framing, movement, and restraint. We had to be careful even with that conclusion. Several apparent differences in lighting and lens vocabulary disappeared once we controlled for customer mix, and because replay clips start from different actor images, we treated performance differences cautiously. The spoken scripts converged wherever our skills constrain them with word budgets and a hook-benefit-CTA structure; the personalities emerged exactly where the skills leave creative latitude.

Sonnet's prompts direct the actor; Luna's prompts direct the camera. Same brief, same renderer.

The judging stayed with people. We had tried an open-ended vision model as the judge, but six of the ten defects in its adjudicated sample were inventions, so humans graded every transcript.

Human grading told us whether an ad was defective. It could not tell us, at scale, whether users liked what they got. For that we used a signal we already track: whether the customer downloads or publishes the finished ad. During the rollout we compared those rates between models, and once the early integration fixes landed, we could not tell the models apart.

A benchmark can tell you whether a model completed the assignment. The rendered work tells you what kind of creative collaborator completed it.

Gate 3: Did it survive production?

Even a model that passes evals still has to be lived with. A resume can get someone an interview, but it cannot tell you what they are like to work with, and we found the same distinction in models: recovery, observability, vision, and harness fit decided whether intelligence became reliable work.

DeepSeek made the sharpest case. It completed a full interactive build and held up well on the deterministic suite through the gateway, but the Anthropic-compatible endpoint we tested did not support images, and our agent has to look at products, reference ads, frames, and finished creative. A text-only model can plan an excellent video and still be a wrong fit for us.

Observability cut the same way: Muse returned its reasoning stream as redacted_thinking, so we paid for thinking we could not read, and our OpenAI route sent no readable summaries at all until we opted in at the gateway. And harness features were not used equally well: Muse handled the SDK's deferred tool search poorly, spending its calls searching for tools instead of using them, so we load the complete tool catalog eagerly for models that search poorly.

We stopped treating these as scattered exceptions. Each model received a small compatibility profile declaring what it should see: which tool schemas, whether the full catalog loads eagerly, whether images are sent. That kept provider quirks at the boundary instead of spreading model-name conditionals through the agent.

Not every behavioural difference was a defect. When a new model read one of our skills and did something the incumbent never did, the cause was often an instruction that was ambiguous or contradicted another one, and the incumbent had simply been resolving it the way we happened to intend. A second model was the first real review our prompts ever got. We rewrote the instructions it tripped on, and the rewrites made the harness clearer for every model, including the one we started with.

Live telemetry became the last eval. We scanned production tool calls for filler values, compared the same session before and after each deployment, and checked that each model was drawing the share of traffic we intended. More than once, what first looked like a model weakness turned out to be an integration artifact, which is why the next section exists.

Field notes: what breaks when you put other models behind the Claude Agent SDK

The three gates are the tidy version of this story. The honest version is that much of our lost time went to integration details that looked like model behavior. Here is the list, in three groups, so you can skip the archaeology.

The route quietly changes what you send.

• Images can be dropped in transit. An Anthropic-compatible endpoint can accept a request and silently discard its image content. Probe every modality you use before trusting a route.

• Thinking effort is chosen for you. The Agent SDK requests adaptive thinking, and Claude models decide their own thinking depth natively. A translated model cannot, so the gateway maps the request onto the provider's reasoning-effort parameter, and with no explicit value it falls back to the provider default. Pin the thinking effort you want per model in its compatibility profile.

• Readable thinking is an opt-in, not a default. OpenAI reasoning models never return their raw trace, and they emit a readable summary only when the request asks for one; a translated request does not ask by itself, so the thinking pipeline goes dark until you add that opt-in at the gateway. Other providers encrypt their reasoning outright, and you pay for tokens you cannot read. Either way, budget for debugging with tool order, validation errors, and final state.

• Strict mode fills every optional parameter. OpenAI's Responses API coerces translated tool schemas into strict mode, whose grammar requires every key to be present, so unused parameters arrive as empty strings and null. No prompt can override a server-side grammar. Pin strict: false on translated tools with a pre-call callback.

A tool surface built for one model can defeat another.

• One schema can block every session. Anthropic's endpoint accepts a tool schema with unlimited nesting; another provider can reject the entire request because of that one tool. Describe the nesting depth you actually use.

• Your canonical schema may be too deep. One model may fill a deeply nested schema on the first attempt; another may never manage it. Serve those models a flatter facade and reparse every call against the canonical schema at the boundary, so the product contract never moves.

• Tool search can become the whole session. A model that handles deferred tool search poorly can spend a session searching the catalog and never reach a product tool. Load the full catalog eagerly for such models.

The harness speaks to the model in your user's voice.

The Agent SDK can enforce a structured final answer: you hand it a schema, and the agent must end its turn by calling a StructuredOutput tool that carries the deliverable. If the agent ends its turn in prose instead, the SDK blocks the stop and injects a corrective message so the model continues. That message arrives in the user role, with fixed wording that begins "Stop hook feedback: You MUST call the StructuredOutput tool…".

"Stop hook" is harness jargon, the lifecycle hook named Stop. The model does not know that vocabulary; it knows yours. Our agent makes ads, where a hook is the opening seconds of a video and feedback is what users leave on a draft. One agent finished a build, wrote its final batch as prose, received the injected reminder, read it as the user saying stop, discarded the finished work, and answered "Understood — stopping here."

The wording is not yours to change, so the fix lives in your prompt: name the message, say what it is, and state the recovery — re-send the entire batch as the tool call. It also helps to keep the final contract to a single artifact. The less the last call must carry, the fewer ways there are to end a turn in prose.

The cost numbers are wrong until you rebuild them.

• The SDK prices unknown models at flagship rates. It reads a bundled price table, and any model it does not recognize falls back to a default flagship rate; even a new first-party model is unknown to an older pinned SDK. Rebuild cost from raw token counts and your own price registry.

• Translation can drop usage fields. A gateway can zero out usage fields it does not map, and a missing cache-read count inflates apparent cost by orders of magnitude. Verify usage fields on the wire against the provider's own billing.

• Long-context surcharges differ per provider. OpenAI charges 2x input and 1.5x output above 272K tokens; MiniMax doubles its rates above 512K. Your average rate depends on your context profile.

What the swap actually saved

With the accounting fixed, the numbers were finally trustworthy. Completed-video sessions produced this distribution:

Dots show the median and bars show the middle 50%.

The median model cost of a completed video fell from $4.44 on Sonnet to $0.50 on Luna, a nearly ninefold drop that the full token-mix repricing below confirms at 9.86 times.

Ten times cheaper, and ready for the next model

We rolled Luna out gradually and stopped the ramp at 95%. The remaining 5% of sessions stays on Sonnet as a long-term holdback, and existing sessions kept the model they started on.

Measured from wire-verified token counts and current base rates, the main-loop harness became almost 10 times cheaper on the same observed token mix. The underlying list-price difference was 8.3 to 10 times, depending on token type. We kept 100% as the correctness target rather than defining success around Sonnet's score. The deterministic results, graded production cohort, and visual review gave us enough evidence to stage the rollout, and we did not see a quality regression that justified stopping the migration.

The more durable result was the system around that saving. We now had direct and translated provider routes, explicit compatibility profiles, deterministic harness evals, human visual evaluation, wire-verified token accounting with explicit pricing caveats, and a gradual rollout path. A new model could enter the same process without a new agent implementation.

We kept the Claude Agent SDK because it was still the right harness for the product we had built. We made our product model-portable around its Anthropic-shaped boundary, and the eval system made that portability safe enough to use.

The cheapest model that can answer a prompt is not necessarily the cheapest model that can operate a product. It has to fit the harness, recover inside it, remain debuggable, and preserve what the user sees.