The Tokenomics of AI Native Engineering: Engineering the Cost Model

In Part 1, we mapped the cost structure of agentic AI development and laid out the FinOps framework for determining whether the spend is producing value. This part covers what to do about it. The patterns below are not workarounds; they're an engineering discipline applied to a new cost surface.

The advisor strategy: Anthropic's answer to escalation

The traditional response to this problem is to use cheaper models for everything and accept quality loss. Anthropic's advisor strategy, released in April 2026, is a more precise answer: pair a cheap executor model with an expensive advisor that only gets involved when the executor hits a genuine reasoning wall.

The executor (Haiku 4.5 or Sonnet 5, whichever fits your quality floor) handles all the mechanical turns: reading files, applying edits, running tests, processing tool outputs. When it encounters a decision point that exceeds its capability (an architectural trade-off, a complex debugging path, a planning problem), it escalates. Opus 4.8 reads the full transcript, produces a plan or course correction in 400 - 700 tokens and hands back to the executor. The advisor never generates the full output. The executor does that at its cheaper rate.

The pattern is implemented through the standard Anthropic tool use API; no bespoke agent framework needed. You define the advisor (Opus 4.8) as a named tool in the executor's tool list; the same way you'd define any other callable function. The executor's system prompt instructs it when to invoke that tool: before committing to an architectural approach, when test output signals something unexpected or when the task is ambiguous enough that the path forward isn't clear. When the executor calls the advisor tool, your application code intercepts that call, makes a separate API request to Opus with the current conversation as context and injects the response back as the tool result. The executor continues from there. You specify both model names explicitly. What you don't have to build is a custom orchestration layer; the escalation decision lives in the model, not in routing logic you write.

The advisor generates 400–700 tokens of guidance, not the full output. That's where the cost savings come from.

Benchmark results from Anthropic: Sonnet with an Opus advisor scored 74.8% on SWE-bench Multilingual, up 2.7 percentage points from Sonnet alone, at 11.9% lower cost per task. With Haiku as the executor, the effect is stronger; BrowseComp score more than doubles (19.7% to 41.2%) at 85% less cost than using Sonnet for everything.

The practical prompting rule: instruct the executor to call the advisor before committing to an approach, and again after test outputs are in the transcript before declaring done. Orientation work (finding files, reading context) doesn't trigger escalation. Writing, editing and deciding does.

One limitation worth noting: the advisor sub-inference doesn't stream. There's a pause while the advisor runs. For background agentic tasks this is irrelevant; for interactive sessions it's a UX consideration.

A minimal sketch of the pattern in Python:

tools = [{
    "name": "consult_advisor",
    "description": "Call when facing an architectural decision, ambiguous requirement, or repeated test failure. 
    The advisor will review context and return a plan.",
    "input_schema": {"type": "object", "properties": {"question": {"type": "string"}}}
}]
# Executor call (Haiku or Sonnet)
response = client.messages.create(model="claude-haiku-4-5", tools=tools, messages=conversation)
# Route advisor tool calls to Opus
if response.stop_reason == "tool_use":
    advisor_response = client.messages.create(
        model="claude-opus-4-8",
        messages=conversation + [{"role": "assistant", "content": response.content}]
    )
    # Inject advisor guidance back as tool result and continue

The structure is the same as any other tool call in the Anthropic API. No routing framework. No custom orchestration. Just two model names and the standard tool use loop. A different variant worth considering, particularly for teams with sensitive codebases or high-volume background work: use the frontier model for planning only, then hand execution to a local open-weight model.

A single Opus or GPT-5 call at session start generates a detailed step-by-step execution plan. A locally hosted model (Llama 3.x, Qwen, Mistral) handles every subsequent mechanical turn, including file reads, edits, test execution, and diff checks, at near-zero marginal API cost.

The trade-offs are real: local models have a lower quality floor than hosted mid-tier models and running inference infrastructure well isn't trivial.

But for teams with high task volume, code that can't leave the network on execution turns, or both, the economics can be compelling. The planning quality comes from one frontier call. The execution cost drops to infrastructure-only for everything that follows.

Model routing: the principle every provider has adopted

The advisor strategy is Anthropic's formalization of a broader principle: most turns in an agentic workflow don't require frontier intelligence. Reading a file doesn't need the same model that designs a system. Running a linter doesn't need the model that debugs a distributed race condition.

Every major provider now structures their model family around this principle, and the cost differences between tiers are not incremental; they're an order of magnitude.

The tier structure is consistent across providers. Nano/Lite models cost 10–25× less than frontier. Routing 70–80% of traffic to the cheaper tier captures most of that gap.

OpenAI's GPT-5.4 Nano at $0.20 per million input tokens undercuts nearly every alternative for classification, extraction and routing decisions. GPT-5.4 Mini at $0.75 handles general production tasks at roughly 85–90% of flagship quality for 20% of the cost. Reserve the full GPT-5.4 at $2.50 for the hardest 10–15% of queries. A real-world example from a high-volume SaaS: routing 70% of traffic to Nano, 25% to the mid-tier and 5% to the frontier model cut monthly spend from $27,000 to $15,000 (a $144,000 annual saving from one architectural decision).

Google's model cascade runs Gemini Flash-Lite ($0.25/M) for classification and summaries, Flash ($1.50/M) for core generation and multi-turn agents and Pro ($2.00/M) for complex reasoning and long-context tasks. The Pro tier has a pricing cliff above 200,000 tokens: the input rate doubles, creating an explicit incentive to keep context tight.

AWS Bedrock takes a platform approach. Bedrock Intelligent Prompt Routing provides a serverless router across Anthropic, Meta Llama and Amazon Nova model families. Rather than hard-coding routing rules, it predicts the response quality each model would produce for a given request and routes to the cheapest model that meets a quality threshold. Average cost reduction across the Nova family: 35%, with no code changes to the application.

Cognition, the team behind Devin, applies the routing principle at the platform level by running their own proprietary model alongside frontier models. Tasks that don't require frontier reasoning get routed to Cognition's internal model, which is optimized specifically for the kinds of work Devin handles at volume: file navigation, test execution, diff generation, code search. Routing a large fraction of agent turns to a proprietary model cuts the per-turn API cost substantially, which matters when Devin is running 50-100 turn sessions on non-trivial tasks. It's the same principle as Nano for classification, just with a model built for the specific task domain rather than a general-purpose cheaper tier.

For teams building their own internal agents at scale, this is where the routing conversation eventually leads: at sufficient volume, fine-tuning or building a narrow model for your most common turn types starts to outperform routing to a general-purpose cheap tier.

On-premises and hybrid: the infrastructure option

The routing principle, pick the cheapest model that meets your quality threshold, has a natural extension: what if one of the models in your routing table isn't an API endpoint at all, but a server in your own data centre?

For many enterprise engineering teams this isn't a theoretical question. Regulated industries such as financial services, healthcare and defence contracting operate under data classification rules that may prohibit sending proprietary code or customer data to external API endpoints. For others, the maths at sufficient volume makes on-prem attractive regardless of compliance: once you're spending enough on API tokens, hardware amortisation starts to look competitive on a per-token basis.

The full on-prem case

A fully on-prem deployment eliminates per-token API costs once hardware is amortised. The marginal cost of an inference run is power and cooling, not a per-call charge. You get absolute data sovereignty, meaning code and context never leave your network, no rate limits, no upstream availability dependency and a cost structure that finance can model without phoning the API vendor every quarter.

The costs are equally real. Serving a frontier-class open-weight model (70B+ parameters) requires substantial GPU infrastructure. A well-configured 4-node H100 inference cluster runs well north of $150k in hardware before power, cooling and networking. The quality gap between the best open-weight models and frontier API models is closing on routine coding tasks such as file edits, test execution and diff generation, but it persists on complex multi-file reasoning and architectural decision-making. That gap matters most at exactly the turns where quality determines whether a loop runs for 20 iterations or 200. You also lose access to proprietary efficiency features: prompt caching at 90% discount, the batch API at 50% off and the advisor pattern as implemented by Anthropic all disappear in a fully on-prem deployment. Maintenance overhead is non-trivial: inference runtimes like vLLM and Ollama require active management, model updates are manual and the infrastructure team takes on a new surface area.

Hybrid as the practical middle ground

For most enterprise teams the right answer isn't full on-prem or full cloud API; it's a hybrid architecture that routes based on task type and data classification. Three patterns have emerged as practical:

Model routing with a local fallback. A gateway proxy (LiteLLM is the most widely deployed; Portkey is another option) sits in front of your model calls and routes based on a routing rule or content classification. Mechanical executor turns go to a locally hosted open-weight model. Complex reasoning and planning turns escalate to the API. The executor runs on-prem; the advisor runs in the cloud. LiteLLM supports this natively; you configure both endpoints and a routing policy, and application code makes a single API call regardless of where it resolves.

Private cloud inference. AWS Bedrock and Azure AI both support importing open-weight models to dedicated inference endpoints within your VPC. You get managed infrastructure without the hardware capital burden, and you can route between your hosted model and frontier API models through the same interface. This is the most common path for teams that need data residency controls but don't want to operate bare-metal GPU infrastructure.

VPC-connected API endpoints. Both Anthropic and OpenAI offer enterprise connectivity options, AWS PrivateLink and Azure Private Endpoints, that keep API traffic within your network perimeter. Not fully on-prem but satisfies many compliance and data residency requirements without the quality trade-off of open-weight models. If your concern is data exfiltration rather than sovereign hosting, this is often the right answer.

Matching the pattern to the requirement

The executor tier handles the majority of agent turns and has the highest volume; it's the natural candidate for on-prem or private cloud hosting. The advisor tier handles a small fraction of turns but generates most of the value; it typically stays on frontier API. That split maps directly onto the advisor pattern and onto the open-weight execution variant covered above: one frontier planning call, on-prem or private cloud execution for everything that follows.

This is an area where architecture decisions have significant infrastructure implications. Getting the model routing right is a software decision. Getting the infrastructure right to support it is a different conversation, one that involves hardware selection, network architecture, compliance review and operational model. For teams evaluating this, the build-versus-buy-versus-hybrid question is worth answering before the API bill makes it urgent.

Prompt caching: The most underused lever

Model routing reduces the cost of the turn. Prompt caching reduces the cost of the context within each turn.

Most agentic workflows carry large blocks of static content on every request: the system prompt that defines the agent's behaviour, the tool definitions, the codebase context loaded at session start, the persona and operating rules. These are re-sent and re-processed on every API call, at full input token rate.

Prompt caching stores the computed key-value representations of a repeated prefix so subsequent requests hit the cache instead of reprocessing. Anthropic charges 90% less for cached input tokens. OpenAI discounts repeated prefixes by 75–90% automatically. Google's context caching on the Pro-tier drops cached reads to $0.20 per million, down from $2.00. The write cost is slightly higher than a standard request (1.25–2×), but the economics are compelling for anything read frequently.

For an agent with a 10,000-token system prompt running 1,000 sessions per month, caching that prompt alone can save thousands of dollars per month. For agents with large codebase context (reading hundreds of files at session start), the savings compound further.

The practical approach: structure your system prompt with the static, cacheable content at the top. Tool definitions, persona, operating constraints, loaded context. Dynamic content (the current task, user input) goes at the bottom. The cache key is the prefix; anything below it bypasses the cache.

The batch API and thinking budget

Two other levers that get less attention.

Most AI coding tasks don't need a real-time response. Test generation, documentation, code review, refactoring, migration tasks; none of these blocks a developer waiting for an answer. Your test suite has never once cared whether results arrived in 30 seconds or two hours. Both Anthropic and OpenAI offer a batch API at 50% off standard rates for asynchronous work. Send the request, get the result back within a few hours. For high-volume background tasks, this is the simplest cost reduction available.

Extended thinking, where the model generates internal reasoning tokens before the visible output, is billed at output rates, which are 3–5× input rates. Frontier models can generate substantial thinking token volumes on complex tasks, and that cost can surprise teams running thinking-enabled models at scale. The practical control: set a maximum thinking token budget, and don't enable extended thinking for tasks that don't need deep reasoning. Reading a file and describing it doesn't need the same thinking budget as designing a concurrency model.

The tools your engineers are actually using

Not every AI coding workflow runs through an API your team controls. Claude Code, Cursor, OpenAI Codex CLI, Devin Desktop(Windsurf); these are the tools most developers open before they open a terminal. They have their own cost structures, and the same principles apply, just with less visibility into the numbers.

Cursor runs on per-seat pricing, which feels like it insulates you from token costs. It doesn't. Under the hood, Cursor routes requests between models; fast requests hit a mid-tier model, slow requests escalate to frontier. Cursor Max mode (direct API, billed per token) drops the abstraction entirely and exposes you to the same compounding math described above. Teams running Max mode for whole-session debugging conversations have hit $30-50 per developer per day without noticing, right up until someone reconciles the Anthropic invoice.

The practical control: use regular mode for autocomplete and navigation, reserve Max mode for hard problems and start new sessions rather than letting context accumulate across unrelated tasks. A 200-turn session debugging one problem and then pivoting to a new feature is paying for the entire debugging history on every token of the new work.

Claude Code and Codex CLI bill directly to your API account. No per-seat abstraction. Every /clear command costs nothing and saves real money; it resets context to zero. Engineers who treat a CLI session like a persistent terminal, never clearing, accumulate context that compounds across hours of work. Establishing team norms around session hygiene (start fresh for new tasks, /clear between unrelated problems) is one of the cheapest optimizations available and the one that requires exactly zero engineering effort.

Devin Desktop runs fully autonomous long-horizon sessions, which is precisely the context compounding scenario the curve above describes, with less opportunity to intervene mid-session. Sessions regularly exceed 50 turns for non-trivial tasks. The practical answer: scope tasks tightly before starting. A Devin session handed a fuzzy requirement will explore, read, backtrack and accumulate context doing work you'll redo anyway. Tight scoping before the session starts is cost control that looks like good engineering because it is.

Across all these tools, the highest-ROI habit is one nobody talks about: don't treat AI context like RAM you can leave running. Start fresh. Scope tight. The context window is a resource, and engineers who treat it like one are cheaper to run than engineers who don't.

Loop engineering: Designing the system that prompts for you

By mid-2026, the conversation in engineering circles had shifted from prompt quality to something upstream of it. Boris Cherny, who built Claude Code at Anthropic, said it directly: "I don't prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops." Addy Osmani framed the same point as loop engineering: replacing yourself as the person who prompts the agent and designing the system that does it instead.

The cost dimension is where this gets interesting for engineering organizations. A loop is also an unmetered spend vector if it's not designed properly. The engineers pulling ahead on both productivity and cost efficiency are treating loop design as an engineering discipline with the same rigour they'd apply to any other system.

Goals and stopping criteria

Both Claude Code and Codex ship /goal as a native command. You give it a verifiable condition, such as all tests in test/auth passing and lint being clean, and the loop runs until that condition holds. After every turn, a separate small model checks whether you're done. The model that wrote the code is not the one grading it.

The cost math here is direct. A loop with 20 iterations averaging 5,000 input and 1,000 output tokens per turn runs about $3.30. A loop with 500 iterations, because the stopping criteria were vague, runs $80+. The most expensive loops are ones where done wasn't defined before the loop started. This is the /goal principle applied to cost: define what done looks like, make it verifiable and let the condition do the stopping rather than a turn limit or a human watching the terminal.

/loop is the cadence variant: a timer that triggers the agent on a schedule to do discovery or triage work. The difference matters for how you control spend. Goal-based loops are bounded by a verifiable condition; cadence loops are bounded by time and scope. An automation that runs every morning to summarise CI failures is a cadence loop. A task to implement and ship a feature is a goal loop. Confusing the two is how teams end up with open-ended agentic sessions that nobody stopped.

Skills: context that doesn't re-cost you every session

Every cold session burn orientation tokens. The agent reads files, discovers conventions, re-derives project context that the previous session already established. A Skill (SKILL.md) is that context written down once, outside the conversation, where the agent reads it at the start of every run. Claude Code and Codex use the same format. An agent that starts informed rather than cold skips the exploration turns that compound earliest in the cost curve, which from the context accumulation diagram above, are the turns that set the baseline for everything that follows.

Skills also make loops maintainable. Rather than embedding project context in every automation prompt and updating it manually whenever something changes, the skill holds it once. When the codebase evolves, you update one file.

Sub-agents: the maker/checker split

Splitting the agent that does the work from the agent that checks it is both a quality practice and a cost practice. An agent grading its own output tends to declare done prematurely. A separate verifier (often a cheaper, narrower model) catches failures earlier and stops the loop from burning turns on output that doesn't pass.

Both Claude Code and Codex support configurable sub-agents. The typical split is one agent to explore, one to implement, one to verify against spec. The verifier can run on a smaller model than the implementer because its job is narrow: does this output meet the condition? A sub-agent architecture also mirrors what /goal does at the loop level, separating evaluation from execution.

Worktrees for parallel loops

Running multiple loops concurrently without isolation leads to file conflicts, the same problem as two engineers committing to the same lines without coordinating. Git worktrees give each agent its own checkout on its own branch, sharing history but not working state. Claude Code exposes this with --worktree and isolation: worktree on sub-agents; Codex builds it into each thread by default. In practice this means you can run three feature loops in parallel without any of them breaking each other's state, which changes the economics of background agents from sequential to concurrent.

The limit loop engineering doesn't solve

A loop running unattended is also a loop making mistakes unattended. Osmani's piece named it plainly: "If I weren't reviewing the code myself or relied entirely on automated loops to fix it, my product's quality would suffer." The loop doesn't distinguish between an engineer who uses it to move faster on work they understand deeply and one who uses it to avoid understanding the work. That distinction stays with the engineer. Loop design is the leverage point. Judgement is still the input.

What this means for engineering organizations

None of these patterns are workarounds. They're the engineering discipline that makes AI native development financially sustainable at scale.

The analogy to cloud infrastructure is direct. In 2012, plenty of engineering teams provisioned on-demand compute for everything; it was simpler. Over time, mature teams developed right-sizing practices, spot instance strategies, reserved capacity models. The teams that treated cloud cost as an engineering problem rather than a finance problem built infrastructure that could scale without the bill spiralling out of control.

AI token spend is the same problem. The teams pulling ahead right now are the ones treating model selection, context window management, caching strategy and batch execution as first-class engineering decisions, not an afterthought that finance handles. These are architecture decisions that happen in code, not in a spreadsheet.

AINE still must make sense in a budget conversation. The productivity gains are real enough to justify the investment. The job of the engineering team is to make sure the cost model is as well-engineered as the software it's producing.