Partner POV | Where AI Inference Needs to Run

This article was written by Paul Dehnert, Vice President, Interconnection & Emerging Solutions at Equinix. 

Every enterprise AI strategy eventually runs into the same question: not just which models to use, but where to run them. That question is getting harder to answer as inference scales across more models, more providers and more geographies. According to Gartner®:

"In 2026, global spending on inference ($23.3 billion) will surpass that of training ($19 billion)."[1]

Yet total AI costs keep climbing even as the models themselves grow cheaper, because the model invoice is only one part of the bill. Token consumption is rising fast, and that's driving up overhead costs. Gartner predicts that "AI inference costs per agentic workflow will increase by more than 5X through 2028."[2]

Here's the paradox: Enterprises are spending more on inference every month, and every dollar buys them less control over it. The money is going into complexity—more agents, more providers, more tokens—rather than into optionality. And that complexity introduces key risks to performance, security and compliance. So, the infrastructure gets bigger and more expensive while the enterprise's ability to change course gets smaller.

No single model or provider has to define how enterprises run inference. What matters more is having an architecture that's flexible enough to evolve as business needs change, and that comes down to one question: Where does inference actually need to run?

How can enterprises optimize inference costs?

Enterprises are moving from simple chatbots into agentic workflows. A chatbot reads a prompt, interprets it and responds. An agent has to understand a goal, decide on an action, call tools, coordinate with other agents, and evaluate whether it made progress, then repeat the loop. That difference in complexity is a big part of why costs keep climbing even as the price per token keeps decreasing.

At its core, this is a consumption problem. Agents take many actions because that's how they're built to work, and consumption grows faster than any efficiency gain from a cheaper model can offset. Semantic caching and tighter controls on agent loops can help. But the more durable lever is the cost of each action, and that's set by two things the enterprise controls: which model handles each hop, and where that model runs.

Start with the model. Most hops in an agent chain don't need the most-recent frontier model. Parsing a request, classifying intent or extracting a field runs well on a small, inexpensive open-weight model, with escalation to a frontier model only when the task warrants it. Closed frontier models leave enterprises no such choice: The weights live with the vendor, so every hop is priced at the vendor's rate and runs where the vendor runs. Open-weight models return both decisions to the enterprise, which is what makes tiering possible in the first place.

Then comes the provider. Most enterprises won't run those open-weight models themselves. They'll buy tokens from a new class of specialized providers, neoclouds and Token as a Service platforms that serve open models on their own hardware, each optimizing for a different point on the price, performance and customization curve. That market is fragmented, and the fragmentation is by design. It gives enterprises a real choice for every hop in the chain, and it makes provider selection, not just model selection, a cost lever.

Cost per token isn't a property of a model. It's a property of a configuration: the hardware generation, how fully it's utilized, how requests are batched, and how far away the work runs from the data it needs. Two enterprises calling the same open model through two different providers can pay very different prices per useful outcome. The difference is set by the provider's configuration and by the distance between that provider and the enterprise's data.

That's why cost optimization comes back to placement. The providers serving open models are concentrating in the same metros where enterprise data and applications already live. An enterprise that can reach several of them privately can tier across them, switch as prices and capabilities move, and keep its most expensive hop from setting its economics. One that can reach only a single provider has no lever at all.

How can enterprises optimize performance and latency?

Placement matters just as much for performance as it does for cost. Latency comes in wherever there's distance between data sources and processing hardware. The problem multiplies with agentic workflows: A single task can hop across several providers and locations before it's finished, and every hop adds latency the same way it adds cost.

Cloud-native environments compound the problem in a way that's easy to overlook. In cloud environments, enterprises often control the logical region where a workload runs without controlling its exact physical placement or the network path between the workload, its data and other AI services.

For latency-sensitive inference, the better architecture is increasingly metro-local rather than continent-spanning. The metro edge is close enough to data sources to keep latency manageable, without requiring enterprises to manage that infrastructure themselves. The goal isn't making a cross-region agent chain faster; it's collapsing it so that most hops stay within the metro (or even better, the campus).

What does it take to keep AI inference secure and compliant?

Enterprises also need to protect data moving to and from their inference workloads. The public internet doesn't follow direct routes, which creates the potential for data breaches and compliance risk. Private, secure connectivity solves that: It enables direct, predictable routing between an enterprise's own tech stack and their AI provider ecosystem, regardless of which AI provider they're connecting to or where they're doing it.

That same private connectivity is what makes sovereignty, regulatory and compliance options possible. With direct, private links into the AI provider ecosystem, enterprises can apply their own compliance controls and adjust them as regulations change, instead of depending on one provider. Sovereignty in practice means being able to demonstrate where inference occurred, where the associated data was processed, and whether those workloads and data flows remained within the required jurisdiction. Private connectivity enables this, because it provides observability that the public internet does not.

Re-architecting infrastructure is often complex and time-consuming, so once enterprises land on something they consider safe and compliant, they tend to leave it in place to avoid introducing new risk. That's a reasonable response to a hard problem, but it also means the architecture rarely gets revisited as regulations shift, even when doing so could improve performance and cost-efficiency at the same time. The right connectivity solution ensures that enterprises don't have to sacrifice infrastructure flexibility in order to limit risk.

Built for choice and flexibility

To help our customers optimize AI inference, we recently announced Equinix® Inference Exchange, a new program to deploy compute from leading inference providers and neoclouds at distributed metros. It combines Equinix AI infrastructure and services with private connectivity to the clouds, networks and inference providers already on the platform, so inference can run where the data is.

The partnership with NVIDIA and Together AI combines three complementary layers designed to simplify distributed AI inference:

Equinix provides the infrastructure foundation, including power, advanced cooling and day-two operations, connected through Equinix Fabric® to the clouds, networks and AI providers that inference depends on.

NVIDIA anchors the build with their Enterprise Reference Architectures and AI infrastructure purpose-built to maximize AI factory throughput and minimize token cost.

Together AI runs the platform on top, with 200+ open-source models that can be deployed inside a sovereign environment. It supports both multi-tenant deployments for shared efficiency and single-tenant environments for workloads that require dedicated capacity.

The solution offers the following benefits:

-Distributed infrastructure: Our data centers are available in strategic metros globally, so customers can scale inference workloads where users and data live.

-Seamless integration: The Equinix ecosystem includes a growing set of inference providers, and customers can use Equinix Fabric Marketplace to find and connect to those providers quickly.

-Resilient performance: Proven architectures like NVIDIA's enable better uptime and higher GPU utilization.

-Unmatched connectivity: The solution is pre-connected to the clouds, networks, data sources and AI providers those workloads depend on. It's already part of Equinix® Connected Cloud, our neutral infrastructure foundation for distributed AI.

-Security and compliance: Equinix offers secure, private connectivity between your enterprise tech stack and AI ecosystem. This includes offerings designed for sovereignty and compliance.