A Practical Guide to Getting AI Systems Into Production

Most AI projects never make it to production. Not some. Most.

RAND spent months interviewing 65 experienced data scientists and engineers about why AI projects fail. The number they cite: more than 80 percent fail, twice the rate of conventional IT projects. That's not a vendor talking point. It's a research firm citing an estimate from the field, backed by a study that dug into the mechanics of failure instead of just counting outcomes.

Generative AI looks worse still. MIT's Project NANDA reviewed more than 300 enterprise AI initiatives and found 95 percent of generative AI pilots show no measurable return. Only 5 percent extract real value. The report is preliminary and hasn't been peer-reviewed, but the pattern shows up everywhere else too. Gartner surveyed 782 infrastructure and operations leaders and found only 28 percent of AI use cases fully meet ROI expectations. Three studies. Three methodologies. Same conclusion.

Here's what should concern you more than the headline number: it's rarely the model that's broken. RAND's research points to leadership and problem framing as the top causes of failure, not technical limits. Teams build something impressive that solves a problem nobody had. The model works fine. The organization around it doesn't.

That's the point of this piece. AI is software. It runs by a few new rules, and most teams are still applying the old ones. Take deployment as an example: thirty years of software practice assumes that once you validate a new version, it behaves like the old one. That assumption holds for a login form. It stops holding the moment the same input can produce a different output depending on the day. The same is true of testing, of monitoring, of how you think about cost. The methods are new. The instinct to reach for old assumptions is exactly what gets teams into trouble, and it shows up in almost every section below.

Here's what that actually looks like in practice:

• Regulation as a day-one architecture decision, not a Phase 6 checkbox
• Deployment patterns built for models where "correct" is a distribution, not a fixed answer
• Choosing classic versus agentic retrieval on purpose, not by default
• What MCP actually is, and why it doesn't replace RAG
• Identity and access design for systems where an agent, not a human, is calling the shots
• Cost and multi-model strategy that assumes you'll never run just one model

The lifecycle step everyone skips

Every major cloud provider has published an AI-specific lens on their Well-Architected frameworks: operational excellence, security, reliability, performance, cost, sustainability. Three competitors arrived at the same six pillars independently. That convergence carries weight. When AWS, Google Cloud and Azure all land in the same place, they're not marketing. They're reporting back from millions of production workloads.

The pillar teams skip is the first one: defining the business goal. Teams jump straight to model selection before anyone writes down what success looks like. Three months later, the team is debating vector databases, and nobody has mentioned the original metric in weeks.

That's one day-one decision teams treat as an afterthought. Here's a newer one, and it's less optional: regulation is now a decision you make at the start, not a checkbox at the end. The details matter here. In May 2026, EU lawmakers agreed to a Digital Omnibus that pushed the AI Act's high-risk obligations, human-in-the-loop checkpoints and formal risk classification, from August 2026 to December 2027. That regulation took effect on July 27, 2026. But August 2, 2026 still matters: disclosure requirements for chatbots land on schedule, along with new enforcement powers over general-purpose AI providers. AI-generated content marking splits by system age: new systems must comply from August 2, but systems already on the market get a grace period until December 2, 2026. Fines for the GPAI enforcement category top out at 15 million euros or 3 percent of global revenue. The bigger number people cite, 35 million euros or 7 percent, applies to prohibited practices, not these transparency rules.

If your systems touch European users, build data lineage in now. It's an architecture decision, not paperwork you add later.

Ship it safely, even when it isn't deterministic

We touched on this at the top: deployment is where the old assumptions break first. Same input, different output, sometimes. Correct isn't a fixed value anymore. It's a distribution.

The pattern that solves this, and the one most teams underuse, is the shadow deployment. Mirror real production traffic to your new model. The new model processes every request as it would in production. Its responses are logged and never returned to a user. You get real production data with zero customer risk. Azure's version is a clean template: deploy the new model with 0% traffic, invoke it directly to test, mirror live traffic, then shift real traffic once you trust what you see. The tool names differ across clouds. The pattern doesn't.

Shadow, then canary, then full cutover. It's a sequence, not a menu. Skipping the shadow stage is where most production incidents start, and a two-week observation window would have caught them.

All three major clouds now support directly grading production traces, without building golden datasets by hand. AWS's AgentCore Evaluations reached general availability at the end of March 2026, with equivalent capability in Microsoft Foundry and Google's Agent Platform. Your shadow deployment traces automatically become your evaluation set.

RAG got less artisanal

RAG solves three problems at once: foundation models hallucinate, they're frozen at their training cutoff and they don't know your proprietary data. The classic pipeline runs four stages: collect documents, chunk and embed them, store the embeddings and retrieve relevant chunks at query time. Every cloud implements this shape with different services underneath.

What changed: You likely shouldn't build all four stages by hand anymore. Amazon Bedrock's Managed Knowledge Base and Microsoft's Foundry IQ, expanded significantly at Build 2026, handle ingestion, chunking, retrieval and agentic retrieval out of the box. Start there on new projects. On existing systems, check whether the managed option now beats your maintenance burden.

Classic retrieval is also giving way to agentic retrieval for complex questions. Classic RAG runs one query, one search, one answer. Fast and cheap, and completely sufficient for simple lookups. Agentic retrieval breaks a complex question into sub-queries, runs them in parallel across sources and returns a structured answer with citations. Choose based on the complexity of the question, not on which one demos better.

Doesn't MCP replace RAG?

This question comes up often enough to answer directly. The two aren't competing for the same job. RAG is a technique. It answers one question: how does a model get grounded in information it wasn't trained on?

MCP is a protocol. Anthropic released it in November 2024 to answer a different question: how does an AI application talk to any external tool or data source the same way, every time? Before MCP, connecting an agent to Slack, a database and an internal API meant three custom integrations. MCP standardizes the connection so you build one server per resource, and any compliant client can call it.

MCP doesn't replace RAG. It's often how you expose a RAG pipeline to an agent. Your vector search doesn't disappear. It gets wrapped as an MCP server instead of being hardcoded into your application. In fact, both Bedrock's Managed Knowledge Base and Foundry IQ are increasingly exposed to agents as MCP servers. That's the real relationship: convergence, not replacement.

The protocol that made agents portable

MCP is the biggest structural shift in this space in the past 18 months.

Anthropic open-sourced MCP to solve the "M×N problem": 10 AI tools, 100 data sources, potentially 1,000 custom integrations. MCP flips that into build the client once, build the server once, and everything works together. OpenAI, Google, Microsoft, IBM and Amazon have all shipped native support since. In December 2025, Anthropic donated MCP to the Agentic AI Foundation under the Linux Foundation, making it a founding project rather than something one company controls. At the time, the project reported more than 97 million monthly SDK downloads and 10,000 active servers, with first-class support across every major AI platform.

This means you can build one MCP server for an internal system, and it works unmodified across every major agent runtime. Tools became portable in a way they weren't a year ago.

The orchestration patterns that have converged across every platform are the same six: sequential chaining, routing to specialists, parallel execution, the reason-act-observe loop known as ReAct, hierarchical delegation for open-ended tasks and evaluator loops where one agent checks another's work. None of these belongs to a single vendor anymore.

The runtimes themselves are production-grade. AWS's Bedrock AgentCore reached general availability in October 2025, framework-agnostic, with policies enforced outside the agent's code. Google's Gemini Enterprise Agent Platform absorbed Vertex AI in April 2026. Microsoft's Agent Framework hit 1.0 that same month, consolidating two separate frameworks into one supported runtime.

The models are portable too. Claude reached general availability on Microsoft Foundry in June 2026, running on Azure with full governance. It's also available through Google's Model Garden and natively on Bedrock. That's the real takeaway for this year: patterns are portable, tools are portable through MCP, and models are increasingly portable too. What differentiates a cloud platform isn't agent capability anymore. It's governance, identity and fit with the rest of your stack.

What actually decides your platform choice

Once tools and models stop being the differentiator, identity becomes the whole conversation. An agent isn't a traditional application with fixed endpoints. It acts on its own reasoning, which means standard access controls don't automatically apply. Least-privilege design stops being optional.

Two patterns have emerged, each solving a different problem. Microsoft's identity passthrough has the agent inherit the calling user's own identity token, so the agent can't see data that the user couldn't see. Google's approach targets autonomous agents with no human in the loop, using certificate-based identity tied to the runtime environment. Most organizations need both, depending on the workflow.

Observability is standardizing, too, though it's worth being precise about how far along it is. The gen_ai.* attribute set for LLM calls, agent steps, and tool executions is already the de facto standard in practice, supported by Datadog, Honeycomb, Grafana and major frameworks. What hasn't happened yet is a formal graduation. As of mid-2026, none of it is marked stable in the OpenTelemetry registry. Adopt it anyway. Just know it will keep evolving.

Cost has shifted from a compute problem to an inference problem, and running multiple models is now standard practice. Databricks' State of AI Agents 2026 report, based on telemetry from more than 20,000 organizations, found that 78 percent of companies run two or more model families in production. Nobody bets the whole system on one model anymore.

Routing between cheap and capable models automatically isn't an optimization at this point. It's table stakes.

The checklist

None of this requires inventing anything new. It requires applying what competing cloud providers, open standards bodies and regulators have already converged on. Before your next release, ask:

Does monitoring feed back into your data pipeline, or does it end in a dashboard nobody opens? Do you have a shadow stage before any model change reaches a real user? Did you choose classic or agentic retrieval on purpose? If you're running agents, can you draw the pattern on a whiteboard in 30 seconds? Are your tools exposed through MCP, so they aren't locked to one vendor? Can your agent structurally see data its user couldn't, and if so, is that the first thing you fix? Are you emitting gen_ai.* telemetry? If you touch European users, do you know what's due this August versus what has runway until December 2027? Do you know your real cost per inference with a routing strategy designed to run more than one model?

Answer those honestly, and you're ahead of most of the field. The gap between the 80 percent and the 20 percent isn't talent. It's whether anyone did the unglamorous engineering work in between.