AMD LLM Router Blueprint
Solution overview
In this lab you will have access to a blueprint deployment of an LLM Router and learn about the full AMD stack it runs on. The stack starts with the AMD Instinct accelerated compute platform and includes ROCm and the AMD Enterprise AI Suite. The various layers of the architecture will be explained allowing you to investigate the benefits that each component brings to power the full solution.
LLM Routing
Choosing the right large language model for a task is harder than it looks. The ideal model would be accurate, fast and inexpensive all at once. In practice, teams are forced to settle for two of the three facets. Some organizations resolve this by standardizing on one large model but then end up paying premium GPU time to answer "hello". Or they standardize on one small model but must then accept weaker results on the work that matters more.
The LLM Router removes the choice. It sits in front of a portfolio of model backends as a single OpenAI-compatible endpoint and makes a routing decision on every request:
- Applying a routing policy: The router loads a named policy that describes the categories a prompt can fall into. (The blueprint ships two: task_router, which sorts prompts by the kind of work being asked for, and complexity_router, which sorts them by how much reasoning effort they demand.)
- Classifying the prompt: The classifier assigns the incoming prompt to a category. The recommended embedding approach compares the prompt against written descriptions of each class and picks the closest match by cosine similarity — fast, deterministic, and requiring no LLM inference to make the decision.
- Routing to the optimal model: Each category maps to a named backend. The original prompt is proxied unchanged to the model best suited to it, and the response returns in standard OpenAI shape. The calling application never has to know routing happened.
A concrete example under the task classification policy: "Create a quicksort in Go" is classified as code generation and escalated to the large secondary model.
Meanwhile a request to condense a three-paragraph article is classified as summarization and handled by the small primary model. A bare "Hello" falls through to the primary model as well. Three requests, three appropriate amounts of GPU spent.
Because the router speaks the OpenAI API, it is a drop-in replacement for an existing endpoint. Teams get portfolio-based inference economics without rewriting a single application.
Why this matters for enterprise inference Routing turns model selection from an architectural commitment into a configuration file. Adding a new model, retiring an old one, or changing which class of work justifies premium compute becomes a values.yaml edit rather than a development cycle across every consuming application. |
AMD accelerated compute platform
We start at the bottom of the stack, with the hardware. This lab runs in the ATC on GPU server built on dual AMD EPYC 7763 series processors and populated with four AMD Instinct MI250 accelerators. Each MI250 carries 128 GB of HBM2e memory with 3.2 TB/s of bandwidth, giving the node 512 GB of aggregate GPU memory connected by AMD Infinity Fabric peer-to-peer links.
That memory capacity is what makes this use case interesting rather than merely possible. Multi-model routing is only useful if you can actually hold several models resident at once; a router in front of a single model is just a proxy. On this node the solution simultaneously serves a small primary 8-billion-parameter model, a large 70-billion-parameter secondary model, and a dedicated embedding service for classification. Each workload is sized to the memory it genuinely needs instead of competing for one oversubscribed accelerator.
The hardware conversation for AI is usually framed around peak training throughput. Inference at steady state asks a different question: how much high-bandwidth memory can you put behind a serving fleet, and how efficiently can you keep it busy? This lab is a chance to look closely at that second question on AMD silicon.
ROCm and the AMD Enterprise AI Suite
Above the hardware sits ROCm, AMD's open software platform for GPU compute. ROCm supplies the driver stack, the HIP programming model, and the communication and math libraries that the inference runtimes are built on. In Kubernetes, the AMD GPU Operator handles driver lifecycle and advertises the accelerators to the scheduler as a standard resource, so GPU capacity is requested the same way as CPU or memory.
The AMD Enterprise AI Suite is the reference software stack layered on top. It is fully open source and deliberately modular. You can adopt the components you need and leave the rest. The layers this lab exercises are:
- AMD Inference Microservices (AIMs): Serving a model well means getting dozens of runtime parameters right for a specific model on specific hardware. AIMs are containerized, vLLM-backed model endpoints that abstract that work away, selecting optimal runtime settings automatically from the model, the hardware and the operator's input. In this lab, every model in the routing portfolio is an AIM.
- AMD Resource Manager: Infrastructure administration and multi-team governance — mapping user groups to compute, data and image resources, enforcing quotas and policy, and reporting GPU utilization by project, department and cluster. This is how a shared inference platform stays fairly shared.
- Kaiwo, the Kubernetes AI workload orchestrator: An AI-aware scheduler that reduces GPU idleness through intelligent job queueing, fair resource sharing, guaranteed quotas and opportunistic gang scheduling.
- AMD AI Workbench. Dynamic workspaces, a model catalog, prebuilt fine-tuning pipelines and side-by-side inference comparison — the path from prototype to production for practitioners.
- Cluster Forge. Automated deployment of the control and compute planes onto Kubernetes, turning platform buildout from a multi-week project into a matter of hours.
The solution itself arrives as an AMD Solution Blueprint: an OCI-compliant Helm chart published to a container registry and applied to the cluster with a single templated command. Blueprints are the packaging layer that makes a reference architecture something you can deploy rather than something you read about.
Open source, end to end Every layer in this stack — ROCm, vLLM, the Enterprise AI Suite, Kaiwo, the blueprint charts — is open source. There is no proprietary inference runtime in the path and no license gate between a proof of concept and production. For organizations weighing accelerator choice, that portability is part of the value, not a footnote to it. |
Full stack AI solution
The final layer of this lab focuses on the complete solution that makes the router usable. The Router Controller is a FastAPI service acting as an OpenAI-compatible reverse proxy. The Router Classifier is a companion service that performs categorization. A WWT-custom web interface lets you exercise routing policies interactively, submit prompts and watch the classification and backend selection for each one side by side.
At WWT, we understand the need to build solutions that fit cleanly into our clients' existing landscape. Some parts of an AI application will always be new; the rest of the technical decisions do not have to be. A router that speaks a standard API, ships as a Helm chart, and runs on open source components is exactly the kind of component that can enter an enterprise environment without demanding that the environment change around it