Inside LLMaaS: the partner stack behind model-as-a-service in AI Proving Ground

WWT's large language model as a service (LLMaaS) gives builders in the AI Proving Ground one governed, OpenAI-compatible endpoint to more than 40 models. Some run on GPUs on our own floor, and some are frontier models from leading AI providers. This article goes layer by layer through the stack, from the GPU servers and storage to the gateway and the model catalog. For each layer it covers the partner technology we chose, what that technology does inside LLMaaS, and why it was the right fit.

Why we built LLMaaS

Every AI project in the AI Proving Ground (AIPG) starts the same way: which model, where does it run and how do I get access? Before LLMaaS, the answer was usually a dedicated GPU, a one-off deployment and an API key handed out by hand. That works for a single proof of concept. It breaks down when you're running dozens of labs, customer proofs of concept (POCs) and internal tools at the same time.

LLMaaS turns that into a shared service:

• One endpoint, many models. Builders send standard OpenAI-style requests, such as POST /v1/chat/completions with "model": "...". The platform routes each request to the right engine, whether it runs on our own GPUs or with a frontier provider.
• Governed access. Every consumer gets an API key, entitlements for each model and token-based rate limits in three tiers (basic, plus and pro).
• Models as code. Adding, changing or retiring a model is a pull request. Automated tests check it end to end before anyone relies on it.

Today the catalog holds 43 models: 22 self-hosted on our own GPUs and 21 served through frontier-model APIs. They serve 162 active consumers across 18 labs and POCs, which together use 4.2B tokens in a recent month (4.1B prompt, 120M completion) across 2.4M requests.

Rate limiting stepped in 220K times in that same month. When one consumer's traffic would slow the platform down for everyone else, the gateway holds that consumer to its tier and every other consumer keeps the performance they expect. That's governance doing its job.

The stack at a glance

Infrastructure: NVIDIA, Dell, HPE and NetApp

GPU compute: NVIDIA DGX, Dell and HPE

LLMaaS runs on three NVIDIA data-center GPU models across two architectures (H100 and H200 on Hopper, B200 on Blackwell). The servers are a mix of NVIDIA DGX systems and Dell and HPE GPU servers. That mix is deliberate. Most enterprises we work with don't run a single-vendor GPU fleet, and LLMaaS shows that one platform can schedule across all of them.

Workloads are placed by GPU class, not by server. Each model is pinned to the GPU it needs: compact models on H100, large mixture-of-experts models on H200 and new releases on B200. The platform labels every node with its exact GPU product, so the newest models land on the newest hardware wherever it sits in the rack.

On the NVIDIA DGX systems, NVIDIA System Management (NVSM) feeds hardware health into the cluster.

Storage: NetApp ONTAP, Trident and StorageGRID

Model weights are big. The largest NVIDIA Nemotron model alone is hundreds of gigabytes, and every inference pod needs them on local or shared storage. NetApp ONTAP, provisioned dynamically through NetApp Trident (NetApp's Container Storage Interface driver), backs the model caches, the shared weight volumes and the stateful platform services.

• Shared model caches. The weights are downloaded once to a Trident-backed volume and mounted by every replica. Scaling a model out doesn't mean downloading it again.
• NetApp StorageGRID provides S3-compatible object storage for nightly backups of gateway state and configuration, kept for 30 days.
• NetApp Data Infrastructure Insights monitors the storage layer alongside the cluster.

Why NetApp: the storage classes are ordinary Kubernetes storage, so a new model needs no storage ticket. A pull request asks for a volume and Trident creates it.

Platform: Red Hat OpenShift

The container platform

Every LLMaaS component runs on Red Hat OpenShift: the inference engines, the gateway data planes, the management APIs and the observability stack. Every node, GPU servers included, runs Red Hat Enterprise Linux CoreOS, the immutable operating system built into OpenShift. Even the GPU driver is delivered as a container, so the hosts need no hand-built driver stack.

GitOps with Red Hat OpenShift GitOps

The whole cluster is declared in Git and reconciled by Red Hat OpenShift GitOps, Red Hat's supported distribution of Argo CD. It uses an app-of-apps pattern, so each component (the NVIDIA NIM and vLLM engines, the NVIDIA Dynamo platform, the Kong data planes, observability and storage) is its own application that syncs from the repo. Drift is corrected automatically, and every change to the cluster can be traced to a commit.

Bare-metal lifecycle with Red Hat Advanced Cluster Management

New GPU servers are added through Red Hat Advanced Cluster Management and the Assisted Installer flow. A server is described in YAML, boots a discovery image, gets approved and joins the cluster as a worker. Adding capacity is a repeatable procedure rather than a project.

Self-healing and observability

• Node Health Check and Self Node Remediation detect a failed GPU node, such as a server that powers off after a thermal trip, and fence it so its workloads reschedule on healthy nodes.
• The Red Hat build of OpenTelemetry, Tempo (distributed tracing) and Loki (logging) operators, together with OpenShift's built-in Prometheus and Thanos monitoring, give us gateway traces, inference metrics, per-GPU NVIDIA Data Center GPU Manager (DCGM) telemetry and request logs in one place. Grafana dashboards are built on top of them, covering everything from an executive summary of gateway traffic to deep dives into individual models.

Why OpenShift: it gives us an enterprise Kubernetes platform where the operators we depend on, from NVIDIA, NetApp and Red Hat itself, are certified and supported. That matters when a customer wants to take the same blueprint home.

GPU enablement and scheduling: NVIDIA GPU Operator and NVIDIA Run:ai

NVIDIA GPU Operator

The NVIDIA GPU Operator manages the full GPU software stack on OpenShift: the driver, container toolkit, device plugin, GPU Feature Discovery, the Multi-Instance GPU (MIG) manager and DCGM telemetry. Driver upgrades roll through the cluster automatically. Time-slicing presents four virtual replicas per physical GPU, so small models and dev workloads don't each claim a whole H100.

NVIDIA Run:ai

NVIDIA Run:ai is the scheduling and quota layer. It lets us:

• Allocate fractional GPUs. For example, two small models can share one H100 at 0.5 GPU each.
• Bin-pack GPU workloads. The Run:ai scheduler places new GPU workloads on nodes and GPUs that are already partly used before it touches empty ones. Small models and fractional workloads fill the gaps, and whole GPUs and whole servers stay free for the large deployments that need them, such as an 8-GPU tensor-parallel model that won't schedule on a fragmented node.

Why it matters: GPUs are the most expensive resource in the building. Fractional allocation and bin-packing let us keep dozens of models online on a fixed fleet without dedicating a GPU to every one, and still have room for the next large model.

Inference: NVIDIA NIM, NVIDIA Dynamo and vLLM

LLMaaS runs three serving engines side by side and picks the right one for each model. Consumers never see the difference. They see one API.

NVIDIA NIM: optimized, supported, ready to run

NVIDIA NIM microservices are prebuilt, performance-tuned inference containers for NVIDIA and partner models. The NVIDIA NIM Operator manages them as Kubernetes resources and handles model caching and lifecycle for us.

NVIDIA NIM serves the largest part of our self-hosted catalog, 12 models in all:

• The NVIDIA Nemotron family, including the Nemotron Super models and the multimodal Nemotron Nano VL and Nemotron Nano Omni (text, image, audio and video in one model)
• The Meta Llama instruct models and a Mistral model
• The NVIDIA NeMo™ Retriever embedding and reranking models, including vision-language variants, which give every retrieval-augmented generation (RAG) builder in the AIPG a production-grade retrieval pipeline from day one

Why NVIDIA NIM: a supported container that is ready on day one and tuned for the GPU underneath it. When NVIDIA releases a model as an NVIDIA NIM microservice, adding it to LLMaaS is mostly a configuration change.

NVIDIA Dynamo: serving a 550B-parameter model with a 1M-token context

Some models are too large, or have context windows too long, for a single-node deployment. For those we use NVIDIA Dynamo, NVIDIA's distributed inference platform. It is managed by the NVIDIA Dynamo Kubernetes operator and adds:

• Disaggregated prefill and decode, so the compute-bound and memory-bound phases of inference can scale independently
• Key-value (KV) cache-aware routing, which sends each request to the worker that already holds the relevant cache
• Multi-node serving profiles that can be swapped declaratively, for example a lower-latency 256K-context profile or a 1M-context profile

NVIDIA Dynamo serves an NVIDIA Nemotron Ultra model (550B parameters, 55B active, in NVIDIA's 4-bit NVFP4 format) with a 1-million-token context window across all 16 H200 GPUs in two NVIDIA DGX H200 systems. In our testing, that deployment holds sub-second time-to-first-token at normal interactive load, scales past 10,000 tokens/s on agentic workloads, and completed every measured load level with zero failed requests.

Why NVIDIA Dynamo: it's the one engine built for models too large for a single node: disaggregation and KV-aware routing keep a 550B-parameter model responsive instead of just resident.

vLLM: open weights, any model

vLLM, the open-source high-throughput serving engine, serves the open-weight models that don't ship as an NVIDIA NIM microservice: Google Gemma, OpenAI gpt-oss, Mistral NeMo, Cisco Foundation AI's Foundation-Sec security model, the Nomic embedding model and the purpose-built research models used in the ATC Cyber Range.

Why three engines: each engine is used where it is strongest, and the gateway hides the differences. NVIDIA NIM gives us optimized, supported NVIDIA models, vLLM covers everything else on Hugging Face, and NVIDIA Dynamo handles the models that need a whole rack.

The AI gateway: Kong AI Gateway with Kong Konnect

The gateway is what turns a collection of model servers into a service. LLMaaS runs Kong AI Gateway in a Kong Konnect hybrid deployment: Konnect is the software-as-a-service (SaaS) control plane, and the data planes run on our OpenShift cluster next to the models.

What Kong does in LLMaaS

• One unified entry point. Kong reads the model field from the request body and routes the request internally to that model's service. Builders use one base URL for every model.
• Authentication and entitlements. Each consumer has an API key and belongs to a group for each model and tier. A key works only for the models it is entitled to.
• Token-based rate limiting. Kong's AI Rate Limiting Advanced plugin counts tokens, not requests, in 60-second windows tracked in Redis across every data-plane replica. Each model has its own basic, plus and pro limits. A consumer that goes over its limit gets a standard HTTP 429 response instead of slowing everyone else down. Applications built on LLMaaS are expected to handle a 429 the way production apps handle any rate-limited API: back off and retry. The OpenAI and Anthropic SDKs already do this by default.
• Multi-provider proxying. The AI Proxy Advanced plugin speaks to self-hosted NVIDIA NIM, vLLM and NVIDIA Dynamo endpoints and to Anthropic, OpenAI, Azure AI Foundry and AWS Bedrock through the same plugin chain. It also injects the provider credentials, so consumers never handle a frontier-provider key.
• Native API formats. The gateway serves OpenAI-compatible chat, completions, embeddings and Responses APIs. It also serves Anthropic's native Messages API, so tools built on Anthropic's SDKs can point at LLMaaS without changes.
• Automatic failover. A Llama model runs on our own GPUs with AWS Bedrock as a standby target. If the local deployment fails or times out, Kong sends the request to Bedrock and the caller never notices.
• MCP governance. Kong's AI MCP Proxy sits in front of the LLMaaS Model Context Protocol server. Agents and AI assistants get the same authentication, access control and observability for their tool calls that model calls get.

Why Kong Konnect hybrid

Kong runs the control plane as a managed service, while the request path stays on data planes we operate. The SaaS control plane never sees a prompt or response for a self-hosted model. Configuration is declarative: our model catalog is rendered into Kong configuration and synced to Konnect by CI with Kong's declarative configuration tool. Tag-scoped sync lets automation own services, routes and plugins while consumer credentials are managed separately, so a config deploy never rotates anyone's key.

A deliberate design choice: the gateway governs, builders add guardrails

LLMaaS doesn't apply prompt filtering, personally identifiable information (PII) redaction or content-safety guardrails at the shared gateway, and that is on purpose. LLMaaS is a foundation for builders, and the right guardrail policy is different for a healthcare RAG assistant than for a security red-team lab. Builders who consume LLMaaS add the guardrails their own solution needs, such as NVIDIA NeMo Guardrails or their own gateway policies. The platform stays a neutral, governed and observable path to the models.

Frontier models: Anthropic, OpenAI, Microsoft Azure AI Foundry and AWS Bedrock

Not every use case should run on self-hosted open weights. LLMaaS puts frontier models behind the same key, the same rate-limit tiers and the same observability as everything else:

• Anthropic. The Claude family, including the Opus, Sonnet, Fable and Haiku models, plus earlier generations. Requests can use either the OpenAI-compatible format or Anthropic's native Messages API.
• OpenAI. The current GPT families, including models available only through the Responses API.
• Microsoft Azure AI Foundry. The Kimi model from Moonshot AI, served by Fireworks AI through Azure AI Foundry. The Foundry resources are provisioned as code with Azure Bicep. Azure Monitor metrics are pulled into our in-cluster Prometheus, so the Foundry-hosted model shows up on the same Grafana dashboards as the self-hosted ones.
• AWS Bedrock. The automatic standby for a self-hosted Llama model.

Why it matters: builders can compare a self-hosted open-weight model with a frontier model by changing one string in the request, with no new contract, key or SDK. That's the comparison most of our customers are trying to make.

How we operate it: models as code

The partners above provide the building blocks. The operating model is what lets a small team run them as a service:

1. One source of truth. Every model's routes, context window, rate-limit tiers and test flags live in one YAML inventory, validated against a JSON Schema.
2. Rendered, not hand-written. A templating step turns the inventory into Kong configuration, so a model's routes and limits can't drift.
3. Continuous integration and delivery (CI/CD) in the cluster. GitHub Actions run on self-hosted runners inside OpenShift. A merge to develop deploys DEV and a merge to main deploys PROD. Each run syncs to Konnect and publishes the live /v1/models catalog.
4. Automated testing of every model. After each sync, a test suite calls every model through the gateway, covering chat, completions, streaming, tool calling, structured output, multimodal input, embeddings and reranking. It uses the OpenAI, LangChain and LlamaIndex clients that builders actually use.
5. API-first onboarding. An admin API manages consumers, keys, entitlements and lab provisioning. The ATC platform calls it automatically when an on-demand lab spins up or tears down.
6. Agent-ready operations. An MCP server exposes a read-only slice of that admin API: the model catalog, usage analytics and lab listings. AI assistants can answer "which models does this lab have, and how many tokens has it used?" without a human looking it up.

What this means for you

LLMaaS is a working, production-grade example of an enterprise model-as-a-service platform, and every layer is built from technology you can deploy in your own data center:

• Run NVIDIA on mixed hardware. NVIDIA NIM, NVIDIA Dynamo, the NVIDIA GPU Operator and NVIDIA Run:ai run across NVIDIA DGX, Dell and HPE servers on Red Hat OpenShift.
• Govern every model with Kong AI Gateway. Self-hosted and frontier models sit behind one policy layer, with token-aware rate limits and provider failover.
• Store and protect model data with NetApp. Model weights, platform state and backups all live on enterprise storage.
• Keep frontier and open-weight options side by side, so model choice comes down to fit, cost and data sensitivity rather than integration effort.

Want to see it in action, or test your own models and use cases against it? Try the LLMaaS Chatbot lab (a chatbot backed by Kong AI Gateway) or the Hermes Agent lab (an agent with persistent memory, also powered by LLMaaS).

Related resources: AI Proving Ground overview · NVIDIA — Overview · Kong AI Gateway — Overview · Red Hat OpenShift — Overview · NetApp — Overview · NVIDIA Run:ai — Explore