AI Gateway Hands-On Lab with Kong

Kong AI Gateway is a control plane that sits between your applications and the language models they use. Without a gateway, API keys scatter across teams, token spend is unpredictable, a single provider outage takes everything offline, and there is no record of who used what or how much it cost.

This lab shows how a single gateway layer solves all of those problems — without changing any application code. Working through ten modules, you will configure authentication, usage tracking, spend limits, prompt governance, load balancing, failover, agent tool access control, and declarative configuration management, each as a focused change followed by a live test.

The environment is pre-built: an AI Gateway data plane connected to WWT's LLMaaS (NVIDIA Nemotron models), a Redis cache, and a sample REST API. You make one change per module — attach a policy, flip a setting, set a cost limit — then immediately see the result in traffic and analytics.

By the end of the lab you will have a working AI gateway that enforces identity, controls spend, governs prompts, balances load across models, fails over gracefully, and manages agentic tool access, all from a single control plane with a full audit trail.

By the end of this lab you will be able to:

• Secure LLM access: require API keys, assign consumers to groups, and restrict model access by group without touching any application code.
• Track usage and cost: assign per-token pricing to models and use Konnect Analytics to see spend broken down by consumer and model.
• Enforce token budgets: attach rate-limiting policies that cap token consumption per consumer group and return a 429 when the budget is exhausted.
• Centralize prompt governance: apply an enterprise system prompt at the gateway so every request to a model includes it, regardless of what the client sends.
• Balance load across models: distribute traffic across multiple models by weight or by lowest latency, and add or remove models with no client change.
• Implement resilient failover: configure retry and failover criteria so a failed or overloaded primary target automatically shifts traffic to a backup.
• Govern agentic tool access: convert a REST API into MCP tools at the gateway, apply per-tool access control by consumer group.

• Windows Jumphost: pre-configured desktop with Google Chrome and PuTTY
• Kong Konnect:  SaaS control plane for configuring and monitoring the AI Gateway
• Kong AI Gateway data plane: Linux VM running the Kong data plane and Redis
• WWT LLMaaS: internal LLM service serving NVIDIA Nemotron models
• Redis 8: in-memory data store used for rate-limit counters
• Swagger Petstore v3: sample REST API used for the MCP server module