NVIDIA Switchyard

NVIDIA Switchyard is an open-source, OpenAI-compatible proxy that decides which model handles each request through configuration, not application code. One TOML file defines three layers: an llm_clients connection, targets naming specific backend models, and routes that either resolve to one fixed target (passthrough) or hand the decision to a judge model (llm_classifier) that picks per request — by difficulty or by task category. A companion dashboard makes those decisions observable: which model answered, what it cost against a frontier baseline, and whether the answer held up.

Goal: Teach config-driven LLM request routing using NVIDIA Switchyard — replacing a hard-coded, one-model-for-everything integration with a runtime routing decision governed by config, so cost and model choice match what each request actually needs.

Learning Objectives — by the end of the lab, you will be able to:

1. Explain what NVIDIA Switchyard is and the problem config-driven LLM routing solves versus hardcoding one model into an application
2. Deploy the NVIDIA Switchyard + dashboard Docker Compose stack and confirm both services report healthy
3. Configure and use passthrough routes to reach a specific backend model directly
4. Differentiate between switchyard/smart (binary weak/strong routing) and switchyard/router (5-way category routing), and explain how each classifier makes its decision
5. Measure the cost, latency, and quality difference between routed and baseline model calls
6. Modify routes.toml.template to add or change a routed model, redeploy, and verify the change took effect

• NVIDIA Switchyard
• Docker Host
• Kong AI Gateway for LLM Access
• Windows 11 Jumpbox