Red Hat AI Inference Sandbox

Training a model gets the headlines, but most of the GPU hours an organization spends on generative AI go to inference: answering prompts, one token at a time, for many users at once. In this lab you take one NVIDIA L40S GPU server and serve the same Llama 3.2 1B Instruct model on it twice. First you use Ollama, the tool many individuals reach for first. Then you use the Red Hat AI Inference container, Red Hat's packaged and supported build of vLLM.

You measure both servers with vllm bench serve, for one user and for a crowd, and see where they diverge. You also compress a model to FP8 with the same LLM Compressor toolkit Red Hat uses, serve it, and compare it with the original. The lab does not claim one tool is better at everything. It shows which of Red Hat's claims a single GPU can and cannot demonstrate.

Imagine your team has been asked to stand up a private LLM endpoint so internal applications can stop sending data to a public API. Someone has already proved it works by running Ollama on a spare GPU box. Your job is to decide what you would actually put into production.

By the end of this lab, you will be able to:

• Verify that a RHEL host and rootless Podman can see an NVIDIA GPU
• Serve a model with Ollama and with the Red Hat AI Inference container, and call each through an OpenAI-compatible API
• Benchmark both servers with vllm bench serve, for one user and for a crowd, and read the results
• Compress a model to FP8 with LLM Compressor and compare it with the original
• Explain what Red Hat AI Inference adds, and which of its claims this lab can and cannot show