The Reality of Running AI Locally

128 GB of unified memory sounds like it can run anything, but a model that fits isn't always a model that runs fast. In this video you'll see why memory bandwidth, not compute, usually sets your speed when one user is generating text. You'll also see why mixture-of-experts models can run many times faster than dense ones of the same size, and what changes when several workloads share one pool of memory. Next, read The Bandwidth Ceiling and Sharing One Memory.