Essential to AI Performance: KV Cache Solutions from Vendors & Academia

We immediately consider the size of a model when thinking about what GPU class and therefore memory we need, but in large-scale inference deployments transformers' full-attention's KV memory use far exceeds the memory needed to store model weights. Meanwhile, full-attention is unrivaled for end-task accuracy. Alternative attention mechanisms with lower demands exist (e.g., MAMBA), but modifying a model's architecture at that level can't be done post-training.

 

"KV management" approaches (including KV caching and compression/quantization) exist as engineering mitigations for the compute and memory demands of traditional attention heads. Crucially, these act within the inference stack as post-training solutions with direct favorable impact to TCO.

There are risks: a bad KV cache eviction algorithm can kill accuracy or increase latency; and some compression algos create significant complexities such as the need for impractical or expensive offline workload analysis. On the other hand, the right algo can significantly enhance throughput with only mild impacts to accuracy or performance, and can avoid offline analysis complexities. It's also important to note the link between expected benefits and the prompting pattern of the workloads: If prompts are unique one-shots, or nearly all traffic shares a single resident prefix, offload adds I/O cost with no reuse to pay for it.

Vendor solutions and recent research target KV cache management and compression. Broadly, they split into two families: capacity/persistence approaches that expand where the cache can live and persist it to kill redundant prefill, and compression approaches that shrink the cache itself through quantization or eviction, with a third, hybrid pattern emerging that layers both.

Capacity & persistence (don't shrink the cache; relocate and reuse it)

• WEKA: Their Augmented Memory Grid offloads KV cache* to a "token warehouse" on the WEKA NeuralMesh data platform via  NVIDIA Magnum IO GPUDirect Storage (GDS). This bypasses CPU bottlenecks and delivers data directly to GPU memory. WEKA claims this expands effective memory capacity by 1000x beyond DRAM at roughly 300GB/s per host, eliminating redundant prefill for shared prefixes across queries and sessions. Production benchmarks on OCI reported nearly 20x faster time to first token (TTFT) at 128K context versus baseline vLLM, with WEKA also citing up to 6.5x more input-token throughput and 10x more simultaneous long-context sessions on the same GPU cluster.
  
  It's a storage-class tier: enormous, cheap and persistent, but slower than DRAM, best where cache-hit rates and prefill reuse dominate the cost. References: 1, 2, 3, 4
  
  *NOTE: A note on "cache" vs "offload": In the ML literature, KV "caching" is frequently about cache eviction mechanisms where the KV's are lost once evicted. This can lead to accuracy degradataion because those tokens can never attend to future tokens; thus the high importance of accurately selecting evictions. Offloading, on the other hand, stores the KVs in lower bandwidth memory, trading space for time (instead of space for accuracy).
   
• KOVE: In Kove:SDM (Software-Defined Memory), DRAM is pooled across servers over the network, giving KV-heavy workloads more memory at DRAM-class latency. Kove positions this against the alternatives explicitly: CXL is hardware-based and adds latency; storage tiering (NVIDIA, DDN, Weka, etc.) offloads cache to SSD/storage and is slower than DRAM (see the DDN, VAST and Pure entries below for how those vendors position that tradeoff). SDM runs on existing x86 servers with no rewrites, no kernel changes and no new hardware. By removing per-server DRAM ceilings it reduces recomputation, lowers GPU idle time and sustains throughput as context windows grow. Benchmarks show 5x larger workloads at stable latency. SDM is a DRAM-class capacity play that sits between HBM and storage tiers on the latency/cost curve.
   
• DDN: DDN integrates KV cache offload directly into its Infinia data intelligence platform, engineered for high-concurrency, sub-millisecond retrieval latency, 100,000+ concurrent calls per second, and purpose-built integration with NVIDIA GPUs and DPUs. Cache misses (and therefore recomputation of context) are positioned as a "GPU tax." In a published disaggregated-inference test, Infinia cut context processing and delivered roughly a 25–27x TTFT improvement. DDN models the downstream economics as a ~75% reduction in input-token processing cost for multi-turn workloads (typical in agentic systems). Like WEKA, it's a storage-class persistence tier whose wins come from prefill reuse but pitches the KV cache capability as native to the platform rather than bolted on (reference).
   
• VAST Data: NVIDIA Dynamo/NIXL handle accelerated movement between GPU memory and storage. vLLM with the LMCache kv_connector persists and retrieves caches from a file namespace on VAST, highlighting its approach's potential to be the KV cache offload solution within the emerging open, disaggregated serving stack. VAST's validation tested persistence layers independenlty, first saturating a single H100 at 35GB/s reads over the NIXL GDS plugin to prove storage wasn't the bottleneck before measuring application-level prefill. For Qwen3-32B at a ~131K-token context (a 32GB cache), the cache cut TTFT from over 11 seconds to about 1.5 seconds, for a ~10x improvement. There is protocol flexibility too, as GDS and NFSoRDMA are supported but optional, with full line-rate claimed over standard NFSoTCP. Like WEKA and DDN, VAST acts as a persistent storage tier below DRAM with an emphasis on standardized interfaces (Dynamo, NIXL, LMCache, llm-d) (reference).
   
• Pure (Everpure) KVA: Pure Key Value Accelerator 1.0, GA as of June 2026, is a managed KV-offload subsystem by disaggregating the storage path from the inference path. A thin plugin interfaces with a long-lived daemon that owns serialization, backend I/O, the cache index, and lifecycle. Associated storage stalls degrade cache performance instead of stalling (or crashing) the engine serving live traffic. KV blocks move from GPU-to-NVMe without the CPU through shared GPU memory handles. The solution has fleet-scale lifecycle management: TTL expiry by hourly time bins, election-coordinated cleanup across hosts sharing one storage pool, and clocks are synchronized for correct expiries; and there is systemd packaging and an OpenTelemetry metrics pipeline.
  
  Clearly, KV offload is maturing from a performance hack into an operable enterprise subsystem, with cache reuse across engines and hosts as the payoff (reference).
   
• NetApp: vLLM offloads to ONTAP via LMCache's GDS backend using industry-standard NFS protocols enterprises have run for decades, without a proprietary client. NetApp's analysis is honest about workload dependence by walking through three traffic patterns, concluding that offloading is worse in two: When there are many unique one-shot prompts (low cache hits); and when one long shared prefix exists in most prompts, thereby never being evicted from HBM. Many long prompts sharing multiple common prefixes, as in agentic systems or RAG serving several departments, benefit most due to routinely evicted and reused KVs. In a simulation with 30K-token prompts with 28K-token prefixes drawn from a set of 20, offload to ONTAP delivered a 61% improvement in total token throughput, 34% lower mean TTFT, and 43% lower inter-token latency versus baseline vLLM (reference).

Compression & eviction (shrinking the cache itself)

• TurboQuant: A vector-quantization algorithm from Google Research (ICLR 2026) for online, data-oblivious KV cache quantization. It pairs a random rotation causes rotated coordinates into a Gaussian distribution, with Lloyd-Max scalar quantization plus a 1-bit QJL residual correction, thereby compressing the KV cache to 3 bits per coordinate and operating within a factor of ≈2.7 of the information-theoretic limit. Critically for the offline-complexity concern mentioned in the introduction to this article, it is data-oblivious and requires no training or calibration. The tradeoffs are real, though: it compresses only the KV-cache storage to 3–4 bits and dequantizes back to BF16 for the attention computation. vLLM's independent study found value quantization is the weak point, concluding that 4bit is likely the most practical variant. This technique helps under KV-cache memory pressure, but trades capacity for moderate accuracy, latency and throughput costs.
  
  My prior blog about TurboQuant is here.
   
• TriAttention: A KV cache eviction/pruning method for long-context reasoning from MIT, NVIDIA and Zhejiang University. Its insight is that mainstream eviction methods (SnapKV, H2O, R-KV) score token importance from post-RoPE attention, which is unstable because queries rotate with position during RoPE, making representative queries very few and leading to poor top-key selection. TriAttention instead works in the pre-RoPE space, where Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions. It scores cached keys using a trigonometric series and Q/K norms without needing any live query observations. Reported results are for 2.5x higher throughput than full attention at equivalent accuracy, and a 10.7x KV cache memory reduction while matching full-attention accuracy on AIME25; on MATH500 it hit 1,405 tokens/s vs. 223 for full attention, at 68.4% vs. 69.6% accuracy.
  
  Materials that will help you better understand TriAttention are available here.

Hybrid (compress and tier, the direction production stacks are converging on)

• LMCache and layered architectures: Increasingly, deployments don't choose between compression and offload; they stack them. LMCache is a leading open example: an efficient KV cache layer for enterprise-scale inference that reflects the trend of moving KV cache outside GPU memory along two emerging directions: cross-query caching, where the cache is persisted to lower-tier storage beyond a query's lifecycle to avoid re-computing shared prefixes, and prefill-decode disaggregation, where prefill and decode run on different GPUs and the KV cache produced by prefill GPUs is transferred to decode GPUs (an architecture we see in some of the latest offerings from major GPU vendors). It is already embedded in mainstream serving stacks: llm-d offloads KV cache with LMCache from GPU memory to CPU memory and network disks; KServe integrates it to reduce inference costs while holding latency and throughput SLOs; and vLLM uses it for CPU offloading, cache sharing between requests and disaggregated prefill. Notably, both DDN's and VAST's published KV cache results run through LMCache as the framework layer — vendor storage tiers and the open caching layer are already shipping as a stack, not as alternatives. In a mature stack, a quantizer (TurboQuant-style) or pruner (TriAttention-style) sets the per-token cost, and a tiering layer (LMCache, WEKA, KOVE, DDN Infinia, VAST, NetApp ONTAP, Pure KVA) sets how much of that reduced cache stays hot and reusable. In this way, the techniques multiply rather than compete.

The capacity plays (WEKA, KOVE, DDN, VAST, NetApp, Pure) and the compression plays (TurboQuant, TriAttention) attack opposite ends of the same bill. The former leave the cache full-precision but make it cheaper and more persistent to hold. Their wins come from prefill reuse and higher concurrency. Within the capacity family there's a spectrum: KOVE at DRAM-class latency, WEKA/DDN/VAST as GDS-fed storage tiers competing on bandwidth and TTFT claims, and Pure KVA competing on operability and fleet lifecycle management. The latter shrink the footprint directly with fewer bits per vector and fewer vectors, respectively. Neither depends on impractical offline workload analysis. Finally, the hybrid pattern shows all of the solutions are complementary layers, not competitors.

TurboQuant and TriAttention deserve emphasis for a second reason: velocity of research into industry. TurboQuant went from an ICLR 2026 paper to community Triton kernels with vLLM integration and an independent, production-oriented evaluation published on the vLLM blog within months. TriAttention arrived in April 2026 with reproducible throughput numbers against the exact eviction baselines (SnapKV, H2O, R-KV) already shipping in popular serving frameworks.

These days, machine learning / language model research papers are not far-future research artifacts awaiting a hardware generation or a retraining cycle; they are often pragmatic drop-in candidates for current inference software stacks. This is precisely because KV management operates post-training.

The practical implication for solutions architects and engineers

The KV-management layer is where academic results convert to deployable TCO improvements on the shortest cycle anywhere in the AI infrastructure stack, often quarters, not years. Maintaining currency with this research stream is not an academic nicety; it is a competitive requirement. Architects should treat the KV cache literature the way they treat CVE feeds or kernel changelogs: something to monitor continuously, validate against their own workloads and fold into refresh cycles as a matter of course. 

A team that evaluated its serving stack twelve months ago is likely leaving 2–10x of throughput or capacity on the table, and the vendors above are productizing these findings on the same timeline. Workload characterization should be an ongoing practice under shifting use cases. The same offload tier that delivers a throughput gain on a multi-prefix agentic workload is pure overhead on a stream of unique one-shot prompts. Profile prefix reuse and eviction rates before buying capacity, not after.