Attention! Keys, Values, Caches, Offloading, WEKA, KOVE and the Research Community

We immediately consider the size of a model when thinking about what GPU class and therefore memory we need, but in large-scale inference deployments transformers' full-attention's KV memory use far exceeds the memory needed to store model weights. Meanwhile, full-attention is unrivaled for end-task accuracy. Alternative attention mechanisms with lower demands exist (e.g., MAMBA), but modifying a model's architecture at that level can't be done post-training.

 

"KV management" approaches (including KV caching and compression/quantization) exist as engineering mitigations for the compute and memory demands of traditional attention heads. Crucially, these act within the inference stack as post-training solutions with direct favorable impact to TCO.

There are risks: a bad KV cache eviction algorithm can kill accuracy or increase latency; and some compression algos create significant complexities such as the need for impractical or expensive offline workload analysis. On the other hand, the right algo can significantly enhance throughput with only mild impacts to accuracy or performance, and can avoid offline complexities.

Vendor solutions and recent research target KV cache management and compression. Broadly, they split into two families: capacity/persistence approaches that expand where the cache can live and persist it to kill redundant prefill, and compression approaches that shrink the cache itself through quantization or eviction, with a third, hybrid pattern emerging that layers both.

Capacity & persistence (don't shrink the cache; relocate and reuse it)

• WEKA: Their Augmented Memory Grid offloads KV cache* to a "token warehouse" on the WEKA NeuralMesh data platform via  NVIDIA Magnum IO GPUDirect Storage (GDS). This bypasses CPU bottlenecks and delivers data directly to GPU memory. WEKA claims this expands effective memory capacity by 1000x beyond DRAM at roughly 300GB/s per host, eliminating redundant prefill for shared prefixes across queries and sessions. Production benchmarks on OCI reported nearly 20x faster time to first token (TTFT) at 128K context versus baseline vLLM, with WEKA also citing up to 6.5x more input-token throughput and 10x more simultaneous long-context sessions on the same GPU cluster.
  
  It's a storage-class tier: enormous, cheap and persistent, but slower than DRAM, best where cache-hit rates and prefill reuse dominate the cost. References: 1, 2, 3, 4
  
  *NOTE: A note on "cache" vs "offload": In the ML literature, KV "caching" is frequently about cache eviction mechanisms where the KV's are lost once evicted. This can lead to accuracy degradataion because those tokens can never attend to future tokens; thus the high importance of accurately selecting evictions. Offloading, on the other hand, stores the KVs in lower bandwidth memory, trading space for time (instead of space for accuracy).
   
• KOVE: In Kove:SDM (Software-Defined Memory), DRAM is pooled across servers over the network, giving KV-heavy workloads more memory at DRAM-class latency. Kove positions this against the alternatives explicitly: CXL is hardware-based and adds latency; storage tiering (NVIDIA, DDN, Weka, etc.) offloads cache to SSD/storage and is slower than DRAM. SDM runs on existing x86 servers with no rewrites, no kernel changes and no new hardware. By removing per-server DRAM ceilings it reduces recomputation, lowers GPU idle time and sustains throughput as context windows grow. Benchmarks show 5x larger workloads at stable latency. SDM is a DRAM-class capacity play that sits between HBM and storage tiers on the latency/cost curve.

Compression & eviction (shrinking the cache itself)

• TurboQuant: A vector-quantization algorithm from Google Research (ICLR 2026) for online, data-oblivious KV cache quantization. It pairs a random rotation causes rotated coordinates into a Gaussian distribution, with Lloyd-Max scalar quantization plus a 1-bit QJL residual correction, thereby compressing the KV cache to 3 bits per coordinate and operating within a factor of ≈2.7 of the information-theoretic limit. Critically for the offline-complexity concern mentioned in the introduction to this article, it is data-oblivious and requires no training or calibration. The tradeoffs are real, though: it compresses only the KV-cache storage to 3–4 bits and dequantizes back to BF16 for the attention computation. vLLM's independent study found value quantization is the weak point, concluding that 4bit is likely the most practical variant. This technique helps under KV-cache memory pressure, but trades capacity for moderate accuracy, latency and throughput costs.
  
  My prior blog about TurboQuant is here.
   
• TriAttention: A KV cache eviction/pruning method for long-context reasoning from MIT, NVIDIA and Zhejiang University. Its insight is that mainstream eviction methods (SnapKV, H2O, R-KV) score token importance from post-RoPE attention, which is unstable because queries rotate with position during RoPE, making representative queries very few and leading to poor top-key selection. TriAttention instead works in the pre-RoPE space, where Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions. It scores cached keys using a trigonometric series and Q/K norms without needing any live query observations. Reported results are for 2.5x higher throughput than full attention at equivalent accuracy, and a 10.7x KV cache memory reduction while matching full-attention accuracy on AIME25; on MATH500 it hit 1,405 tokens/s vs. 223 for full attention, at 68.4% vs. 69.6% accuracy.
  
  Materials that will help you better understand TriAttention are available here.

Hybrid (compress and tier, the direction production stacks are converging on)

• LMCache and layered architectures: Increasingly, deployments don't choose between compression and offload; they stack them. LMCache is a leading open example: an efficient KV cache layer for enterprise-scale inference that reflects the trend of moving KV cache outside GPU memory along two emerging directions: cross-query caching, where the cache is persisted to lower-tier storage beyond a query's lifecycle to avoid re-computing shared prefixes, and prefill-decode disaggregation, where prefill and decode run on different GPUs and the KV cache produced by prefill GPUs is transferred to decode GPUs (an architecture we see in some of the latest offerings from major GPU vendors). It is already embedded in mainstream serving stacks: llm-d offloads KV cache with LMCache from GPU memory to CPU memory and network disks; KServe integrates it to reduce inference costs while holding latency and throughput SLOs; and vLLM uses it for CPU offloading, cache sharing between requests and disaggregated prefill. In a mature stack, a quantizer (TurboQuant-style) or pruner (TriAttention-style) sets the per-token cost, and a tiering layer (LMCache, WEKA, KOVE) sets how much of that reduced cache stays hot and reusable. In this way, the techniques multiply rather than compete.

The capacity plays (WEKA, KOVE) and the compression plays (TurboQuant, TriAttention) attack opposite ends of the same bill. The former leave the cache full-precision but make it cheaper and more persistent to hold. Their wins come from prefill reuse and higher concurrency. The latter shrink the footprint directly with fewer bits per vector and fewer vectors, respectively. Neither depends on impractical offline workload analysis. Finally, the hybrid pattern shows all of the solutions are complementary layers, not competitors.

TurboQuant and TriAttention deserve emphasis for a second reason: velocity of research into industry. TurboQuant went from an ICLR 2026 paper to community Triton kernels with vLLM integration and an independent, production-oriented evaluation published on the vLLM blog within months. TriAttention arrived in April 2026 with reproducible throughput numbers against the exact eviction baselines (SnapKV, H2O, R-KV) already shipping in popular serving frameworks.

These days, machine learning / language model research papers are not far-future research artifacts awaiting a hardware generation or a retraining cycle; they are often pragmatic drop-in candidates for current inference software stacks. This is precisely because KV management operates post-training.

The practical implication for solutions architects and engineers

The KV-management layer is where academic results convert to deployable TCO improvements on the shortest cycle anywhere in the AI infrastructure stack, often quarters, not years. Maintaining currency with this research stream is not an academic nicety; it is a competitive requirement.

A team that evaluated its serving stack twelve months ago is likely leaving 2–10x of throughput or capacity on the table, and the vendors above are productizing these findings on the same timeline.

Architects should treat the KV cache literature the way they treat CVE feeds or kernel changelogs: something to monitor continuously, validate against their own workloads and fold into refresh cycles as a matter of course.