Ep. 034 - Engrams: How DeepSeek Offloads KV Cache to DRAM and SSD (Core Research) | Jordan Nanos, Cam Quilici, Alec Ibarra, Bryan Shan

Memory-bandwidth innovation (engrams, TPU v7, Vera Rubin's HBM4) is driving inference costs down while agentic-coding GPU serving margins approach 50%.

• Engrams offload memory: DeepSeek V4.1 Flash and Kimi-style models use n-gram-derived "engram" layers stored in hash tables, offloaded via UVA to CPU DRAM or SSD, to cut HBM pressure while overlapping retrieval with compute.
• AgentX replaces 8K1K benchmark: Built from replayed real SemiAnalysis coding-agent traffic, AgentX shows ~98% theoretical KV cache hit rates and 20-30% DRAM-offload hit rates at 40-60 TPS, high-concurrency workloads.

Themes, sentiment and reasons are AI-written paraphrases, checked so they never quote an episode. They may contain errors. Not investment advice. Episodes remain the property of their creators.