Running large language models at scale has always involved a brutal tradeoff: longer context windows deliver better results but tank performance and explode costs. A new technique from Tsinghua University and Z.ai researchers attacks this problem from an unexpected angle, achieving up to 1.82x faster response times on 200,000-token contexts by eliminating redundant computation that even the most efficient sparse attention models still perform.
The breakthrough, called IndexCache, targets a specific bottleneck in DeepSeek Sparse Attention (DSA) architecture—the same attention mechanism powering DeepSeek-V3.2 and the GLM model family. While DSA already reduces attention complexity from quadratic to linear by having models selectively focus on relevant tokens rather than processing every relationship, its "indexer" modules that determine which tokens matter still operate at quadratic complexity across every layer. As context lengths stretch into hundreds of thousands of tokens, these indexers become the dominant computational cost.
IndexCache's core insight is deceptively simple: adjacent layers in DSA models select nearly identical sets of important tokens, with 70-100% overlap between consecutive layers. Rather than recalculating these selections at every layer, the technique designates certain layers as "full" layers that compute and cache token indices, while "shared" layers simply reuse the cached results. In practice, this means eliminating 75% of indexer computations with minimal impact on model quality.
Why this matters beyond raw speed gains
The timing of this research addresses a critical inflection point in enterprise AI deployment. As organizations move beyond proof-of-concept chatbots into production systems handling complex document analysis, multi-step reasoning chains, and retrieval-augmented generation pipelines, context length requirements have exploded. A legal contract review system might need to process 50-page agreements. A customer service agent analyzing conversation history could easily hit 100,000 tokens. These aren't edge cases—they're becoming standard workloads.
Traditional approaches to the attention bottleneck have focused on KV cache compression, reducing the memory footprint of stored attention values. IndexCache operates orthogonally by targeting compute rather than memory. This means it can stack with existing optimizations. An enterprise running compressed KV caching can layer IndexCache on top for compounding efficiency gains—a crucial consideration when every percentage point of improvement translates to thousands of dollars in infrastructure costs at scale.
The researchers validated this on GLM-4.7 Flash, a 30-billion-parameter model, where removing 75% of indexers cut prefill latency from 19.5 to 10.7 seconds at 200K context length. During generation, throughput jumped from 58 to 86 tokens per second. When servers ran at full capacity, total decode throughput increased 51%. These aren't marginal improvements—they represent the difference between a system that feels sluggish and one that feels responsive.
Two paths to implementation, each with distinct tradeoffs
The research team developed separate approaches for different deployment scenarios. For teams using pre-trained DSA models where retraining isn't practical, a training-free method uses greedy layer selection. Run a small calibration dataset through the model, and an algorithm automatically determines which layers should compute fresh indices versus reusing cached ones. No weight updates required.
The catch: calibration data quality matters significantly. Generic datasets might produce suboptimal layer configurations that work adequately on average but underperform on your specific workload. Co-author Yushi Bai recommends using domain-specific data during calibration so the discovered sharing pattern aligns with real-world usage. A financial services firm processing earnings reports should calibrate on financial documents, not general web text.
For organizations training or heavily fine-tuning foundation models, a training-aware version optimizes the network to natively support cross-layer sharing. This introduces a multi-layer distillation loss during training, forcing retained indexers to learn consensus token selections that remain relevant across multiple subsequent layers. The result: better quality at higher compression ratios, but requiring full training runs.
Preliminary tests on the 744-billion-parameter GLM-5 model using the training-free approach showed at least 1.3x speedup on contexts exceeding 100K tokens while maintaining nearly identical quality scores on long-context benchmarks. More remarkably, on the AIME 2025 math reasoning benchmark, the optimized 30B model actually outperformed the baseline, scoring 92.6 versus 91.0—suggesting that aggressive indexer removal doesn't just preserve reasoning capability but might occasionally improve it by reducing noise.
Practical deployment considerations and cost implications
For development teams ready to implement IndexCache, open-source patches are available for major serving engines including vLLM and SGLang. Integration requires minimal configuration changes, though teams should budget time for proper calibration. The process isn't plug-and-play, but it's far simpler than retraining models or rebuilding inference infrastructure.
The ROI calculation varies by workload. Bai notes that long-context applications like RAG systems, document analysis, and agentic pipelines see approximately 20% deployment cost reductions alongside similar latency improvements. Short-context tasks benefit less, hovering around 5% gains. This makes IndexCache particularly valuable for enterprises whose AI systems increasingly handle complex, context-heavy workflows rather than simple question-answering.
The broader implication extends beyond immediate performance wins. IndexCache represents a shift toward designing models with inference constraints as first-class considerations rather than afterthoughts. As Bai observes, future foundation models will likely be architected from the ground up for real-world throughput and latency, not just parameter count and benchmark scores. This matters because the industry's current trajectory—ever-larger models with ever-longer contexts—hits hard physical limits without architectural innovations that make inference economically viable.
For enterprises evaluating whether to adopt IndexCache, the decision hinges on context length requirements and model architecture. If you're running DSA-based models (DeepSeek or GLM families) with contexts regularly exceeding 50K tokens, the technique offers substantial benefits with manageable implementation complexity. Teams using other architectures will need to wait for similar techniques adapted to their models, though the underlying principle of exploiting cross-layer redundancy likely generalizes beyond DSA.
The research also highlights a less obvious point: optimization opportunities still exist even in models already using advanced efficiency techniques. DSA was already considered highly optimized, yet IndexCache found 75% of its indexer computations were redundant. This suggests the current generation of "efficient" architectures may still harbor significant waste, and that careful empirical analysis of how models actually process information can reveal optimization opportunities that pure architectural reasoning misses.
