QubitCache: Quantum-Inspired Probabilistic Attention Preservation for KV-Cache Compression

Abstract

Large language model inference faces a critical memory bottleneck from KV cache, which grows linearly with sequence length and dominates GPU memory during long-context generation. Existing compression methods reduce memory through token eviction but irreversibly discard attention relationships essential for complex reasoning. We present QubitCache, a framework that reframes KV cache compression as attention distribution compression rather than token selection. Our approach employs a quantum-inspired mathematical formalism (using the hierarchical amplitude encoding structure from quantum computing as a classical probability distribution compressor, without requiring quantum hardware) to represent attention weight distributions in logarithmic parameter space. QubitCache introduces a hybrid architecture: critical tokens (15% of sequence) remain in full-precision classical storage, while attention distributions over remaining tokens are compressed using hierarchical amplitude encoding, achieving a 57 & times; parameter reduction per segment. We provide formal analysis showing that the pipeline output error is bounded by the fraction of attention mass assigned to compressed tokens, which is small ( <= 15 %) due to inherent attention sparsity (Theorem 1). An important property of our design is that the compressed distribution is fixed after prefill and is therefore query-agnostic during decoding; we analyze the implications of this trade-off and provide empirical evidence that the resulting lambda(mass) stability is sufficient for the tasks evaluated. During inference, compressed distributions are reconstructed to provide soft attention weights, maintaining contextual influence of compressed tokens through probabilistic rather than binary decisions. We evaluate QubitCache on four long-context models across sixteen LongBench tasks and the RULER benchmark, comparing against both classical baselines (H2O, SnapKV) and recent methods (MiniKV, Compactor). QubitCache achieves approximately 6.7 & times; memory reduction while retaining 91-99% of full-KV performance, with an average of 74.8 on RULER, which is 10.5 points above token-eviction baselines. Latency analysis confirms that per-token decode overhead remains within 2 & times; of eviction-based methods. Comprehensive analysis including sensitivity, quantization integration, and failure cases characterizes when this approach succeeds and where it has limitations.