Breaking the Long-Context Memory Wall
Serve 3× more concurrent long-context users per GPU
Measured end-to-end at 4,096 context length and across 4K–8K decode attention workloads on identical single-GPU hardware.
4×–8× Smaller KV Cache
Shrinks 8-stream 4K KV cache from 3,584 MB → 896 MB (4× mode) or 448 MB (8× mode), eliminating out-of-memory ceilings.
3.07× Attention · 2.03× Batch
Sustains 24 concurrent 4K streams cleanly in 1.34 GB of KV memory where standard uncompressed serving runs out of GPU memory.
100% 1K–8K NIAH · 95% CoT Math
At 4× KV compression: 100% 1K–8K single & multi-needle recall, 100% 3-hop reading, 93.8% executable Python Pass@1, and 16 concurrent 8K streams on one GPU.
1.7×–3.1× faster across batch 1 to 16
| Batch & Context | PyTorch SDPA | CoreKV 4× | CoreKV 8× | Peak Gain |
|---|---|---|---|---|
| batch=1, ctx=4,096 | 49.4 μs | 35.5 μs (1.39×) | 29.3 μs | 1.69× faster |
| batch=8, ctx=4,096 | 444.8 μs | 193.6 μs (2.30×) | 149.4 μs | 2.98× faster |
| batch=8, ctx=8,192 | 845.2 μs | 375.4 μs (2.25×) | 289.8 μs | 2.92× faster |
| batch=16, ctx=4,096 | 890.8 μs | 375.7 μs (2.37×) | 290.0 μs | 3.07× faster |
Higher concurrency without upgrading GPUs
| Serving Mode | KV Memory | Throughput | Batch Gain |
|---|---|---|---|
| Standard Uncompressed (b=8) | 3,584 MB | 583.6 tok/s | 1.00× |
| CoreKV 4× Mode (b=8) | 896 MB (4× cut) | 546.3 tok/s | 0.94× |
| CoreKV 8× Mode (b=8) | 448 MB (8× cut) | 624.9 tok/s | 1.07× |
| CoreKV 8× Mode (b=24)* | 1,344 MB | 1,187.0 tok/s | 2.03× |
*Uncompressed serving OOMs at batch=24 (ctx=4,096) on identical GPU hardware.
100% Needle-in-a-Haystack (1K–8K), 100% Multi-Needle & 95% Multi-Step Reasoning at 4× KV Cut
| Unseen Long-Context / Multi-Step Benchmark | Uncompressed Standard (1×) | CoreKV 4× Mode | CoreKV 8× Mode |
|---|---|---|---|
| Needle-in-a-Haystack (1K / 2K / 4K / 8K) | 100% / 100% / 100% / 100% | 100% / 100% / 100% / 100% (20/20) | 100% / 100% / 100% / 100% (20/20) |
| Multi-Needle 3× Simultaneous (2K / 4K) | 100.0% / 77.8% | 100.0% / 100.0% (18/18) | 77.8% / 55.6% |
| 4K Positional Perplexity — Scientific | 6.50 | 6.72 | 7.90 |
| 4K Positional Perplexity — Python Code | 5.02 | 4.05 (beats baseline) | 3.96 (beats baseline) |
| Multi-Hop 3-Hop Reading (~1.5K ctx) | 43.8% | 100.0% (16/16) | 75.0% (12/16) |
| Multi-Step Reasoning (CoT Math, 20 tasks) | 85.0% | 95.0% (19/20) | 60.0% |
| Executable Python Unit-Test Pass@1 (16 tasks) | 100.0% | 93.8% (15/16) | 50.0% |
| High-Concurrency 4K Serving (24 streams, Single GPU) | Out of Memory (343 tok/s max @ b=8) | 696.0 tok/s (2.03× · 3,456 MB) | 434.6+ tok/s (1,728 MB) |
| High-Concurrency 8K Serving (16 streams, Single GPU) | Out of Memory (OOM) | 363.4 tok/s (4,608 MB) | 229.8+ tok/s (2,304 MB) |
| Peak Decode Attention Speedup vs. PyTorch SDPA | 1.00× | 2.70× faster | 4.52× faster |
Evaluated on unseen scientific archives, held-out Python unit tests, and 1K–8K needle/multi-hop benchmarks on identical single-GPU hardware.
Enterprise Use Cases
Where CoreKV slashes serving infrastructure costs immediately
Built for teams self-hosting open or custom domain-adapted models on cloud GPUs or private on-premise clusters.
Cut Self-Hosted GPU Fleet Bills by 50–75%
Challenge: Multi-tenant enterprise copilots with 8K–32K contexts hit GPU VRAM limits at low batch sizes, forcing ops teams to scale out expensive GPU replicas while compute utilization sits below 30%.
Outcome with CoreKV: Cuts KV cache memory by 4×–8× so each GPU handles up to 3× more simultaneous users, raising aggregate throughput up to 2.03× on the same fleet.
Best fit: AI SaaS platforms, customer chat hosts, inference providers
Repository-Scale Coding Without Context Truncation
Challenge: Internal code assistants must hold multi-file diffs, documentation, and compiler traces across hundreds of concurrent engineers without losing syntax or logic accuracy.
Outcome with CoreKV: Compresses KV state 4× while maintaining ultra-sharp unseen code quality (5.42 vs. 7.30 baseline Python PPL) and fast interactive token streaming.
Best fit: Enterprise DevTools, IDE copilots, automated PR review
Long-Document Analysis on Compact Private Hardware
Challenge: Privacy-regulated teams reviewing 100-page contracts, M&A data rooms, or clinical charts often need multi-GPU nodes inside air-gapped VPCs solely for long-context KV RAM.
Outcome with CoreKV: Fits high-concurrency long-document serving onto compact single-GPU on-premise appliances with zero external cloud dependencies.
Best fit: Law firms, hospital networks, sovereign & defense VPCs
Persistent Multi-Step Agent Working Memory
Challenge: Background agents running 20–50 tool calls accumulate massive KV histories that stall parallel worker queues or force expensive context recomputation.
Outcome with CoreKV: Keeps dozens of long-horizon agent trajectories resident in VRAM simultaneously, dramatically lowering cost per completed task.
Best fit: Autonomous research, coding, finance & ops agent fleets
Turnkey Integration
Pre-optimized checkpoints or custom conversion of your own models
Deploy our ready-to-serve CoreKV bundles immediately, or let our team retrofit 4×–8× CoreKV compression onto your proprietary enterprise models inside your VPC.
Self-Contained Serving Bundle
Zero architecture lock-in
Each CoreKV delivery ships as a self-contained model directory with its drop-in runtime—ready for single-node PyTorch serving or containerized VPC deployment.
- 4× High-Fidelity Mode: Near-zero quality loss, 4× KV cut, 1.78×+ max-batch gain
- 8× Max-Density Mode: 8× KV cut, up to 3.07× faster decode attention, 2.03× max-batch gain
- Custom Model Conversion: Rapid retrofit onto your private enterprise checkpoints
# Drop-in self-contained CoreKV serving runtime
from corekv_runtime import load_corekv_model, corekv_session
model, tokenizer, kv_engine = load_corekv_model(
"./models/enterprise-corekv-4x",
device="cuda"
)
with corekv_session(model):
out = model.generate(
**inputs,
past_key_values=kv_engine.new_cache(),
max_new_tokens=512
)
Next Step
Benchmark CoreKV on your target context length & batch size
Share your target concurrency, context length, and GPU type for a custom throughput & GPU-cost reduction projection.