Longinus Research.

Breaking the Long-Context Memory Wall

Serve 3× more concurrent long-context users per GPU

Measured end-to-end at 4,096 context length and across 4K–8K decode attention workloads on identical single-GPU hardware.

Memory Savings

4×–8× Smaller KV Cache

Shrinks 8-stream 4K KV cache from 3,584 MB → 896 MB (4× mode) or 448 MB (8× mode), eliminating out-of-memory ceilings.

Serving Throughput

3.07× Attention · 2.03× Batch

Sustains 24 concurrent 4K streams cleanly in 1.34 GB of KV memory where standard uncompressed serving runs out of GPU memory.

Verified Long-Context Quality

100% 1K–8K NIAH · 95% CoT Math

At 4× KV compression: 100% 1K–8K single & multi-needle recall, 100% 3-hop reading, 93.8% executable Python Pass@1, and 16 concurrent 8K streams on one GPU.

Decode Attention Latency vs. PyTorch SDPA

1.7×–3.1× faster across batch 1 to 16

Batch & Context PyTorch SDPA CoreKV 4× CoreKV 8× Peak Gain
batch=1, ctx=4,096 49.4 μs 35.5 μs (1.39×) 29.3 μs 1.69× faster
batch=8, ctx=4,096 444.8 μs 193.6 μs (2.30×) 149.4 μs 2.98× faster
batch=8, ctx=8,192 845.2 μs 375.4 μs (2.25×) 289.8 μs 2.92× faster
batch=16, ctx=4,096 890.8 μs 375.7 μs (2.37×) 290.0 μs 3.07× faster
End-to-End 4,096-Context Serving

Higher concurrency without upgrading GPUs

Serving Mode KV Memory Throughput Batch Gain
Standard Uncompressed (b=8) 3,584 MB 583.6 tok/s 1.00×
CoreKV 4× Mode (b=8) 896 MB (4× cut) 546.3 tok/s 0.94×
CoreKV 8× Mode (b=8) 448 MB (8× cut) 624.9 tok/s 1.07×
CoreKV 8× Mode (b=24)* 1,344 MB 1,187.0 tok/s 2.03×

*Uncompressed serving OOMs at batch=24 (ctx=4,096) on identical GPU hardware.

Long-Context & Multi-Step Verification (Strictly Unseen Tasks)

100% Needle-in-a-Haystack (1K–8K), 100% Multi-Needle & 95% Multi-Step Reasoning at 4× KV Cut

Unseen Long-Context / Multi-Step Benchmark Uncompressed Standard (1×) CoreKV 4× Mode CoreKV 8× Mode
Needle-in-a-Haystack (1K / 2K / 4K / 8K) 100% / 100% / 100% / 100% 100% / 100% / 100% / 100% (20/20) 100% / 100% / 100% / 100% (20/20)
Multi-Needle 3× Simultaneous (2K / 4K) 100.0% / 77.8% 100.0% / 100.0% (18/18) 77.8% / 55.6%
4K Positional Perplexity — Scientific 6.50 6.72 7.90
4K Positional Perplexity — Python Code 5.02 4.05 (beats baseline) 3.96 (beats baseline)
Multi-Hop 3-Hop Reading (~1.5K ctx) 43.8% 100.0% (16/16) 75.0% (12/16)
Multi-Step Reasoning (CoT Math, 20 tasks) 85.0% 95.0% (19/20) 60.0%
Executable Python Unit-Test Pass@1 (16 tasks) 100.0% 93.8% (15/16) 50.0%
High-Concurrency 4K Serving (24 streams, Single GPU) Out of Memory (343 tok/s max @ b=8) 696.0 tok/s (2.03× · 3,456 MB) 434.6+ tok/s (1,728 MB)
High-Concurrency 8K Serving (16 streams, Single GPU) Out of Memory (OOM) 363.4 tok/s (4,608 MB) 229.8+ tok/s (2,304 MB)
Peak Decode Attention Speedup vs. PyTorch SDPA 1.00× 2.70× faster 4.52× faster

Evaluated on unseen scientific archives, held-out Python unit tests, and 1K–8K needle/multi-hop benchmarks on identical single-GPU hardware.

Enterprise Use Cases

Where CoreKV slashes serving infrastructure costs immediately

Built for teams self-hosting open or custom domain-adapted models on cloud GPUs or private on-premise clusters.

Use Case 01 · High-Concurrency SaaS Copilots

Cut Self-Hosted GPU Fleet Bills by 50–75%

Challenge: Multi-tenant enterprise copilots with 8K–32K contexts hit GPU VRAM limits at low batch sizes, forcing ops teams to scale out expensive GPU replicas while compute utilization sits below 30%.

Outcome with CoreKV: Cuts KV cache memory by 4×–8× so each GPU handles up to 3× more simultaneous users, raising aggregate throughput up to 2.03× on the same fleet.

Best fit: AI SaaS platforms, customer chat hosts, inference providers

Use Case 02 · Engineering & Code Assistants

Repository-Scale Coding Without Context Truncation

Challenge: Internal code assistants must hold multi-file diffs, documentation, and compiler traces across hundreds of concurrent engineers without losing syntax or logic accuracy.

Outcome with CoreKV: Compresses KV state 4× while maintaining ultra-sharp unseen code quality (5.42 vs. 7.30 baseline Python PPL) and fast interactive token streaming.

Best fit: Enterprise DevTools, IDE copilots, automated PR review

Use Case 03 · Legal, Clinical & M&A Review

Long-Document Analysis on Compact Private Hardware

Challenge: Privacy-regulated teams reviewing 100-page contracts, M&A data rooms, or clinical charts often need multi-GPU nodes inside air-gapped VPCs solely for long-context KV RAM.

Outcome with CoreKV: Fits high-concurrency long-document serving onto compact single-GPU on-premise appliances with zero external cloud dependencies.

Best fit: Law firms, hospital networks, sovereign & defense VPCs

Use Case 04 · Stateful Autonomous Agents

Persistent Multi-Step Agent Working Memory

Challenge: Background agents running 20–50 tool calls accumulate massive KV histories that stall parallel worker queues or force expensive context recomputation.

Outcome with CoreKV: Keeps dozens of long-horizon agent trajectories resident in VRAM simultaneously, dramatically lowering cost per completed task.

Best fit: Autonomous research, coding, finance & ops agent fleets

Turnkey Integration

Pre-optimized checkpoints or custom conversion of your own models

Deploy our ready-to-serve CoreKV bundles immediately, or let our team retrofit 4×–8× CoreKV compression onto your proprietary enterprise models inside your VPC.

Self-Contained Serving Bundle

Zero architecture lock-in

Each CoreKV delivery ships as a self-contained model directory with its drop-in runtime—ready for single-node PyTorch serving or containerized VPC deployment.

  • 4× High-Fidelity Mode: Near-zero quality loss, 4× KV cut, 1.78×+ max-batch gain
  • 8× Max-Density Mode: 8× KV cut, up to 3.07× faster decode attention, 2.03× max-batch gain
  • Custom Model Conversion: Rapid retrofit onto your private enterprise checkpoints
Enterprise Licensing
# Drop-in self-contained CoreKV serving runtime
from corekv_runtime import load_corekv_model, corekv_session

model, tokenizer, kv_engine = load_corekv_model(
    "./models/enterprise-corekv-4x",
    device="cuda"
)

with corekv_session(model):
    out = model.generate(
        **inputs,
        past_key_values=kv_engine.new_cache(),
        max_new_tokens=512
    )

Next Step

Benchmark CoreKV on your target context length & batch size

Share your target concurrency, context length, and GPU type for a custom throughput & GPU-cost reduction projection.

Request CoreKV Evaluation Back to CoreRank →