01 · Official 5-Dataset Public BEIR (500 Unseen Queries · 10,000 Pairs)
#1 overall across 5 public BEIR retrieval benchmarks
Evaluated on strictly held-out test queries across SciFact (science), NFCorpus (biomedical), FiQA (finance), ArguAna (counter-argument logic), and SciDocs (scientific citations) over identical BM25 Top-20 candidate pools. CoreRank Ultra ranks #1 overall (0.4326 Mean NDCG@10 · 0.5041 MRR@10).
| Reranker | 5-BEIR Mean | SciFact | NFCorpus | FiQA | ArguAna | SciDocs | Mean MRR@10 |
|---|---|---|---|---|---|---|---|
| CoreRank Ultra | 0.4326 | 0.6650 | 0.3680 | 0.3824 | 0.5614 | 0.1862 | 0.5041 |
| BGE-Reranker-v2-M3 | 0.4179 | 0.6489 | 0.3427 | 0.3611 | 0.5794 | 0.1575 | 0.4790 |
| BGE-Reranker-Large | 0.3612 | 0.6416 | 0.3438 | 0.3530 | 0.3199 | 0.1478 | 0.4105 |
| MS-MARCO MiniLM-L12 | 0.3596 | 0.6344 | 0.3492 | 0.3507 | 0.3194 | 0.1444 | 0.4120 |
| Mixedbread Rerank Large | 0.3586 | 0.6753 | 0.3561 | 0.3650 | 0.2338 | 0.1627 | 0.4231 |
| BM25 Lexical Baseline | 0.3434 | 0.6214 | 0.3292 | 0.2603 | 0.3700 | 0.1362 | 0.3830 |
02 · Enterprise Adversarial Accuracy vs. Latency
6-Vertical Enterprise Accuracy under Distractors (40 Candidates)
Upper-left is better: fast latency with high accuracy even when weaker items contain SEO keyword stuffing, verbose confidence, or angry ALL-CAPS tone.
Adversarial Accuracy vs. Latency (40 Candidates, log scale)
6-vertical mean accuracy under realistic adversarial distractors vs. wall-clock latency.
Adversarial Flip Rate (Lower is Better)
Percentage of rankings inverted when realistic fluff or tone distractors are added to weaker items.
03 · Enterprise Vertical Matrix & Latency Scaling
Head-to-head results across 6 enterprise domains
Filter by vertical to compare clean accuracy, adversarial accuracy, flip rate, and latency scaling from 10 to 100 candidates.
Latency Scaling: 10 to 100 Candidates
Measured wall-clock milliseconds on standardized single-GPU hardware.
04 · AI Agent Decisions vs. Category Specialists & 8B Agent LLMs
One unified engine beating dedicated specialists — 3–7× faster than an 8B agent LLM
Benchmarked head-to-head against the best-in-class specialist model in each category (ProtectAI Prompt-Injection, Vectara HHEM-2.1, DeBERTa-v3-Large NLI, ModernBERT-Large NLI, 4B Safety Guard LLM, and an 8B Frontier Open Agent LLM) across 996 unseen evaluation cases spanning 5 real-world agentic datasets.
Unified Agent Decisions Benchmark (ROC-AUC & Accuracy + BFCL Tool Selection)
Datasets: xTRam1/safe-guard-prompt-injection (N=300), deepset/prompt-injections zero-shot transfer (N=116), PatronusAI/HaluBench across FinanceBench/PubMedQA/RAGTruth/DROP/COVID-QA/HaluEval (N=300), Hard Tool-Call Safety (N=80 disguised attacks vs. benign dev commands), and BFCL v3 Multiple tool selection (N=200).
| System | Scope | SafeGuard PI | Deepset PI (0-Shot) | HaluBench RAG | Hard Tool Safety | BFCL v3 Choice | Avg Latency/Item |
|---|---|---|---|---|---|---|---|
| CoreRank Max / Decide | All 5 tasks | 0.9989 (99.3%) | 0.9539 (65.5%) | 0.7594 (69.0%) | 0.9981 (93.8%) | 100.0% | 8.6 ms |
| CoreRank Ultra / Decide | All 5 tasks | 0.9976 (97.0%) | 0.9363 (69.8%) | 0.7216 (64.3%) | 0.9931 (93.8%) | 100.0% | 5.1 ms |
| 8B Frontier Open Agent LLM | All 5 tasks | 0.9835 (94.0%) | 0.9406 (69.8%) | 0.7350 (66.7%) | 0.9847 (90.0%) | 99.5% | 23.5 ms |
| 1.7B Fast Agent LLM | All 5 tasks | 0.9023 (80.0%) | 0.9589 (87.1%) | 0.6116 (50.0%) | 0.9641 (50.0%) | 99.5% | 6.8 ms |
| 4B Dedicated Safety Guard LLM | Safety / PI | 0.9840 (91.3%) | 0.8073 (75.9%) | — | 0.9869 (50.0%) | — | 23.9 ms |
| ProtectAI DeBERTa-v3-PI-v2 | Prompt Injection | 0.9976 (91.3%) | 0.8987 (67.2%) | — | 0.7450 (61.3%) | — | 1.3 ms |
| DeBERTa-v3-Large NLI (MNLI/FEVER) | Fact / Choice | — | — | 0.7073 (64.3%) | — | 90.5% | 4.8 ms |
| ModernBERT-Large NLI (8K) | Fact / Choice | — | — | 0.6824 (64.0%) | — | 97.0% | 3.1 ms |
| Vectara HHEM-2.1-Open | RAG Hallucination | — | — | 0.5957 (59.3%) | — | — | 3.4 ms |
Coding-Agent Code Search (20,604 Python Functions)
150 unseen developer queries over 20,604 GitHub Python functions (Top-20 BM25 pools). Supports full 32K code context and cuts frontier coding-LLM prompt tokens by 75%+.
| System | NDCG@10 | MRR@10 | Recall@5 | Context |
|---|---|---|---|---|
| CoreRank Max (:programming) | 0.2484 | — | 30.7% | 32K tokens |
| Mixedbread Rerank Large | 0.2464 | 0.1876 | 30.7% | 512–8K |
| BGE-Reranker-Large | 0.2441 | 0.1903 | 30.7% | 512–8K |
| CoreRank Ultra (:programming) | 0.2417 | 0.1853 | 30.0% | 32K tokens |
| BGE-Reranker-v2-M3 | 0.2296 | 0.1741 | 28.7% | 512–8K |
| MS-MARCO MiniLM-L12 | 0.2268 | 0.1742 | 28.0% | 512–8K |
| BM25 Lexical Baseline | 0.2115 | 0.1630 | 24.0% | Lexical |
Zero-config routing across all 12 OpenRouter categories
Pass "category": "auto" to automatically detect query domain with 93.3% accuracy on 120 unseen multi-domain queries — including 100% on Programming, Legal, and Science, and 95% on Health.
- OpenRouter Rerank API:
POST /api/v1/rerankwith all 12 categories +auto - OpenRouter Decisions API:
POST /api/alpha/decisions(noul,choice,score) - Coding-Agent MCP Integration: Drop-in context filter & tool-call safety gate for Claude Code, Cursor, and Windsurf
05 · Commercial API Pricing & Latency Reference
CoreRank vs. Cohere Rerank 4 Pro, Voyage Rerank 3 & Judge APIs
Published vendor pricing and typical production cloud latency compared against CoreRank Cloud API (OpenRouter compatible) and Self-Hosted VPC deployments.
| Product | Published Pricing | Typical Latency (40 Docs) | Custom Domain Rubrics | Private VPC | Position Bias |
|---|---|---|---|---|---|
| CoreRank Self-Hosted VPC | < $0.01 / 1k (amortized) | 26–57 ms | Yes · Adaptive | 100% Private VPC | 0.0% |
| CoreRank Cloud / OpenRouter | $0.03 / 1M tok (~$0.15 / 1k) | ~60–110 ms (incl. RTT) | Yes · Adaptive | Zero Data Retention | 0.0% |
| Cohere Rerank 4 Pro / 3.5 | $2.00–$2.50 / 1k searches | ~450–600 ms (p95 cloud) | Generic search | Enterprise AWS/Azure | 0.0% |
| Voyage / MongoDB Rerank 3 | $0.05 / 1M tokens | ~280–400 ms (p95 cloud) | Instruction-steered | Cloud API only | 0.0% |
| Frontier LLM-as-a-Judge API | $30–$120+ / 1k lists | 2,000–12,000 ms | Via prompting | Cloud API only | 20–50% raw |
Sections 01–03 reflect standardized local single-GPU measurements across open-weight baselines and CoreRank tiers. Section 04 reflects published commercial API pricing and typical cloud round-trip latencies.
06 · CoreCompile Agent Hot-Path Benchmarks
Sub-millisecond guardrails, hallucination checks & tool routing
Evaluated across four public enterprise agent benchmarks against standard 16-token generative LLM judging (~234–240 ms/request). Single-span guardrails run on CPU in 0.32 ms (up to 742× faster), while cross-span relational tasks run on GPU in 3.9–8.9 ms (27×–60× faster).
| Public Agent Benchmark | Test Set | High-Accuracy GPU Tier | Ultra-Fast CPU Tier | Hybrid Auto-Fallback | Headline Result |
|---|---|---|---|---|---|
| SafeGuard Prompt Injection | 300 | 0.9811 AUC · 4.00 ms (60×) | 0.9937 AUC · 0.32 ms (742×) | 0.9857 AUC · 0.77 ms (311×) | 99.4% AUC @ 0.30 ms |
| Deepset Prompt Injection | 116 | 0.8726 AUC · 3.90 ms (60×) | 0.7610 AUC · 0.28 ms (825×) | 0.7839 AUC · 0.81 ms (291×) | 89.5% AUC · 35×–166× |
| HaluBench Hallucination | 250 | 0.8548 AUC · 4.50 ms (53×) | 0.5985 AUC · 0.41 ms | 0.7297 AUC · 2.07 ms (115×) | 85.5% AUC @ 4.5 ms |
| BFCL v3 Multi-Tool Routing | 279 (90 queries) | 0.9998 AUC · 8.88 ms (27×) | — (cross-span GPU tier) | — | 100.0% Top-1 Accuracy |
CoreCompile automatically assigns lexical/intent guardrails to the Ultra-Fast CPU tier (0.3 ms) and cross-span verification/routing workloads to the High-Accuracy GPU tier (2–5 ms), with automatic fallback to your upstream LLM on low-confidence requests.
07 · CoreKV Long-Context Serving Benchmarks
4×–8× smaller KV memory & up to 3.07× faster decode attention
Standard long-context serving hits GPU memory walls at high batch sizes. CoreKV compresses KV cache memory by 4×–8× while accelerating decode attention up to 3.07× faster than PyTorch SDPA and unlocking 2.03× higher max-batch throughput where uncompressed serving runs out of memory.
Consistent 1.7×–3.1× acceleration from batch 1 to 16
| Batch & Context | PyTorch SDPA | CoreKV 4× | CoreKV 8× | Peak Speedup |
|---|---|---|---|---|
| batch=1, ctx=4,096 | 49.4 μs | 35.5 μs (1.39×) | 29.3 μs | 1.69× faster |
| batch=8, ctx=4,096 | 444.8 μs | 193.6 μs (2.30×) | 149.4 μs | 2.98× faster |
| batch=8, ctx=8,192 | 845.2 μs | 375.4 μs (2.25×) | 289.8 μs | 2.92× faster |
| batch=16, ctx=4,096 | 890.8 μs | 375.7 μs (2.37×) | 290.0 μs | 3.07× faster |
8× lower KV memory & 2.03× max-batch throughput
| Engine Mode | KV Memory | Throughput | Gain |
|---|---|---|---|
| Uncompressed Standard (b=8) | 3,584 MB | 583.6 tok/s | 1.00× |
| CoreKV 4× Mode (b=8) | 896 MB (4× smaller) | 546.3 tok/s | 0.94× |
| CoreKV 8× Mode (b=8) | 448 MB (8× smaller) | 624.9 tok/s | 1.07× |
| CoreKV 8× Mode (b=24)* | 1,344 MB | 1,187.0 tok/s | 2.03× |
*Uncompressed serving runs out of GPU memory at batch=24 (ctx=4,096) on identical hardware; CoreKV sustains 24 concurrent streams in 1.34 GB of KV memory.
100% Needle-in-a-Haystack (1K–8K), 100% Multi-Needle & 95% Multi-Step Reasoning at 4× KV Cut
| Unseen Long-Context / Multi-Step Benchmark | Uncompressed Standard (1×) | CoreKV 4× Mode | CoreKV 8× Mode |
|---|---|---|---|
| Needle-in-a-Haystack (1K / 2K / 4K / 8K) | 100% / 100% / 100% / 100% | 100% / 100% / 100% / 100% (20/20) | 100% / 100% / 100% / 100% (20/20) |
| Multi-Needle 3× Simultaneous (2K / 4K) | 100.0% / 77.8% | 100.0% / 100.0% (18/18) | 77.8% / 55.6% |
| 4K Positional Perplexity — Scientific | 6.50 | 6.72 | 7.90 |
| 4K Positional Perplexity — Python Code | 5.02 | 4.05 (beats baseline) | 3.96 (beats baseline) |
| Multi-Hop 3-Hop Reading (~1.5K ctx) | 43.8% | 100.0% (16/16) | 75.0% (12/16) |
| Multi-Step Reasoning (CoT Math, 20 tasks) | 85.0% | 95.0% (19/20) | 60.0% |
| Executable Python Unit-Test Pass@1 (16 tasks) | 100.0% | 93.8% (15/16) | 50.0% |
| High-Concurrency 4K Serving (24 streams, Single GPU) | Out of Memory (343 tok/s max @ b=8) | 696.0 tok/s (2.03× · 3,456 MB) | 434.6+ tok/s (1,728 MB) |
| High-Concurrency 8K Serving (16 streams, Single GPU) | Out of Memory (OOM) | 363.4 tok/s (4,608 MB) | 229.8+ tok/s (2,304 MB) |
| Peak Decode Attention Speedup vs. PyTorch SDPA | 1.00× | 2.70× faster | 4.52× faster |
Evaluated on unseen scientific archives, held-out Python unit tests, and 1K–8K needle/multi-hop benchmarks on identical single-GPU hardware.