Longinus Research.

01 · Official 5-Dataset Public BEIR (500 Unseen Queries · 10,000 Pairs)

#1 overall across 5 public BEIR retrieval benchmarks

Evaluated on strictly held-out test queries across SciFact (science), NFCorpus (biomedical), FiQA (finance), ArguAna (counter-argument logic), and SciDocs (scientific citations) over identical BM25 Top-20 candidate pools. CoreRank Ultra ranks #1 overall (0.4326 Mean NDCG@10 · 0.5041 MRR@10).

Reranker 5-BEIR Mean SciFact NFCorpus FiQA ArguAna SciDocs Mean MRR@10
CoreRank Ultra0.43260.66500.36800.38240.56140.18620.5041
BGE-Reranker-v2-M30.41790.64890.34270.36110.57940.15750.4790
BGE-Reranker-Large0.36120.64160.34380.35300.31990.14780.4105
MS-MARCO MiniLM-L120.35960.63440.34920.35070.31940.14440.4120
Mixedbread Rerank Large0.35860.67530.35610.36500.23380.16270.4231
BM25 Lexical Baseline0.34340.62140.32920.26030.37000.13620.3830

02 · Enterprise Adversarial Accuracy vs. Latency

6-Vertical Enterprise Accuracy under Distractors (40 Candidates)

Upper-left is better: fast latency with high accuracy even when weaker items contain SEO keyword stuffing, verbose confidence, or angry ALL-CAPS tone.

Adversarial Accuracy vs. Latency (40 Candidates, log scale)

6-vertical mean accuracy under realistic adversarial distractors vs. wall-clock latency.

Adversarial Flip Rate (Lower is Better)

Percentage of rankings inverted when realistic fluff or tone distractors are added to weaker items.

03 · Enterprise Vertical Matrix & Latency Scaling

Head-to-head results across 6 enterprise domains

Filter by vertical to compare clean accuracy, adversarial accuracy, flip rate, and latency scaling from 10 to 100 candidates.

Vertical Filter
CoreRank Pro: 89.7% Clean · 90.3% Adv · 1.8% Flips

Latency Scaling: 10 to 100 Candidates

Measured wall-clock milliseconds on standardized single-GPU hardware.

04 · AI Agent Decisions vs. Category Specialists & 8B Agent LLMs

One unified engine beating dedicated specialists — 3–7× faster than an 8B agent LLM

Benchmarked head-to-head against the best-in-class specialist model in each category (ProtectAI Prompt-Injection, Vectara HHEM-2.1, DeBERTa-v3-Large NLI, ModernBERT-Large NLI, 4B Safety Guard LLM, and an 8B Frontier Open Agent LLM) across 996 unseen evaluation cases spanning 5 real-world agentic datasets.

Unified Agent Decisions Benchmark (ROC-AUC & Accuracy + BFCL Tool Selection)

Datasets: xTRam1/safe-guard-prompt-injection (N=300), deepset/prompt-injections zero-shot transfer (N=116), PatronusAI/HaluBench across FinanceBench/PubMedQA/RAGTruth/DROP/COVID-QA/HaluEval (N=300), Hard Tool-Call Safety (N=80 disguised attacks vs. benign dev commands), and BFCL v3 Multiple tool selection (N=200).

System Scope SafeGuard PI Deepset PI (0-Shot) HaluBench RAG Hard Tool Safety BFCL v3 Choice Avg Latency/Item
CoreRank Max / DecideAll 5 tasks0.9989 (99.3%)0.9539 (65.5%)0.7594 (69.0%)0.9981 (93.8%)100.0%8.6 ms
CoreRank Ultra / DecideAll 5 tasks0.9976 (97.0%)0.9363 (69.8%)0.7216 (64.3%)0.9931 (93.8%)100.0%5.1 ms
8B Frontier Open Agent LLMAll 5 tasks0.9835 (94.0%)0.9406 (69.8%)0.7350 (66.7%)0.9847 (90.0%)99.5%23.5 ms
1.7B Fast Agent LLMAll 5 tasks0.9023 (80.0%)0.9589 (87.1%)0.6116 (50.0%)0.9641 (50.0%)99.5%6.8 ms
4B Dedicated Safety Guard LLMSafety / PI0.9840 (91.3%)0.8073 (75.9%)—0.9869 (50.0%)—23.9 ms
ProtectAI DeBERTa-v3-PI-v2Prompt Injection0.9976 (91.3%)0.8987 (67.2%)—0.7450 (61.3%)—1.3 ms
DeBERTa-v3-Large NLI (MNLI/FEVER)Fact / Choice——0.7073 (64.3%)—90.5%4.8 ms
ModernBERT-Large NLI (8K)Fact / Choice——0.6824 (64.0%)—97.0%3.1 ms
Vectara HHEM-2.1-OpenRAG Hallucination——0.5957 (59.3%)——3.4 ms
CoIR · CosQA Official Benchmark

Coding-Agent Code Search (20,604 Python Functions)

150 unseen developer queries over 20,604 GitHub Python functions (Top-20 BM25 pools). Supports full 32K code context and cuts frontier coding-LLM prompt tokens by 75%+.

System NDCG@10 MRR@10 Recall@5 Context
CoreRank Max (:programming)0.2484—30.7%32K tokens
Mixedbread Rerank Large0.24640.187630.7%512–8K
BGE-Reranker-Large0.24410.190330.7%512–8K
CoreRank Ultra (:programming)0.24170.185330.0%32K tokens
BGE-Reranker-v2-M30.22960.174128.7%512–8K
MS-MARCO MiniLM-L120.22680.174228.0%512–8K
BM25 Lexical Baseline0.21150.163024.0%Lexical
Automatic Domain Routing & MCP

Zero-config routing across all 12 OpenRouter categories

Pass "category": "auto" to automatically detect query domain with 93.3% accuracy on 120 unseen multi-domain queries — including 100% on Programming, Legal, and Science, and 95% on Health.

  • OpenRouter Rerank API: POST /api/v1/rerank with all 12 categories + auto
  • OpenRouter Decisions API: POST /api/alpha/decisions (noul, choice, score)
  • Coding-Agent MCP Integration: Drop-in context filter & tool-call safety gate for Claude Code, Cursor, and Windsurf

05 · Commercial API Pricing & Latency Reference

CoreRank vs. Cohere Rerank 4 Pro, Voyage Rerank 3 & Judge APIs

Published vendor pricing and typical production cloud latency compared against CoreRank Cloud API (OpenRouter compatible) and Self-Hosted VPC deployments.

Product Published Pricing Typical Latency (40 Docs) Custom Domain Rubrics Private VPC Position Bias
CoreRank Self-Hosted VPC < $0.01 / 1k (amortized) 26–57 ms Yes · Adaptive 100% Private VPC 0.0%
CoreRank Cloud / OpenRouter $0.03 / 1M tok (~$0.15 / 1k) ~60–110 ms (incl. RTT) Yes · Adaptive Zero Data Retention 0.0%
Cohere Rerank 4 Pro / 3.5 $2.00–$2.50 / 1k searches ~450–600 ms (p95 cloud) Generic search Enterprise AWS/Azure 0.0%
Voyage / MongoDB Rerank 3 $0.05 / 1M tokens ~280–400 ms (p95 cloud) Instruction-steered Cloud API only 0.0%
Frontier LLM-as-a-Judge API $30–$120+ / 1k lists 2,000–12,000 ms Via prompting Cloud API only 20–50% raw

Sections 01–03 reflect standardized local single-GPU measurements across open-weight baselines and CoreRank tiers. Section 04 reflects published commercial API pricing and typical cloud round-trip latencies.

06 · CoreCompile Agent Hot-Path Benchmarks

Sub-millisecond guardrails, hallucination checks & tool routing

Evaluated across four public enterprise agent benchmarks against standard 16-token generative LLM judging (~234–240 ms/request). Single-span guardrails run on CPU in 0.32 ms (up to 742× faster), while cross-span relational tasks run on GPU in 3.9–8.9 ms (27×–60× faster).

Public Agent Benchmark Test Set High-Accuracy GPU Tier Ultra-Fast CPU Tier Hybrid Auto-Fallback Headline Result
SafeGuard Prompt Injection 300 0.9811 AUC · 4.00 ms (60×) 0.9937 AUC · 0.32 ms (742×) 0.9857 AUC · 0.77 ms (311×) 99.4% AUC @ 0.30 ms
Deepset Prompt Injection 116 0.8726 AUC · 3.90 ms (60×) 0.7610 AUC · 0.28 ms (825×) 0.7839 AUC · 0.81 ms (291×) 89.5% AUC · 35×–166×
HaluBench Hallucination 250 0.8548 AUC · 4.50 ms (53×) 0.5985 AUC · 0.41 ms 0.7297 AUC · 2.07 ms (115×) 85.5% AUC @ 4.5 ms
BFCL v3 Multi-Tool Routing 279 (90 queries) 0.9998 AUC · 8.88 ms (27×) — (cross-span GPU tier) — 100.0% Top-1 Accuracy

CoreCompile automatically assigns lexical/intent guardrails to the Ultra-Fast CPU tier (0.3 ms) and cross-span verification/routing workloads to the High-Accuracy GPU tier (2–5 ms), with automatic fallback to your upstream LLM on low-confidence requests.

07 · CoreKV Long-Context Serving Benchmarks

4×–8× smaller KV memory & up to 3.07× faster decode attention

Standard long-context serving hits GPU memory walls at high batch sizes. CoreKV compresses KV cache memory by 4×–8× while accelerating decode attention up to 3.07× faster than PyTorch SDPA and unlocking 2.03× higher max-batch throughput where uncompressed serving runs out of memory.

Decode Attention Latency vs. PyTorch SDPA

Consistent 1.7×–3.1× acceleration from batch 1 to 16

Batch & Context PyTorch SDPA CoreKV 4× CoreKV 8× Peak Speedup
batch=1, ctx=4,096 49.4 μs 35.5 μs (1.39×) 29.3 μs 1.69× faster
batch=8, ctx=4,096 444.8 μs 193.6 μs (2.30×) 149.4 μs 2.98× faster
batch=8, ctx=8,192 845.2 μs 375.4 μs (2.25×) 289.8 μs 2.92× faster
batch=16, ctx=4,096 890.8 μs 375.7 μs (2.37×) 290.0 μs 3.07× faster
End-to-End Long-Context Decode (4,096 Context)

8× lower KV memory & 2.03× max-batch throughput

Engine Mode KV Memory Throughput Gain
Uncompressed Standard (b=8) 3,584 MB 583.6 tok/s 1.00×
CoreKV 4× Mode (b=8) 896 MB (4× smaller) 546.3 tok/s 0.94×
CoreKV 8× Mode (b=8) 448 MB (8× smaller) 624.9 tok/s 1.07×
CoreKV 8× Mode (b=24)* 1,344 MB 1,187.0 tok/s 2.03×

*Uncompressed serving runs out of GPU memory at batch=24 (ctx=4,096) on identical hardware; CoreKV sustains 24 concurrent streams in 1.34 GB of KV memory.

Long-Context & Multi-Step Verification (Strictly Unseen Tasks)

100% Needle-in-a-Haystack (1K–8K), 100% Multi-Needle & 95% Multi-Step Reasoning at 4× KV Cut

Unseen Long-Context / Multi-Step Benchmark Uncompressed Standard (1×) CoreKV 4× Mode CoreKV 8× Mode
Needle-in-a-Haystack (1K / 2K / 4K / 8K) 100% / 100% / 100% / 100% 100% / 100% / 100% / 100% (20/20) 100% / 100% / 100% / 100% (20/20)
Multi-Needle 3× Simultaneous (2K / 4K) 100.0% / 77.8% 100.0% / 100.0% (18/18) 77.8% / 55.6%
4K Positional Perplexity — Scientific 6.50 6.72 7.90
4K Positional Perplexity — Python Code 5.02 4.05 (beats baseline) 3.96 (beats baseline)
Multi-Hop 3-Hop Reading (~1.5K ctx) 43.8% 100.0% (16/16) 75.0% (12/16)
Multi-Step Reasoning (CoT Math, 20 tasks) 85.0% 95.0% (19/20) 60.0%
Executable Python Unit-Test Pass@1 (16 tasks) 100.0% 93.8% (15/16) 50.0%
High-Concurrency 4K Serving (24 streams, Single GPU) Out of Memory (343 tok/s max @ b=8) 696.0 tok/s (2.03× · 3,456 MB) 434.6+ tok/s (1,728 MB)
High-Concurrency 8K Serving (16 streams, Single GPU) Out of Memory (OOM) 363.4 tok/s (4,608 MB) 229.8+ tok/s (2,304 MB)
Peak Decode Attention Speedup vs. PyTorch SDPA 1.00× 2.70× faster 4.52× faster

Evaluated on unseen scientific archives, held-out Python unit tests, and 1K–8K needle/multi-hop benchmarks on identical single-GPU hardware.