Benchmark verified Sep 5, 2026

Adaptive Inference D v001: Preregistered Efficiency Benchmark

KESPA D v001 tested adaptive inference on the frozen 54-case workload. It achieved a composite quality score of 55.2586 at 2.443 s mean latency and 0.153616 Wh mean energy, but failed the preregistered primary success criterion because retrieval Hit@1, gold recall, and escalation-rate thresholds were not met.

adaptive-inference benchmark preregistered compute-efficiency negative-result

MARKDOWN

README.md

3,393 bytes SHA-256 37d4015c69499481…

KESPA-BENCH-002 — Adaptive Inference D v001

Date: 2026-09-05 Status: Verified — user environment Primary preregistered result: FAIL Project: KESPA AI / NexLabs Studios

Research question

Can KESPA's adaptive inference path retain near-best observed answer quality while reducing latency, energy, and model-call cost under a preregistered set of retrieval, efficiency, and escalation constraints?

Frozen workload

D v001 used the same 54-case benchmark family as the prior A/B/C experiment.

The final scoring protocol was:

Frozen A + frozen B + C-envelope with D answer; C slot = D

The blinded evaluator processed 162 answer rows.

Judge:

Qwen/Qwen3.6-35B-A3B

D v001 result

Composite quality: 55.2586
Mean latency: 2443.03 ms
Mean GPU energy: 0.153616 Wh
Retrieval Hit@1: 42/54
Gold micro recall: 59.5238%
Mean LLM calls: 1.0185
Escalation rate: 62.9630%
Quality/sec: 22.618858
Quality/Wh: 359.718120

Same-batch comparison:

A quality: 23.2600
B quality: 56.2379
D quality: 55.2586

So D came very close to B's same-batch composite quality while meeting its latency and energy budgets.

Preregistered checks

| Check | Threshold | Observed | Result | |---|---:|---:|---| | Quality | >= 51.50 | 55.2586 | PASS | | Hit@1 | >= 50/54 | 42/54 | FAIL | | Gold micro recall | >= 75% | 59.5238% | FAIL | | Mean latency | <= 2500 ms | 2443.03 ms | PASS | | Mean energy | <= 0.1600 Wh | 0.153616 Wh | PASS | | Mean inference calls | <= 1.30 | 1.0185 | PASS | | Escalation rate | <= 25% | 62.9630% | FAIL | | Quality/sec | > C | 22.618858 | PASS | | Quality/Wh | > C | 359.718120 | PASS |

Primary result

FAIL.

That result is intentionally preserved rather than reframed as a success.

D v001 passed 6 of 9 preregistered checks, including quality, latency, energy, model-call count, quality/sec, and quality/Wh. But the experiment failed three important operational targets:

retrieval Hit@1;
gold micro recall;
escalation rate.

The 62.963% escalation rate was particularly important because the adaptive design was intended to avoid expensive escalation on most requests.

Why the negative result matters

The run showed that adaptive inference was promising, but the tested routing/retrieval policy was not yet good enough.

It demonstrated a useful separation:

answer quality and compute efficiency could be strong;
retrieval coverage and escalation behavior could still violate the intended architecture.

That is more informative than optimizing the policy until every metric turns green.

Judge integrity

The historical finalization recorded:

162 blinded judgments;
461,717 reported judge tokens;
estimated judge cost $0.140612;
raw results SHA-256

EC4717DA7745532124C8F02BC256501B85C01CAC346A63E837BE89C9E72D028E;

judge prompts exposed A/B/C identity: NO;
raw results modified: NO.

A previously known harness metadata defect left gold_claim_ids empty. Gold association was repaired offline from the locked reference_claim_ids without modifying the raw result file.

Boundary

This benchmark did not modify production Chroma, Brain, or the database.

It is evidence about one frozen adaptive-inference design, not a universal claim about all KESPA routing policies.

TEXT

SHA256SUMS.txt

489 bytes SHA-256 da3139c1f2b80bbb…

37d4015c694994817c66145df929c6e7cb2303411f4a17d6971d0de13eb79c56  README.md
72bf52dbd1da58154324df0ecdedfe8bdf044515ad5cd98db95752a01b0963eb  benchmark.json
51a069992801b798387328afea100a32a12f3881408c193872808927da364352  results.csv
5f71a5d60a194fb057efbc77aa4c5e3fa191b9816c86cacb65101f04b2f9c9df  preregistered_checks.csv
f5cad64b65ba7ff5f1e1b6b5dd3ca5f44024f2a108d9093c34935fa25dcfbdfd  methodology.md
c2503dd51ba75e45a36dd2376dd596df3807747c6163c5600c56121c6019ef01  provenance.json

JSON

benchmark.json

4,001 bytes SHA-256 72bf52dbd1da5815…

{
    "schema": "kespa.public_benchmark.v1",
    "id": "KESPA-BENCH-002",
    "title": "Adaptive Inference D v001: Preregistered Efficiency Benchmark",
    "date": "2026-09-05",
    "status": "verified",
    "run_id": "d_v001_20260905T225726Z",
    "research_question": "Can KESPA's adaptive inference path retain near-best observed answer quality while reducing latency, energy, and model-call cost under a preregistered set of retrieval, efficiency, and escalation constraints?",
    "workload": {
        "cases": 54,
        "scoring_policy": "Frozen A + frozen B + C-envelope with D answer; C slot = D",
        "blinded_judgments": 162,
        "judge_model": "Qwen/Qwen3.6-35B-A3B"
    },
    "adaptive_D": {
        "quality_score": 55.2586,
        "latency_ms": 2443.03,
        "energy_wh": 0.153616,
        "hit_at_1": "42/54",
        "gold_micro_recall_percent": 59.5238,
        "mean_llm_calls": 1.0185185185185186,
        "escalation_rate_percent": 62.963,
        "quality_per_second": 22.618858,
        "quality_per_wh": 359.71812
    },
    "same_batch_comparison": {
        "A_quality": 23.26,
        "B_quality": 56.2379
    },
    "historical_frozen_comparison": {
        "A_quality": 20.71,
        "B_quality": 54.22
    },
    "preregistered_checks": [
        {
            "metric": "quality",
            "threshold": ">= 51.50",
            "observed": 55.2586,
            "result": "PASS"
        },
        {
            "metric": "retrieval_hit_at_1",
            "threshold": ">= 50/54",
            "observed": "42/54",
            "result": "FAIL"
        },
        {
            "metric": "gold_micro_recall",
            "threshold": ">= 75%",
            "observed": "59.5238%",
            "result": "FAIL"
        },
        {
            "metric": "mean_latency",
            "threshold": "<= 2500 ms",
            "observed": "2443.03 ms",
            "result": "PASS"
        },
        {
            "metric": "mean_energy",
            "threshold": "<= 0.1600 Wh",
            "observed": "0.153616 Wh",
            "result": "PASS"
        },
        {
            "metric": "mean_inference_calls",
            "threshold": "<= 1.30",
            "observed": 1.0185185185185186,
            "result": "PASS"
        },
        {
            "metric": "escalation_rate",
            "threshold": "<= 25%",
            "observed": "62.9630%",
            "result": "FAIL"
        },
        {
            "metric": "quality_per_second",
            "threshold": "> C",
            "observed": 22.618858,
            "result": "PASS"
        },
        {
            "metric": "quality_per_wh",
            "threshold": "> C",
            "observed": 359.71812,
            "result": "PASS"
        }
    ],
    "primary_success": "FAIL",
    "judge_run": {
        "reported_judge_tokens": 461717,
        "estimated_judge_cost_usd": 0.140612,
        "raw_result_rows": 162,
        "raw_results_sha256": "EC4717DA7745532124C8F02BC256501B85C01CAC346A63E837BE89C9E72D028E",
        "gold_association_repaired_offline": true,
        "raw_results_modified": false,
        "evaluation_gold_claims_restored": 336,
        "judge_inputs_exposed_arm_identity": false
    },
    "interpretation": [
        "D v001 achieved its preregistered quality, latency, energy, inference-call, quality-per-second, and quality-per-Wh targets.",
        "It did not meet preregistered retrieval Hit@1, gold recall, or escalation-rate thresholds.",
        "Because the primary success rule required the preregistered condition set rather than selective metric wins, the experiment's primary result was FAIL.",
        "The run still demonstrated that an adaptive path could approach the clean verified-retrieval arm's quality while operating substantially faster and with less energy than the earlier always-on orchestration arm."
    ],
    "limitations": [
        "The result is one adaptive-inference experiment on one frozen 54-case workload.",
        "The composite quality score is a benchmark/judge score, not an absolute factual-accuracy percentage.",
        "The quality evaluation used one blinded LLM judge.",
        "A raw-run metadata defect left gold_claim_ids empty; gold associations were repaired offline from locked reference_claim_ids without modifying raw outputs.",
        "The high 62.963% escalation rate means the tested adaptive policy did not achieve the intended low-escalation operating regime."
    ]
}

MARKDOWN

methodology.md

1,384 bytes SHA-256 f5cad64b65ba7ff5…

Methodology

Experimental design

D v001 was evaluated against the frozen KESPA benchmark workload using the same locked evaluation cases and quality-evaluation machinery established for the earlier A/B/C benchmark.

The final comparison used:

Frozen A + frozen B + C-envelope with D answer; C slot = D

This preserved the existing evaluation contract while substituting D's adaptive response into the third scoring slot.

Runtime lock

The historical finalization references a frozen D runtime-result lock and two D execution runs. The final quality run was d_v001_20260905T225726Z.

Quality evaluation

The blinded judge evaluated 162 one-answer inputs.

The historical record states that judge inputs contained no A/B/C labels, runtime metrics, or system identity.

Preregistration

Success was not defined as "quality looks good."

The experiment had explicit thresholds for:

answer quality;
retrieval Hit@1;
gold recall;
latency;
energy;
inference-call count;
escalation rate;
quality per second;
quality per Wh.

The public record preserves the overall primary result as FAIL because not all required behavioral targets were satisfied.

Production isolation

The benchmark finalization recorded:

Production Chroma opened: NO
Production Chroma modified: NO
Brain modified: NO
Database modified: NO
Raw results modified: NO

CSV

preregistered_checks.csv

385 bytes SHA-256 5f71a5d60a194fb0…

metric,threshold,observed,result
quality,>= 51.50,55.2586,PASS
retrieval_hit_at_1,>= 50/54,42/54,FAIL
gold_micro_recall,>= 75%,59.5238%,FAIL
mean_latency,<= 2500 ms,2443.03 ms,PASS
mean_energy,<= 0.1600 Wh,0.153616 Wh,PASS
mean_inference_calls,<= 1.30,1.0185185185185186,PASS
escalation_rate,<= 25%,62.9630%,FAIL
quality_per_second,> C,22.618858,PASS
quality_per_wh,> C,359.71812,PASS

JSON

provenance.json

1,431 bytes SHA-256 c2503dd51ba75e45…

{
    "schema": "kespa.public_provenance.v1",
    "research_id": "KESPA-BENCH-002",
    "source_basis": "Historical KESPA changelog/Jira export supplied for public research reconstruction.",
    "run_id": "d_v001_20260905T225726Z",
    "recorded_artifacts": [
        {
            "path": "benchmark_v002/quality_runs/d_v001_20260905T225726Z/abd_evaluator_run/raw_results.jsonl",
            "recorded_sha256": "EC4717DA7745532124C8F02BC256501B85C01CAC346A63E837BE89C9E72D028E"
        },
        {
            "path": "benchmark_v002/quality_runs/d_v001_20260905T225726Z/d_v001_final_comparison.json",
            "role": "final preregistered comparison"
        },
        {
            "path": "benchmark_v002/runtime_result_locks/d_v001_20260905T225726Z/runtime_result_lock.json",
            "role": "frozen runtime result lock"
        },
        {
            "path": "benchmark_v002/quality_runs/d_v001_20260905T225726Z/abd_scoring_protocol_v001.json",
            "role": "scoring protocol"
        }
    ],
    "hash_verification_note": "The raw-results SHA-256 is transcribed from the historical user-environment finalization record. The underlying benchmark files were not supplied in the current attachment, so this public package does not claim to have independently rehashed them.",
    "publication_note": "Aggregate measurements, preregistered pass/fail outcomes, methodology, and recorded provenance are published. Raw prompts, raw answers, runtime internals, and private source code are not republished."
}

CSV

results.csv

306 bytes SHA-256 51a069992801b798…

system,quality_score,mean_latency_ms,mean_energy_wh,hit_at_1,gold_micro_recall_percent,mean_llm_calls,escalation_rate_percent,quality_per_second,quality_per_wh
A_same_batch,23.2600,,,,,,,,
B_same_batch,56.2379,,,,,,,,
D_v001,55.2586,2443.03,0.153616,42/54,59.5238,1.0185185185,62.9630,22.618858,359.718120