MARKDOWN
README.md
3,653 bytes
SHA-256 838718d0d9c8450e…
KESPA-BENCH-001 — A/B/C Verified-Intelligence Quality and Compute Benchmark
Date: 2026-09-04 Status: Verified — user environment Project: KESPA AI / NexLabs Studios
Research question
Under the same frozen evaluation workload, how do three approaches compare?
- A — Raw local 7B
- B — Clean verified atomic retrieval
- C — Then-current KESPA orchestration / legacy retrieval path
The benchmark was designed to test whether KESPA mechanisms materially improve useful small-model output per unit compute rather than assuming that more orchestration or more stored data is automatically better.
Frozen evaluation
- 54 locked evaluation questions
- 3 arms
- 162 system executions
- 162 blinded quality judgments
- clean corpus: 423 verified atomic claims
- linked frozen evidence: 56 excerpts
All three runtime arms completed 54/54 cases.
The quality judge was blinded to A/B/C identity, runtime metrics, and system identity until scoring was complete.
Results
| Metric | A — Raw local 7B | B — Verified retrieval | C — KESPA orchestration | |---|---:|---:|---:| | Composite quality | 20.71 | 54.22 | 27.71 | | Mean latency | 3.757 s | 1.675 s | 9.066 s | | Mean GPU energy | 0.2450 Wh | 0.1085 Wh | 0.5759 Wh | | Retrieval Hit@1 | — | 54/54 | 27/54 | | Gold retrieval recall | — | 79.17% | 22.02% | | Case-quality wins | 2 | 39 | 3 |
Ties: 10
Within this benchmark, Arm B's composite quality score was about 2.62× Arm A's score while mean latency was about 55.4% lower and mean GPU energy about 55.7% lower.
Those ratios describe this benchmark only. They are not claims that KESPA made the language model itself 2.62× more intelligent.
Retrieval policy was frozen before evaluation
A separate 12-question calibration set was used before the 54 locked evaluation questions.
The final policy freeze selected plain_vector_top6:
- Hit@1: 12/12
- claim recall: 67/87
- retrieval precision: 93.1%
- mean latency: 22.74 ms
The comparison atomic ranking path produced:
- Hit@1: 12/12
- claim recall: 66/87
- retrieval precision: 91.7%
- mean latency: 49.89 ms
The locked 54-question evaluation set was not parsed during that calibration.
Important benchmark repair
The raw execution completed successfully, but a harness metadata defect omitted reference_claim_ids from the gold-ID extraction path. As a result, all 162 raw rows contained empty gold_claim_ids.
The repair was performed offline using the already locked reference_claim_ids. The original raw_results.jsonl was left unchanged.
This matters because the retrieval and quality evaluation could be repaired without rerunning or post-hoc tuning the model outputs.
Finding
The strongest result was not simply that retrieval helped.
It was that clean, verified, atomic retrieval outperformed both the raw-model path and the more expensive orchestration path on the reported composite quality measure while also requiring less latency and GPU energy.
The benchmark therefore motivated KESPA V2's emphasis on:
- verified externalized intelligence;
- trusted-retrieval fast paths;
- selective rather than always-on orchestration;
- compute-aware escalation.
Limitations
This is a controlled system benchmark, not a universal model leaderboard.
It used one frozen corpus/workload, one local-model/hardware configuration, and one evidence-bound LLM judge. Composite quality scores are benchmark scores and must not be interpreted as absolute factual-accuracy percentages.
TEXT
SHA256SUMS.txt
398 bytes
SHA-256 5463680767021c06…
838718d0d9c8450ed442c3b9f28c82a377ea3501ace0cc79266e81cce97f2bf8 README.md
17aed1ab63c862ae27df58afb8fc4d64d9b390058acbe13ed26fbba95b5d65de benchmark.json
8f25ade266d46e8ab5c5d0aa5433a56e7db280732cfd91219ca6421288a2e0b5 results.csv
cf9af82e91237d9e44a75fc03aa263dd4832d64ff894cec6ea385037560fdee7 methodology.md
593f9249159edf072da5e86c4027c110f702a3789ec26d20d154ad082b282588 provenance.json
JSON
benchmark.json
3,588 bytes
SHA-256 17aed1ab63c862ae…
{
"schema": "kespa.public_benchmark.v1",
"id": "KESPA-BENCH-001",
"title": "A/B/C Verified-Intelligence Quality and Compute Benchmark",
"date": "2026-09-04",
"status": "verified",
"research_question": "Under the same frozen evaluation workload, how do a raw local 7B path, clean verified atomic retrieval, and the then-current KESPA orchestration compare on judged answer quality, retrieval, latency, and GPU energy?",
"design": {
"locked_evaluation_questions": 54,
"arms": 3,
"system_executions": 162,
"blinded_quality_judgments": 162,
"clean_verified_claims": 423,
"clean_verified_evidence_excerpts": 56,
"quality_judging": "blinded; system identity hidden until scoring completed",
"production_state_changed": false
},
"arms": {
"A": {
"description": "Raw local 7B",
"quality": 20.71,
"latency_ms": 3757,
"gpu_energy_wh": 0.245,
"case_wins": 2
},
"B": {
"description": "Clean verified atomic retrieval",
"quality": 54.22,
"latency_ms": 1675,
"gpu_energy_wh": 0.1085,
"case_wins": 39,
"hit_at_1": "54/54",
"gold_recall_percent": 79.17
},
"C": {
"description": "Then-current KESPA orchestration / legacy retrieval path",
"quality": 27.71,
"latency_ms": 9066,
"gpu_energy_wh": 0.5759,
"case_wins": 3,
"hit_at_1": "27/54",
"gold_recall_percent": 22.02
}
},
"case_quality_outcomes": {
"A_wins": 2,
"B_wins": 39,
"C_wins": 3,
"ties": 10
},
"derived_comparisons": {
"B_quality_score_ratio_vs_A": 2.618,
"B_latency_reduction_vs_A_percent": 55.4,
"B_energy_reduction_vs_A_percent": 55.7,
"B_latency_reduction_vs_C_percent": 81.5,
"B_energy_reduction_vs_C_percent": 81.2
},
"prebenchmark_retrieval_policy_freeze": {
"calibration_questions": 12,
"selected_policy": "plain_vector_top6",
"selected_policy_hit_at_1": "12/12",
"selected_policy_claim_recall": "67/87",
"selected_policy_precision_percent": 93.1,
"selected_policy_mean_latency_ms": 22.74,
"comparison_policy_claim_recall": "66/87",
"comparison_policy_precision_percent": 91.7,
"comparison_policy_mean_latency_ms": 49.89,
"evaluation_questions_used_during_calibration": 0
},
"judge_cost_usd": 0.144283,
"interpretation": [
"Within this frozen workload, the clean verified-retrieval arm produced the strongest reported composite quality score while also using less latency and GPU energy than the raw-model and then-current KESPA orchestration arms.",
"The result supports the hypothesis that knowledge quality and retrieval policy can materially affect useful output and compute cost even when the underlying local model remains small.",
"The benchmark also showed that additional orchestration was not automatically beneficial: the then-current KESPA path was slower and more energy-intensive than the verified-retrieval fast path."
],
"limitations": [
"The composite quality score is a benchmark/judge score, not an absolute factual-accuracy percentage.",
"The result covers one verified corpus/workload and one local-model/hardware setup.",
"Quality evaluation used one evidence-bound LLM judge.",
"A benchmark metadata defect left gold_claim_ids empty in the raw rows; gold association was repaired offline from locked reference_claim_ids without modifying raw_results.jsonl.",
"The benchmark does not establish that verified RAG will outperform every model, corpus, workload, or orchestration design."
]
}
MARKDOWN
methodology.md
1,568 bytes
SHA-256 cf9af82e91237d9e…
Methodology
Corpus
The benchmark used a frozen clean corpus containing:
- 423 verified atomic claims;
- 56 linked evidence excerpts.
The isolated benchmark Chroma collection was validated independently before the benchmark. The source record states that 423 IDs/documents/metadata were exact and that retrieval equivalence was confirmed after allowing harmless float-round-trip differences in embeddings.
Question split
The benchmark question builder created:
- 66 total questions;
- 12 calibration questions;
- 54 locked evaluation questions.
The calibration set was used to freeze retrieval policy without parsing the locked evaluation questions.
Runtime comparison
The 54 locked questions were executed against each of three arms:
- A — raw local 7B;
- B — clean verified atomic retrieval;
- C — then-current KESPA orchestration / legacy retrieval path.
This produced 162 raw system outputs.
Blinded quality scoring
A standalone quality scorer preserved the original raw benchmark outputs.
The historical record states that:
- frozen corpus/evaluation hashes were verified;
- 162/162 quality judgments completed;
- system identity was hidden during blinded judging;
- a missing-gold-ID metadata defect was repaired offline from locked reference IDs;
raw_results.jsonl was not modified.
Production isolation
The benchmark did not modify production Chroma, Brain API behavior, or database state. Private memory, web/cloud retrieval, and trust callbacks were disabled for the benchmark path described in the source record.
JSON
provenance.json
4,117 bytes
SHA-256 593f9249159edf07…
{
"schema": "kespa.public_provenance.v1",
"research_id": "KESPA-BENCH-001",
"source_basis": "Historical KESPA changelog/Jira export supplied for public research reconstruction.",
"recorded_artifact_hashes": [
{
"path": "verified_corpus/clean_verified_claim_corpus_v001/manifest.json",
"recorded_sha256": "48764DDEDB2D094DCD621E030321653C557494C872EE9ADAC405FD58022FAAA9"
},
{
"path": "verified_corpus/clean_verified_claim_corpus_v001/validation_summary.txt",
"recorded_sha256": "30F75EC56D22C811CFC171F31F1689AF7126C3DE6E9B12F9108EA672C77A91EA"
},
{
"path": "verified_corpus/clean_verified_claim_corpus_v001/verified_claims.jsonl",
"recorded_sha256": "D0BD8E02484472263A8D38E82023DD8CBEFC4B70AC86BA31713E04A63044352B"
},
{
"path": "verified_corpus/clean_verified_claim_corpus_v001/verified_evidence.jsonl",
"recorded_sha256": "0C181ED53224EE5E14A09037A49840F3D3A171C7CFF8322E090C3C64ABF13C4A"
},
{
"path": "benchmark_v001/benchmark_calibration_v001.jsonl",
"recorded_sha256": "366CA6A74CE2EDE4ABAF2DC9EE4641241AC5E57BFF99BCB1319415626A3BC60C"
},
{
"path": "benchmark_v001/benchmark_evaluation_v001.jsonl",
"recorded_sha256": "0EE67DE597B793A7A7665558AA97382494989A681376C59454ED4257455ACC62"
},
{
"path": "benchmark_v001/benchmark_question_manifest_v001.json",
"recorded_sha256": "D319B7F8E3ED9379ECAEFE47E4F0C3DEDE78F57450390C054B87AD3D757A74F7"
},
{
"path": "benchmark_v001/benchmark_questions_v001.jsonl",
"recorded_sha256": "D10807FCE5C049C991B5AAA4611D1D6D30D2EE61A526B4874DFE05D1F5BB7DBC"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/gold_repair_summary_v001.json",
"recorded_sha256": "3E3DE2E1C8B56EEC9906F696207E5C6F439DEC7F3389D24236FC869BFB033C06"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/judge_rubric_v001.txt",
"recorded_sha256": "435A800ABF05280760DFD5C18A8D5BB9A592F37FD367118B34D28E7FE96CEA0A"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_metrics_v001.csv",
"recorded_sha256": "0289FFEFB818EE8E9A7A797033AB10C9C3CCE2C03B79FD31D226E4F11ABD8033"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_results_blinded_v001.jsonl",
"recorded_sha256": "423DF203072E5A16BCCEE9B5CDBFB156EB71AF7D3DA2022287153ECBA267BF59"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_results_unblinded_v001.jsonl",
"recorded_sha256": "2F80C31C5055B14F61A13A021E0097F2CB770C75C37B407ADF35F5F6BF17FA6F"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_summary_v001.json",
"recorded_sha256": "590CE771F4858CFA4FA34210E7921BD9F7D0F3268217FFDD4B08C3C228EC0265"
},
{
"path": "runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_summary_v001.txt",
"recorded_sha256": "767414A5AF94F72D4383FC524F3795701427AFCC94768B3BF9F979CF1C2A5961"
},
{
"path": "runs/abc_v001_20260904T174419Z/raw_results.jsonl",
"recorded_sha256": "C9CD588A1308D58710680A536C9E2509315B7B122A1B574357E6B6C2675C3049"
},
{
"path": "runs/abc_v001_20260904T174419Z/raw_summary.json",
"recorded_sha256": "9E320CEE7437F558F042DC9933480B11818F38E1AFDCC74EBEE59162331945CE"
},
{
"path": "runs/abc_v001_20260904T174419Z/run_manifest.json",
"recorded_sha256": "6D80070BA8F8CE134CD7B441EB1751143E6C91BB5AF068737DDA9FB5A713D4B7"
}
],
"hash_verification_note": "These SHA-256 values are transcribed from the historical source record. The underlying benchmark artifacts were not included in the supplied changelog attachment, so this public package does not claim to have independently rehashed them.",
"publication_note": "The public package publishes aggregate benchmark measurements, methodology, limitations, and recorded provenance hashes. Raw prompts, raw model answers, private implementation code, and internal repository contents are not republished."
}
CSV
results.csv
302 bytes
SHA-256 8f25ade266d46e8a…
arm,description,quality_score,mean_latency_ms,gpu_energy_wh,hit_at_1,gold_recall_percent,case_quality_wins
A,Raw local 7B,20.71,3757,0.2450,,,2
B,Clean verified atomic retrieval,54.22,1675,0.1085,54/54,79.17,39
C,Then-current KESPA orchestration / legacy retrieval path,27.71,9066,0.5759,27/54,22.02,3