Benchmark verified Sep 4, 2026

Atomic Knowledge Retrieval Policy Calibration

A locked 12-question calibration showed that legacy long-card retrieval filters were poorly matched to short verified atomic claims. Plain vector top-6 achieved 12/12 Hit@1 and 67/87 claim recall, while legacy minimum-length filtering produced severe retrieval loss; the final policy freeze selected plain_vector_top6.

retrieval calibration atomic-claims vector-search policy-freeze

MARKDOWN

README.md

3,107 bytes SHA-256 d74f26c5a705f024…

KESPA-BENCH-003 — Atomic Knowledge Retrieval Policy Calibration

Date: 2026-09-04 Status: Verified — user environment Project: KESPA AI / NexLabs Studios

Research question

Do retrieval safeguards designed for long synthetic/legacy cards remain appropriate after KESPA moves to short, strictly verified atomic claims?

Context

KESPA had moved to an isolated corpus of 423 verified atomic claims.

Before the locked 54-question benchmark could be run, retrieval policy had to be calibrated using a separate 12-question calibration set.

The 54 locked evaluation questions were hash-verified but not parsed during calibration.

Initial calibration

Plain vector top-6

Hit@1: 12/12
Hit@3: 12/12
Hit@6: 12/12
empty retrievals: 0/12
claim recall: 67/87 (77.0%)
retrieval precision: 93.1%
mean question recall: 80.6%
MRR: 1.0000
average returned: 6.00
mean latency: 29.15 ms

Legacy KESPA minimum-length policy (`min150`)

Hit@1: 4/12
empty retrievals: 8/12
claim recall: 10/87 (11.5%)
retrieval precision: 100%
MRR: 0.3333
average returned: 0.83
mean latency: 60.03 ms

Atomic-compatible reduced floor (`min40`)

Hit@1: 8/12
empty retrievals: 4/12
claim recall: 21/87 (24.1%)
retrieval precision: 100%
MRR: 0.6667
average returned: 1.75
mean latency: 45.61 ms

Why the legacy policy failed

A separate compatibility validation found that the old 150-character minimum length floor excluded 260 of 423 verified claims.

That filter had made sense as a defensive heuristic against weak/short legacy cards. After KESPA moved to deliberately atomic verified claims, it became destructive.

This is an important architecture lesson:

A quality heuristic can become a quality defect when the knowledge unit changes.

Final policy freeze

The final 12-question policy comparison selected:

plain_vector_top6

Recorded final comparison:

| Policy | Hit@1 | Claim recall | Precision | Mean latency | |---|---:|---:|---:|---:| | Plain vector top-6 | 12/12 | 67/87 | 93.1% | 22.74 ms | | Atomic-compatible KESPA ranking | 12/12 | 66/87 | 91.7% | 49.89 ms |

The more complex ranking path matched Hit@1, but did not equal plain-vector claim recall and was more than twice as slow in the final recorded policy-freeze comparison.

KESPA therefore froze plain_vector_top6 for the later A/B/C evaluation.

Supporting index validation

Before policy freeze, isolated benchmark Chroma compatibility was checked:

IDs/documents/metadata exact: 423/423
top-1/top-10 retrieval-equivalence checks: 32/32
43/423 embeddings differed only by float round-trip precision
maximum observed embedding delta: 1.49e-08

Those numerical differences were treated as harmless rather than as corpus corruption.

Boundary

No production Chroma, Brain API, or database state changed during this work.

The purpose was calibration and policy selection for a frozen research benchmark.

TEXT

SHA256SUMS.txt

398 bytes SHA-256 73262320f766abc8…

d74f26c5a705f0242a3e705895f1e6aa33ab13087d08e96369c689836218aebc  README.md
cd0e204dc8a32a945b113776b7683b770fbcbc22dfbbfb584021e2e53c2e0fcf  benchmark.json
99615a340a0893186c293bec7c308f696f9f069786c28697b193a9b9337a21fe  results.csv
7d76b89117de261aee4dbd150351017be03b2e59b045790c53174b01bb53a92e  methodology.md
88fbac001a0d2ff2d0b88371494a4de61f1424f3611f941cea0526f20e2d58e1  provenance.json

JSON

benchmark.json

3,798 bytes SHA-256 cd0e204dc8a32a94…

{
    "schema": "kespa.public_benchmark.v1",
    "id": "KESPA-BENCH-003",
    "title": "Atomic Knowledge Retrieval Policy Calibration",
    "date": "2026-09-04",
    "status": "verified",
    "research_question": "Do retrieval safeguards designed for long synthetic/legacy cards remain appropriate after KESPA moves to short, strictly verified atomic claims?",
    "fixed_context": {
        "calibration_questions": 12,
        "locked_evaluation_questions": 54,
        "evaluation_questions_parsed_during_calibration": 0,
        "isolated_verified_claims": 423,
        "embedding_model": "all-MiniLM-L6-v2",
        "production_chroma_modified": false,
        "brain_modified": false,
        "database_modified": false
    },
    "initial_calibration": {
        "plain_rag_top6": {
            "hit_at_1": "12/12",
            "hit_at_3": "12/12",
            "hit_at_6": "12/12",
            "empty_retrievals": "0/12",
            "claim_recall": "67/87",
            "claim_recall_percent": 77,
            "retrieval_precision_percent": 93.1,
            "mean_question_recall_percent": 80.6,
            "mrr": 1,
            "avg_returned": 6,
            "avg_latency_ms": 29.15
        },
        "forge_current_min150": {
            "hit_at_1": "4/12",
            "hit_at_3": "4/12",
            "hit_at_6": "4/12",
            "empty_retrievals": "8/12",
            "claim_recall": "10/87",
            "claim_recall_percent": 11.5,
            "retrieval_precision_percent": 100,
            "mean_question_recall_percent": 12.3,
            "mrr": 0.3333,
            "avg_returned": 0.83,
            "avg_latency_ms": 60.03
        },
        "forge_atomic_min40": {
            "hit_at_1": "8/12",
            "hit_at_3": "8/12",
            "hit_at_6": "8/12",
            "empty_retrievals": "4/12",
            "claim_recall": "21/87",
            "claim_recall_percent": 24.1,
            "retrieval_precision_percent": 100,
            "mean_question_recall_percent": 28,
            "mrr": 0.6667,
            "avg_returned": 1.75,
            "avg_latency_ms": 45.61
        }
    },
    "final_policy_freeze": {
        "selected_policy": "plain_vector_top6",
        "plain_vector_top6": {
            "hit_at_1": "12/12",
            "claim_recall": "67/87",
            "claim_recall_percent": 77,
            "retrieval_precision_percent": 93.1,
            "mean_latency_ms": 22.74
        },
        "forge_atomic_candidate": {
            "hit_at_1": "12/12",
            "claim_recall": "66/87",
            "claim_recall_percent": 75.9,
            "retrieval_precision_percent": 91.7,
            "mean_latency_ms": 49.89
        },
        "freeze_reason": "The more complex atomic-compatible KESPA ranking did not equal or exceed plain vector claim recall and was slower, despite matching Hit@1."
    },
    "supporting_compatibility_validation": {
        "ids_documents_metadata_exact": "423/423",
        "top1_and_top10_retrieval_matches": "32/32",
        "embeddings_with_float_roundtrip_delta": "43/423",
        "max_embedding_delta": 1.49011611938e-8,
        "legacy_150_char_floor_excluded_claims": "260/423"
    },
    "interpretation": [
        "Legacy retrieval filters designed for long cards were structurally mismatched to short verified atomic claims.",
        "The 150-character minimum excluded 260 of 423 verified claims and caused severe empty-retrieval behavior in calibration.",
        "Reducing the minimum length helped but still underperformed plain vector retrieval.",
        "The final policy freeze selected the simpler plain_vector_top6 path because it preserved claim recall and lower latency without losing Hit@1."
    ],
    "limitations": [
        "The calibration used 12 predeclared questions; it was not a full benchmark by itself.",
        "The detailed initial calibration and final policy-freeze measurements came from separate calibration steps and should not be treated as one identical timing run.",
        "The 54 locked evaluation questions were hash-verified but not parsed during calibration.",
        "The result is specific to the 423-claim verified atomic corpus and the tested retrieval stack."
    ]
}

MARKDOWN

methodology.md

1,242 bytes SHA-256 7d76b89117de261a…

Methodology

Question isolation

The benchmark builder created:

66 total questions;
12 calibration questions;
54 locked evaluation questions.

Only the 12 calibration questions were loaded during retrieval-policy tuning. The locked evaluation file hash was verified without parsing the evaluation questions.

Corpus

The isolated retrieval collection contained 423 verified atomic claims.

A compatibility validator confirmed exact IDs/documents/metadata and retrieval equivalence while allowing harmless float-scale embedding round-trip differences.

Calibration sequence

The initial comparison tested:

1. plain vector top-6 retrieval;

2. the then-current KESPA minimum-150-character retrieval policy;

3. a reduced minimum-40-character atomic-compatible variant.

A later final policy-freeze comparison tested plain vector top-6 against the refined atomic-compatible KESPA ranking path.

The public record keeps the initial and final measurements separate because the reported timing values came from different calibration steps.

Selection rule

The final policy was frozen based on retrieval effectiveness and latency before opening the 54-question evaluation workload.

The selected policy was plain_vector_top6.