Technical note verified Sep 4, 2026

Frozen Benchmark Construction and Evaluation Leakage Prevention

KESPA built a 66-question benchmark from the 423-claim verified corpus, separating 12 calibration questions from 54 locked evaluation questions. Evaluation hashes were frozen before retrieval-policy tuning, and calibration tooling verified the locked evaluation hash without parsing its contents.

benchmark-design evaluation-freeze data-leakage reproducibility verified-corpus

MARKDOWN

README.md

2,819 bytes SHA-256 c2cce59b32b531df…

KESPA-NOTE-013 — Frozen Benchmark Construction and Evaluation Leakage Prevention

Date: 2026-09-04 Status: Verified — user environment Project: KESPA AI / NexLabs Studios

Purpose

Before comparing retrieval policies or running the A/B/C benchmark, KESPA created a fixed benchmark from the clean 423-claim verified corpus.

The important design decision was to separate:

calibration

from:

evaluation

so retrieval-policy tuning could not simply inspect and optimize against the final benchmark questions.

Benchmark split

The benchmark builder produced:

66 total questions
12 calibration questions
54 locked evaluation questions
coverage of 423 gold claims
0 high-stakes calibration questions
2 high-stakes evaluation questions

Freeze procedure

Before calibration:

the evaluation artifact was frozen;
its SHA-256 was recorded;
calibration tooling verified the evaluation hash;
the evaluation questions themselves were not parsed;
only the 12 calibration questions were loaded.

Recorded calibration output explicitly showed:

Evaluation questions parsed: 0

and:

Evaluation hash verified: YES

Why this matters

A retrieval policy can look artificially strong if its thresholds, filters, or ranking rules are repeatedly tuned while the final evaluation questions are visible.

KESPA instead used:

12 calibration questions -> policy selection

followed by:

54 locked evaluation questions -> final experiment

That does not make the benchmark perfect, but it creates a much cleaner experimental boundary.

Frozen artifact hashes

```text benchmark_calibration_v001.jsonl 366CA6A74CE2EDE4ABAF2DC9EE4641241AC5E57BFF99BCB1319415626A3BC60C

benchmark_evaluation_v001.jsonl 0EE67DE597B793A7A7665558AA97382494989A681376C59454ED4257455ACC62

benchmark_question_manifest_v001.json D319B7F8E3ED9379ECAEFE47E4F0C3DEDE78F57450390C054B87AD3D757A74F7

benchmark_questions_v001.jsonl D10807FCE5C049C991B5AAA4611D1D6D30D2EE61A526B4874DFE05D1F5BB7DBC ```

Experimental isolation

During calibration:

production Chroma opened: NO
production Chroma modified: NO
Brain modified: NO
database modified: NO

The work stayed inside the isolated benchmark environment.

Research significance

This benchmark freeze is the methodological bridge between the clean-corpus work and the later A/B/C and D experiments.

It provides a reproducible answer to:

Which questions were used for tuning, and which questions were reserved for final evaluation?

Limitations

The freeze prevents direct calibration-set/evaluation-set mixing, but it does not prove the benchmark is universally representative.

The questions still came from one fixed verified corpus and one research program's domain coverage.

TEXT

SHA256SUMS.txt

393 bytes SHA-256 9d7b6dae0284b557…

c2cce59b32b531dff068400ecf0ad81d9a4ffbae0f5c47113937551bbde1de8e  README.md
22cd0c2f6dcf0190ae34fba10c884bd576c3fcb52f66814658815219f7dd9f20  note.json
c22acc166d735ea0cbab9dff735faf60fce02aca8889313a06a7f7d1f48d1b8e  metrics.csv
d0c6457f5a939a3895728c7908cd11c7b3ffc4f199cdad9fc8711519fbf9aaec  methodology.md
a18bdc80152f8ef92daa913e9f0b1d113dade5260177bf951bfbb4f08a5762b4  provenance.json

MARKDOWN

methodology.md

856 bytes SHA-256 d0c6457f5a939a38…

Methodology

Question construction

The benchmark question builder created 66 questions from the clean 423-claim verified corpus.

The set was split into 12 calibration questions and 54 locked evaluation questions.

Artifact freeze

The calibration, evaluation, combined-question, and manifest artifacts were assigned fixed SHA-256 hashes.

Those hashes formed the identity of the benchmark inputs used by later experiments.

Calibration isolation

Retrieval-policy calibration loaded only the 12 calibration questions.

The calibration process verified the SHA-256 of the locked evaluation artifact without parsing the evaluation questions.

Production isolation

Calibration and benchmark preparation were performed against isolated benchmark artifacts and Chroma collections.

Production Chroma, Brain, and the database were not modified.

CSV

metrics.csv

468 bytes SHA-256 c22acc166d735ea0…

metric,value,unit_or_status
total_questions,66,questions
calibration_questions,12,questions
locked_evaluation_questions,54,questions
gold_claims_represented,423,claims
high_stakes_calibration,0,questions
high_stakes_evaluation,2,questions
evaluation_questions_parsed_during_calibration,0,questions
evaluation_hash_verified,YES,control
production_chroma_opened,NO,boundary
production_chroma_modified,NO,boundary
brain_modified,NO,boundary
database_modified,NO,boundary

JSON

note.json

2,306 bytes SHA-256 22cd0c2f6dcf0190…

{
    "schema": "kespa.public_technical_note.v1",
    "id": "KESPA-NOTE-013",
    "title": "Frozen Benchmark Construction and Evaluation Leakage Prevention",
    "date": "2026-09-04",
    "status": "verified",
    "purpose": "Document the benchmark-freeze procedure used to separate retrieval-policy calibration from the locked evaluation workload before the A/B/C experiment.",
    "benchmark_set": {
        "total_questions": 66,
        "calibration_questions": 12,
        "locked_evaluation_questions": 54,
        "gold_claims_represented": 423,
        "high_stakes_calibration_questions": 0,
        "high_stakes_evaluation_questions": 2
    },
    "freeze_policy": {
        "evaluation_file_hash_verified_before_calibration": true,
        "evaluation_questions_parsed_during_calibration": 0,
        "calibration_questions_loaded": 12,
        "production_chroma_opened": false,
        "production_chroma_modified": false,
        "brain_modified": false,
        "database_modified": false
    },
    "recorded_hashes": {
        "benchmark_calibration_v001.jsonl": "366CA6A74CE2EDE4ABAF2DC9EE4641241AC5E57BFF99BCB1319415626A3BC60C",
        "benchmark_evaluation_v001.jsonl": "0EE67DE597B793A7A7665558AA97382494989A681376C59454ED4257455ACC62",
        "benchmark_question_manifest_v001.json": "D319B7F8E3ED9379ECAEFE47E4F0C3DEDE78F57450390C054B87AD3D757A74F7",
        "benchmark_questions_v001.jsonl": "D10807FCE5C049C991B5AAA4611D1D6D30D2EE61A526B4874DFE05D1F5BB7DBC"
    },
    "interpretation": [
        "Retrieval-policy tuning was confined to the 12-question calibration set.",
        "The 54-question evaluation set was frozen before policy tuning and its hash could be checked without opening the question contents.",
        "This separation reduced the risk that retrieval-policy choices were tuned directly against the final evaluation cases.",
        "The same frozen evaluation set later supported the A/B/C and adaptive-D research sequence."
    ],
    "limitations": [
        "A frozen evaluation set reduces one form of leakage but does not eliminate all benchmark-design bias.",
        "The question set was derived from one 423-claim verified corpus and therefore reflects that corpus's domain coverage.",
        "Only two high-stakes questions were present in the 54-question locked evaluation set.",
        "Hash freezing establishes artifact identity, not benchmark representativeness."
    ]
}

JSON

provenance.json

1,731 bytes SHA-256 a18bdc80152f8ef9…

{
    "schema": "kespa.public_provenance.v1",
    "research_id": "KESPA-NOTE-013",
    "source_basis": "Historical KESPA benchmark builder and retrieval-calibration records supplied in the project changelog/Jira export.",
    "recorded_hashes": [
        {
            "path": "benchmark_v001/benchmark_calibration_v001.jsonl",
            "recorded_sha256": "366CA6A74CE2EDE4ABAF2DC9EE4641241AC5E57BFF99BCB1319415626A3BC60C"
        },
        {
            "path": "benchmark_v001/benchmark_evaluation_v001.jsonl",
            "recorded_sha256": "0EE67DE597B793A7A7665558AA97382494989A681376C59454ED4257455ACC62"
        },
        {
            "path": "benchmark_v001/benchmark_question_manifest_v001.json",
            "recorded_sha256": "D319B7F8E3ED9379ECAEFE47E4F0C3DEDE78F57450390C054B87AD3D757A74F7"
        },
        {
            "path": "benchmark_v001/benchmark_questions_v001.jsonl",
            "recorded_sha256": "D10807FCE5C049C991B5AAA4611D1D6D30D2EE61A526B4874DFE05D1F5BB7DBC"
        }
    ],
    "hash_verification_note": "These hashes are transcribed from the historical user-environment record. The underlying benchmark question files were not supplied in the current public reconstruction package, so this package does not claim independent rehashing.",
    "source_observations": [
        "The benchmark contained 66 questions: 12 calibration and 54 locked evaluation.",
        "All 423 verified gold claims were represented.",
        "The calibration process verified the locked evaluation hash without parsing the evaluation questions.",
        "Production Chroma, Brain, and database state were not modified."
    ],
    "publication_note": "This public note publishes benchmark-construction metadata, freeze hashes, and methodology. The locked benchmark question text is intentionally not republished here."
}