← Research archive
Technical note verified Sep 4, 2026

Blinded Quality Judging and Offline Gold-Metadata Repair

KESPA preserved a completed 162-row A/B/C benchmark after discovering that all raw rows had empty gold_claim_ids. The evaluator repaired gold associations offline from locked reference_claim_ids without modifying raw results, verified frozen corpus/evaluation hashes, and used blinded one-answer judge inputs containing no arm labels, runtime metrics, or system identity.

benchmark-integrity blinded-judging metadata-repair evaluation reproducibility
MARKDOWN

README.md

3,023 bytes SHA-256 0eb91ab38fe8b3eb…

KESPA-NOTE-014 — Blinded Quality Judging and Offline Gold-Metadata Repair

Date: 2026-09-04 Status: Verified — user environment Project: KESPA AI / NexLabs Studios

Problem discovered after the raw benchmark

The A/B/C runtime benchmark completed successfully:

  • 54 cases
  • 3 arms
  • 162/162 successful raw rows

But a harness metadata defect was discovered afterward:

162/162 raw rows had empty gold_claim_ids

The underlying issue was that reference_claim_ids had been omitted from the gold-ID extraction path.

The benchmark answers themselves were already complete.

What KESPA did not do

The experiment was not rerun merely to make the metadata prettier.

The original raw_results.jsonl remained unchanged.

Recorded raw-results SHA-256:

C9CD588A1308D58710680A536C9E2509315B7B122A1B574357E6B6C2675C3049

That preserved the original execution evidence.

Offline repair

Gold associations were reconstructed offline from the already locked reference_claim_ids.

Validation recorded:

  • frozen evaluation hash: PASS
  • frozen verified-claim corpus hash: PASS
  • raw matrix: 54 × 3 = 162
  • empty raw gold-ID arrays confirmed: 162/162
  • gold association repaired offline: PASS
  • evaluation gold claims restored: 336
  • raw results modified: NO

This separated:

original experimental output

from:

repaired evaluation metadata

Blinded judging

The repair pipeline then built 162 one-answer judge inputs.

The judge inputs contained no:

  • A/B/C arm labels;
  • runtime metrics;
  • system identity.

Judge:

Qwen/Qwen3.6-35B-A3B

Completed:

  • 162/162 judgments
  • 461,717 reported judge tokens
  • estimated judge cost: $0.140612

Repaired retrieval metrics

The offline gold repair also restored retrieval evaluation:

B — plain verified top-6

  • Hit@1: 54/54
  • Hit@3: 54/54
  • Hit@6: 54/54
  • gold micro recall: 79.17%

C — cards actually used

  • Hit@1: 27/54
  • Hit@3: 27/54
  • Hit@6: 27/54
  • gold micro recall: 22.02%

Why this matters

A tempting response to a benchmark defect is to rerun everything.

That can be worse than the original bug because the new model outputs, runtime conditions, and random effects may no longer represent the frozen experiment.

KESPA instead preserved the original run and repaired only metadata that could be reconstructed from already locked artifacts.

That creates a cleaner audit trail:

raw execution -> immutable

metadata repair -> explicit / offline / reproducible

quality judging -> blinded

Boundary

During the repair/judging process:

  • production Chroma opened: NO
  • production Chroma modified: NO
  • Brain modified: NO
  • database modified: NO
  • raw results modified: NO

Limitation

The final quality scores still depend on one LLM judge.

Blinding and deterministic metadata repair improve benchmark integrity, but they do not turn model-based evaluation into objective ground truth.

TEXT

SHA256SUMS.txt

393 bytes SHA-256 bd09fcbaee93cf72… 
0eb91ab38fe8b3eb601d28ebd045b6405d2e521d82ae9521f1479b003b9c71bd  README.md
db9621c56be3e584a18a1eeeb3d339f79e2c2c2e5dc5ca0f7d28d3393821cf80  note.json
d230f0665a30bbf86f5990e74c97af3293cccb26a121284b34fd6f2a057e3aa9  metrics.csv
179f3ef50b6e2b9963a400fe61ea24a50d8015fe6989ae42fad71069c148324b  methodology.md
858cb4c9a1d7ce4d29c8fb079f661fb92aaf40c9d4e36650ed0c04b095ff76a4  provenance.json
MARKDOWN

methodology.md

1,006 bytes SHA-256 179f3ef50b6e2b99…

Methodology

Defect detection

The completed A/B/C raw result matrix was audited after execution.

All 162 rows were successful, but every row had an empty gold_claim_ids array.

The defect was traced to the evaluator metadata path rather than to missing model execution.

Immutable raw-run rule

The original raw result file was retained unchanged.

Its recorded SHA-256 was used as the identity of the frozen run.

Offline gold reconstruction

Gold associations were rebuilt from the locked reference_claim_ids already present in the frozen evaluation inputs.

No answer was regenerated and no raw benchmark output was rewritten.

Blinded judge inputs

One-answer judge records were produced from the repaired evaluation metadata.

Arm identity, runtime metrics, and system identity were omitted from judge inputs.

Quality evaluation

A single evidence-bound LLM judge produced 162 judgments.

Aggregate quality scoring was performed only after the blinded judgments were complete.

CSV

metrics.csv

577 bytes SHA-256 d230f0665a30bbf8… 
metric,value,unit_or_status
raw_rows,162,rows
unique_cases,54,cases
arms,3,arms
successful_raw_rows,162/162,rows
empty_gold_claim_ids,162/162,rows
evaluation_gold_claims_restored,336,claim_associations
blinded_judge_inputs,162,inputs
judge_completed,162/162,judgments
judge_tokens,461717,tokens
estimated_judge_cost_usd,0.140612,usd
B_hit_at_1,54/54,cases
B_gold_micro_recall,79.17,percent
C_hit_at_1,27/54,cases
C_gold_micro_recall,22.02,percent
raw_results_modified,NO,integrity
production_chroma_modified,NO,boundary
brain_modified,NO,boundary
database_modified,NO,boundary
JSON

note.json

2,973 bytes SHA-256 db9621c56be3e584… 
{
    "schema": "kespa.public_technical_note.v1",
    "id": "KESPA-NOTE-014",
    "title": "Blinded Quality Judging and Offline Gold-Metadata Repair",
    "date": "2026-09-04",
    "status": "verified",
    "purpose": "Document the benchmark-integrity repair that recovered missing gold-claim associations without rerunning or modifying the completed A/B/C raw benchmark, then performed blinded quality judging.",
    "defect": {
        "raw_rows": 162,
        "unique_cases": 54,
        "arms": 3,
        "successful_rows": "162/162",
        "rows_with_empty_gold_claim_ids": "162/162",
        "cause": "reference_claim_ids omitted from extract_gold_claim_ids()",
        "raw_results_sha256": "C9CD588A1308D58710680A536C9E2509315B7B122A1B574357E6B6C2675C3049"
    },
    "repair": {
        "method": "Offline reconstruction of gold association from locked reference_claim_ids.",
        "raw_results_modified": false,
        "evaluation_gold_claims_restored": 336,
        "frozen_evaluation_hash_verified": true,
        "frozen_verified_claim_corpus_hash_verified": true
    },
    "blinding": {
        "judge_inputs": 162,
        "one_answer_per_input": true,
        "arm_labels_exposed": false,
        "runtime_metrics_exposed": false,
        "system_identity_exposed": false
    },
    "judge": {
        "model": "Qwen/Qwen3.6-35B-A3B",
        "judgments_completed": "162/162",
        "reported_tokens": 461717,
        "estimated_cost_usd": 0.140612
    },
    "repaired_retrieval_metrics": {
        "B_plain_top6_hit_at_1": "54/54",
        "B_plain_top6_hit_at_3": "54/54",
        "B_plain_top6_hit_at_6": "54/54",
        "B_plain_top6_micro_recall_percent": 79.17,
        "C_cards_used_hit_at_1": "27/54",
        "C_cards_used_hit_at_3": "27/54",
        "C_cards_used_hit_at_6": "27/54",
        "C_cards_used_micro_recall_percent": 22.02
    },
    "safety_boundary": {
        "production_chroma_opened": false,
        "production_chroma_modified": false,
        "brain_modified": false,
        "database_modified": false,
        "raw_results_modified": false
    },
    "interpretation": [
        "A benchmark metadata defect does not always require rerunning the experiment if the missing association can be reconstructed deterministically from already-locked inputs.",
        "Preserving raw_results.jsonl unchanged retained the original runtime evidence while allowing downstream evaluation metadata to be repaired separately.",
        "Blinding arm identity and runtime metrics reduced the risk that the judge would score an answer based on knowing which system produced it."
    ],
    "limitations": [
        "The quality evaluator used one model-based judge; blinded judging reduces some bias but does not make the judge infallible.",
        "The offline repair is defensible only because it was reconstructed from locked reference metadata rather than regenerated model output.",
        "NOT_ESTABLISHED in the judge framework means evidence-unsupplied under the rubric, not necessarily objectively false.",
        "This note documents evaluation integrity, not a separate claim that any arm is universally superior."
    ]
}
JSON

provenance.json

2,961 bytes SHA-256 858cb4c9a1d7ce4d… 
{
    "schema": "kespa.public_provenance.v1",
    "research_id": "KESPA-NOTE-014",
    "source_basis": "Historical KESPA A/B/C benchmark defect-repair and blinded-judge records from the project changelog/Jira export.",
    "recorded_hashes": [
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/raw_results.jsonl",
            "recorded_sha256": "C9CD588A1308D58710680A536C9E2509315B7B122A1B574357E6B6C2675C3049"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/gold_repair_summary_v001.json",
            "recorded_sha256": "3E3DE2E1C8B56EEC9906F696207E5C6F439DEC7F3389D24236FC869BFB033C06"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/judge_rubric_v001.txt",
            "recorded_sha256": "435A800ABF05280760DFD5C18A8D5BB9A592F37FD367118B34D28E7FE96CEA0A"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_metrics_v001.csv",
            "recorded_sha256": "0289FFEFB818EE8E9A7A797033AB10C9C3CCE2C03B79FD31D226E4F11ABD8033"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_results_blinded_v001.jsonl",
            "recorded_sha256": "423DF203072E5A16BCCEE9B5CDBFB156EB71AF7D3DA2022287153ECBA267BF59"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_results_unblinded_v001.jsonl",
            "recorded_sha256": "2F80C31C5055B14F61A13A021E0097F2CB770C75C37B407ADF35F5F6BF17FA6F"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_summary_v001.json",
            "recorded_sha256": "590CE771F4858CFA4FA34210E7921BD9F7D0F3268217FFDD4B08C3C228EC0265"
        },
        {
            "path": "benchmark_v001/runs/abc_v001_20260904T174419Z/quality_judge_v001/quality_summary_v001.txt",
            "recorded_sha256": "767414A5AF94F72D4383FC524F3795701427AFCC94768B3BF9F979CF1C2A5961"
        }
    ],
    "hash_verification_note": "These SHA-256 values are transcribed from the historical user-environment benchmark record. The underlying benchmark artifacts were not supplied in the public reconstruction package, so this package does not claim independent rehashing.",
    "source_observations": [
        "All 162 raw A/B/C rows completed successfully but had empty gold_claim_ids.",
        "Gold associations were repaired offline from locked reference_claim_ids while raw_results.jsonl remained unchanged.",
        "The repaired evaluator restored 336 evaluation gold-claim associations.",
        "162 blinded one-answer judge inputs contained no arm labels, runtime metrics, or system identity.",
        "Production Chroma, Brain, database, and raw results remained unchanged."
    ],
    "publication_note": "This public note publishes benchmark-integrity methodology, aggregate repair/judge metrics, and recorded hashes. Raw answers, full judge prompts, private implementation code, and proprietary evaluation payloads are not republished."
}