MARKDOWN
README.md
4,884 bytes
SHA-256 727338ef0d6c77d5…
KESPA-EXP-008 — Cross-Model Claim Verification and Human-Review Gating Pilot
Date: 2026-09-02 Status: Verified — user environment Project: KESPA AI / NexLabs Studios
Research question
Can KESPA independently check generated candidate claims against their frozen evidence using a different model family — while still failing closed on weak evidence and preserving separate human-review requirements for higher-risk material?
The recorded pilot verifier used:
- verifier:
forge-kb-v2-claim-verifier-002-free-tier - verifier model:
qwen/qwen3.6-27b - generator/verifier same model family: NO
Four recorded verification runs
Across four recorded runs, KESPA performed 21 card-evaluation executions.
Aggregate recorded outcomes:
- verifier passes: 19
- verifier failures: 1
- execution failures: 1
- human-review-required flags: 7
- supported claim verdicts: 130
- partially supported: 1
- unsupported: 1
- contradicted: 0
- unclear: 0
- reported tokens: 105,688
These totals are the sum of recorded verifier runs. They are not presented as a separately deduplicated corpus count.
Run 1 — six-card clean verification
batch-13a/verification-6b
Result:
- candidate cards: 6
- verifier pass: 6
- verifier fail: 0
- execution failures: 0
- supported claims: 40
- total reported tokens: 29,634
All six passed the model-verification stage.
But the state remained:
trust=null
verified=false
production_ready=false
The next gate was still the final policy/trust stage.
Run 2 — verifier catches unsupported material
batch-13a/verification-7a
Result:
- candidate cards: 7
- verifier pass: 6
- verifier fail: 1
- supported claims: 45
- partially supported: 1
- unsupported: 1
- total reported tokens: 30,762
This is important because the verifier did not simply agree with everything the generator produced.
A candidate with incomplete evidence support failed the card-level verification gate.
Run 3 — high-stakes model pass does not remove human review
batch-13b/verification-highstakes-4
Topics included:
- Cognitive Behavioral Therapy CBT
- Investing
Result:
- candidate cards: 4
- verifier pass: 4
- verifier fail: 0
- supported claims: 26
- human review required: 4
- total reported tokens: 21,601
Every card passed model verification.
Every card still required human review.
The resulting state remained equivalent to:
MODEL_VERIFIER_PASS_PENDING_HUMAN_REVIEW
with no trust assignment and no production-ready status.
That demonstrates a deliberate policy boundary:
model agreement does not erase a higher-risk review requirement.
Run 4 — structural verifier failure is blocking
batch-13b/verification-safety-4
Result:
- candidate cards: 4
- verifier pass: 3
- verifier fail: 0
- execution failures: 1
- supported claims: 19
- human review required: 3
- total reported tokens: 23,691
The execution failure was surfaced as:
missing_claim_number:7
The next gate explicitly required resolving the verifier execution failure first.
KESPA did not infer that an incomplete verifier response meant the missing claim was safe.
Trust and production boundary
Across the recorded verifier runs:
- frozen evidence hashes rechecked: YES
- candidate content/claim identity checked: YES
- trust assigned: NO
- candidate files mutated: NO
- cards marked verified by this stage: NO
- production-ready cards: 0
- Chroma writes: 0
The model verifier was an analytical gate, not the authority that promoted knowledge.
Why cross-model verification mattered
The verifier used a different model family from the generator.
That does not guarantee independence in the statistical or epistemic sense, but it reduces one obvious failure mode:
generator creates claim -> same model simply endorses its own phrasing
Instead, candidate claims were checked against the frozen evidence under a separate verifier contract.
Research significance
This pilot demonstrated three useful behaviors at once:
1. supported candidates could pass an independent model-verification stage;
2. partially/unsupported material could fail that stage;
3. high-stakes material could remain blocked for human review even after a model pass.
That is a much stronger trust boundary than treating model confidence or model agreement as permission to publish.
Limitations
This was a small pilot and should not be interpreted as a general verifier-accuracy benchmark.
The aggregate count is a sum of four recorded evaluation runs, not a claim about a deduplicated production corpus.
Model-based evidence verification still inherits model limitations and therefore remains only one layer in the KESPA trust lifecycle.
TEXT
SHA256SUMS.txt
399 bytes
SHA-256 644d8a6e37ad5ecb…
727338ef0d6c77d583ad8b41f2ba8d53816689058356525055f9e84d31267039 README.md
2165a7c29815b0a97747008faa1ab2c555a67145226f7640c2a2f93f841b49b5 experiment.json
0e7694c135ac4dde67323965b58202fc5ad9695c1cf1f047a4b2de1043459a19 results.csv
7d8b9417ffa177486a69aa18d2099a8e3657a159178125a3dcce80e0825330a7 methodology.md
9080eb0f05c13ac9189900a3ec285883a3d6cb2834bc37fca79d808c3ea323a4 provenance.json
JSON
experiment.json
4,992 bytes
SHA-256 2165a7c29815b0a9…
{
"schema": "kespa.public_experiment.v1",
"id": "KESPA-EXP-008",
"title": "Cross-Model Claim Verification and Human-Review Gating Pilot",
"date": "2026-09-02",
"status": "verified",
"research_question": "Can KESPA independently verify generated candidate claims against frozen evidence using a different model family while preserving fail-closed behavior and separate human-review requirements for higher-risk material?",
"verifier": {
"version": "forge-kb-v2-claim-verifier-002-free-tier",
"model": "qwen/qwen3.6-27b",
"generator_and_verifier_same_model_family": false,
"provider_mode": "Groq free-tier quota guard"
},
"recorded_runs": [
{
"run": "batch-13a/verification-6b",
"candidate_cards": 6,
"verifier_pass": 6,
"verifier_fail": 0,
"execution_failures": 0,
"human_review_required": 0,
"supported": 40,
"partially_supported": 0,
"unsupported": 0,
"contradicted": 0,
"unclear": 0,
"prompt_tokens": 21823,
"completion_tokens": 7811,
"total_tokens": 29634
},
{
"run": "batch-13a/verification-7a",
"candidate_cards": 7,
"verifier_pass": 6,
"verifier_fail": 1,
"execution_failures": 0,
"human_review_required": 0,
"supported": 45,
"partially_supported": 1,
"unsupported": 1,
"contradicted": 0,
"unclear": 0,
"prompt_tokens": 23725,
"completion_tokens": 7037,
"total_tokens": 30762
},
{
"run": "batch-13b/verification-highstakes-4",
"candidate_cards": 4,
"verifier_pass": 4,
"verifier_fail": 0,
"execution_failures": 0,
"human_review_required": 4,
"supported": 26,
"partially_supported": 0,
"unsupported": 0,
"contradicted": 0,
"unclear": 0,
"prompt_tokens": 17341,
"completion_tokens": 4260,
"total_tokens": 21601
},
{
"run": "batch-13b/verification-safety-4",
"candidate_cards": 4,
"verifier_pass": 3,
"verifier_fail": 0,
"execution_failures": 1,
"human_review_required": 3,
"supported": 19,
"partially_supported": 0,
"unsupported": 0,
"contradicted": 0,
"unclear": 0,
"prompt_tokens": 19171,
"completion_tokens": 4520,
"total_tokens": 23691
}
],
"aggregate_recorded_run_totals": {
"candidate_cards": 21,
"verifier_pass": 19,
"verifier_fail": 1,
"execution_failures": 1,
"human_review_required": 7,
"supported": 130,
"partially_supported": 1,
"unsupported": 1,
"contradicted": 0,
"unclear": 0,
"prompt_tokens": 82060,
"completion_tokens": 23628,
"total_tokens": 105688
},
"aggregate_note": "Totals are sums of four recorded verifier runs and are reported as card-evaluation executions, not asserted as a deduplicated unique-card corpus.",
"safety_invariants": {
"frozen_evidence_hashes_rechecked": true,
"candidate_content_claim_identity_checked": true,
"trust_assigned": false,
"candidate_files_mutated": false,
"cards_marked_verified_by_model_stage": false,
"production_ready_cards": 0,
"chroma_writes": 0
},
"observed_fail_closed_cases": [
{
"case": "batch-13a/verification-7a",
"result": "VERIFIER_FAIL",
"evidence": "One candidate contained one PARTIALLY_SUPPORTED and one UNSUPPORTED claim; the card failed rather than being promoted."
},
{
"case": "batch-13b/verification-safety-4",
"result": "EXECUTION_FAILURE",
"evidence": "A missing claim number was surfaced as missing_claim_number:7 and the run required resolution before proceeding."
}
],
"high_stakes_behavior": {
"recorded_high_stakes_batch_cards": 4,
"model_passes": 4,
"human_review_required": 4,
"trust_assigned": false,
"production_ready": false,
"example_topics": [
"Cognitive Behavioral Therapy CBT",
"Investing"
]
},
"interpretation": [
"Using a different model family for verification reduced direct generator/verifier self-agreement risk.",
"Model-level support was not sufficient to assign trust or mark a card production-ready.",
"Higher-risk content retained a separate human-review gate even when every tested claim was model-supported.",
"Partial support, unsupported claims, and structural verification errors were surfaced as blocking outcomes instead of being coerced into a pass."
],
"limitations": [
"These are four recorded pilot runs, not a statistically representative estimate of verifier accuracy.",
"The aggregate card count is a sum of recorded evaluation executions and is not presented as a deduplicated corpus cardinality.",
"Model verdicts measure agreement with supplied frozen evidence; they do not independently establish objective truth.",
"Human-review-required means the model stage passed but policy still required a person before trust/promotion."
]
}
MARKDOWN
methodology.md
1,454 bytes
SHA-256 7d8b9417ffa17748…
Methodology
Candidate/evidence identity
Before model evaluation, the verifier rechecked frozen evidence hashes and candidate content/claim identity.
This prevents the verifier from silently judging different evidence or mutated candidate content from the material that was originally bound together.
Cross-model verifier
Candidate claims were evaluated with qwen/qwen3.6-27b.
The run records explicitly state that the generator and verifier were not from the same model family.
Claim-level verdicts
Each material claim could receive a support verdict such as:
- SUPPORTED
- PARTIALLY_SUPPORTED
- UNSUPPORTED
- CONTRADICTED
- UNCLEAR
Card-level verification could fail if claim support was insufficient.
Structural validation
Malformed or incomplete verifier structure was treated as an execution failure.
The missing_claim_number:7 case demonstrates that a structurally incomplete response did not become an implicit pass.
High-stakes review
High-stakes/safety-sensitive candidates could pass model verification while still requiring human review.
A model pass did not assign trust, mark the candidate verified, or make it production-ready.
Aggregate calculation
Public aggregate metrics are simple sums across the four recorded verifier summaries.
They are labeled as card-evaluation executions because this reconstruction does not independently prove corpus-level deduplication across every historical batch.
JSON
provenance.json
1,667 bytes
SHA-256 9080eb0f05c13ac9…
{
"schema": "kespa.public_provenance.v1",
"research_id": "KESPA-EXP-008",
"source_basis": [
{
"record": "batch-13a/verification-6b",
"observations": "6 candidates; 6 pass; 40 supported claims; 29,634 tokens; no trust/chroma writes."
},
{
"record": "batch-13a/verification-7a",
"observations": "7 candidates; 6 pass; 1 fail; 45 supported, 1 partial, 1 unsupported; 30,762 tokens."
},
{
"record": "batch-13b/verification-highstakes-4",
"observations": "4 candidates; 4 model passes; 4 human-review-required; 26 supported claims; 21,601 tokens."
},
{
"record": "batch-13b/verification-safety-4",
"observations": "4 candidates; 3 passes; 1 execution failure; 3 human-review-required; 19 supported claims; 23,691 tokens."
}
],
"aggregate_derivation": {
"candidate_card_evaluations": "6 + 7 + 4 + 4 = 21",
"verifier_passes": "6 + 6 + 4 + 3 = 19",
"verifier_failures": "0 + 1 + 0 + 0 = 1",
"execution_failures": "0 + 0 + 0 + 1 = 1",
"human_review_required": "0 + 0 + 4 + 3 = 7",
"supported_claims": "40 + 45 + 26 + 19 = 130",
"reported_tokens": "29,634 + 30,762 + 21,601 + 23,691 = 105,688"
},
"boundary_note": "Aggregate values describe recorded verifier executions, not an independently deduplicated unique-card corpus. No trust, production promotion, or Chroma writes are inferred beyond the explicit source records.",
"publication_note": "This package publishes aggregate verifier behavior and methodology only. Raw third-party evidence text, private candidate bodies, credentials, provider keys, and private implementation code are excluded."
}
CSV
results.csv
532 bytes
SHA-256 0e7694c135ac4dde…
run,candidate_cards,verifier_pass,verifier_fail,execution_failures,human_review_required,supported,partially_supported,unsupported,contradicted,unclear,prompt_tokens,completion_tokens,total_tokens
batch-13a/verification-6b,6,6,0,0,0,40,0,0,0,0,21823,7811,29634
batch-13a/verification-7a,7,6,1,0,0,45,1,1,0,0,23725,7037,30762
batch-13b/verification-highstakes-4,4,4,0,0,4,26,0,0,0,0,17341,4260,21601
batch-13b/verification-safety-4,4,3,0,1,3,19,0,0,0,0,19171,4520,23691
TOTAL_RECORDED_RUNS,21,19,1,1,7,130,1,1,0,0,82060,23628,105688