← Research archive
Experiment verified Sep 2, 2026

Groq Evidence-Bound Generation Canary and Fail-Closed Verification

A four-blueprint cross-domain Groq canary generated four evidence-bound knowledge candidates with zero generation failures for $0.003482 in reported provider cost, while assigning no trust and making no Chroma writes. Downstream verification then demonstrated the intended fail-closed boundary: a PostgreSQL candidate passed claim support verification, a calculus candidate failed on a partially supported claim, and a diabetes candidate surfaced a structural missing-claim-number execution failure.

knowledge-factory groq evidence-bound-generation claim-verification fail-closed canary
MARKDOWN

README.md

4,401 bytes SHA-256 59358c67200f19dd…

KESPA-EXP-007 — Groq Evidence-Bound Generation Canary and Fail-Closed Verification

Date: 2026-09-02 Status: Verified historical experiment Project: KESPA AI / NexLabs Studios

Research question

Can a background external model generate useful structured knowledge candidates from frozen evidence without being allowed to decide what becomes trusted knowledge?

And, separately:

Will the downstream verifier actually stop candidates whose claims are only partially supported or structurally invalid?

This four-blueprint canary provides a small but concrete test of both boundaries.

Generation run

KESPA ran the evidence-bound pilot generator using:

  • generator: forge-kb-v2-generator-006-groq
  • provider: Groq
  • model: openai/gpt-oss-120b
  • blueprints selected: 4
  • candidates generated: 4
  • generation failures: 0
  • prompt tokens: 15,445
  • completion tokens: 1,942
  • total tokens: 17,387 (derived from the two reported token counters)
  • reported run cost: $0.003482

The generator verified frozen evidence hashes before generation.

The historical synthetic corpus was not used as factual evidence.

Trust boundary

Successful generation did not mean successful verification.

Immediately after generation, all four outputs remained candidates:

  • verified=false
  • trust=null
  • production_ready=false
  • Chroma writes: 0

The next explicit gate was human-readable review followed by claim/evidence verification.

That separation is fundamental to KESPA:

evidence-bound generation != verified knowledge

Downstream verifier behavior

The supplied verification retry record demonstrates multiple outcomes.

PostgreSQL

PostgreSQL EXPLAIN and query-plan basics

Result:

  • card verdict: VERIFIER_PASS
  • scope: IN_SCOPE
  • seven claims recorded as supported
  • status: MODEL_VERIFIER_PASS_PENDING_FINAL_POLICY
  • trust remained null
  • verified remained false
  • production-ready remained false

Even a verifier pass did not bypass the final policy boundary.

Calculus

One-variable calculus reference concepts

Result:

  • card verdict: VERIFIER_FAIL
  • scope: IN_SCOPE
  • claims 1–5 were supported
  • claim 6 was only PARTIALLY_SUPPORTED
  • final status: MODEL_VERIFIER_FAIL

The candidate asserted specific Taylor-theorem and convex-function details that were not actually present in the supplied frozen evidence excerpt.

The verifier therefore failed the card instead of treating plausible model output as fact.

Diabetes

What diabetes is: blood glucose and insulin

The verification run recorded a structural execution failure:

missing_claim_number:7

The failure was surfaced explicitly rather than silently interpreted as successful verification.

Result

The canary established two separate properties:

Generation path: 4/4 candidates were produced with 0 generation failures.

Trust path: generated text still had to survive independent claim/evidence verification and later policy gates.

The observed verifier behavior included both a supported pass and fail-closed outcomes for partial evidence and malformed verification structure.

Why this matters

A knowledge-acquisition system becomes dangerous if the same model that writes a candidate can implicitly make that candidate true.

This experiment kept those roles separate.

The external model could help transform evidence into structured candidate claims, but KESPA retained downstream authority over:

  • evidence binding
  • claim support
  • scope
  • verification state
  • trust
  • production eligibility

That design is more important than the 4/4 generation success rate.

Safety boundary

This experiment did not:

  • assign trust during generation;
  • mark generated cards production-ready;
  • write generated candidates to Chroma;
  • use the old synthetic corpus as factual evidence; or
  • interpret a verifier execution error as a successful result.

Limitations

This was a four-blueprint canary, not a scale-quality benchmark.

The reconstruction inputs preserve individual downstream verification outcomes but do not contain a complete final-policy summary for all four generated cards. This record therefore does not invent an overall post-verification pass rate.

The experiment supports a claim about pipeline behavior and trust separation, not broad factual-accuracy performance.

TEXT

SHA256SUMS.txt

399 bytes SHA-256 cdc4c14f4ce05ce2… 
59358c67200f19dd01ec88024c8daacf93602ac26240a739bd0d759fcca7d1f4  README.md
2b074c4a6dd90cef816bc4c070047ddafba564a4df78f594fe3d14a88a571442  experiment.json
18055c254cc92eb705e31cdb0d4aa4022bbdf29b9147903e482de2ccd7e37e92  results.csv
8e5d7d6c22cc516fa367c4b5206a12862d567bd4b4f69a34874d00b98a05252f  methodology.md
e90242382a4e64426ecce13ead0b195199400d92deccb4033626eb1f1733cebe  provenance.json
JSON

experiment.json

3,236 bytes SHA-256 2b074c4a6dd90cef… 
{
    "schema": "kespa.public_experiment.v1",
    "id": "KESPA-EXP-007",
    "title": "Groq Evidence-Bound Generation Canary and Fail-Closed Verification",
    "date": "2026-09-02",
    "status": "verified",
    "question": "Can an external background model generate structured candidates from frozen evidence without crossing the KESPA trust boundary, and will downstream verification reject partially supported or structurally invalid outputs instead of silently promoting them?",
    "generation_run": {
        "generator_version": "forge-kb-v2-generator-006-groq",
        "provider": "groq",
        "model": "openai/gpt-oss-120b",
        "blueprints_selected": 4,
        "candidates_generated": 4,
        "generation_failures": 0,
        "prompt_tokens_reported": 15445,
        "completion_tokens_reported": 1942,
        "total_tokens_derived": 17387,
        "reported_cost_usd": 0.003482
    },
    "generation_safety_state": {
        "evidence_hashes_verified_before_generation": true,
        "old_synthetic_corpus_used_as_factual_evidence": false,
        "trust_assigned": false,
        "cards_verified_at_generation": false,
        "production_ready_cards": 0,
        "chroma_writes": 0,
        "candidate_state": {
            "verified": false,
            "trust": null,
            "production_ready": false
        }
    },
    "observed_downstream_verification": [
        {
            "topic": "PostgreSQL",
            "card": "PostgreSQL EXPLAIN and query-plan basics",
            "verdict": "VERIFIER_PASS",
            "status": "MODEL_VERIFIER_PASS_PENDING_FINAL_POLICY",
            "scope": "IN_SCOPE",
            "trust_assigned": false,
            "production_ready": false
        },
        {
            "topic": "Calculus",
            "card": "One-variable calculus reference concepts",
            "verdict": "VERIFIER_FAIL",
            "status": "MODEL_VERIFIER_FAIL",
            "scope": "IN_SCOPE",
            "reason": "Claim 6 was only partially supported because the frozen evidence did not contain the specific Taylor-theorem and convex-function definitions asserted by the candidate.",
            "trust_assigned": false,
            "production_ready": false
        },
        {
            "topic": "Diabetes",
            "card": "What diabetes is: blood glucose and insulin",
            "verdict": "VERIFICATION_EXECUTION_FAILURE",
            "reason": "missing_claim_number:7",
            "trust_assigned": false,
            "production_ready": false
        }
    ],
    "interpretation": [
        "External generation was treated as candidate construction, not factual authority.",
        "Candidate generation succeeded independently of trust assignment.",
        "The verifier rejected a candidate when frozen evidence only partially supported a claim.",
        "A structural verification defect was surfaced as an execution failure rather than treated as a pass.",
        "The observed PostgreSQL verifier pass still remained pending final policy and did not itself make the card trusted or production-ready."
    ],
    "limitations": [
        "The canary contained only four blueprints.",
        "The supplied reconstruction records do not contain a complete final-policy outcome for all four generated cards.",
        "The experiment demonstrates generation and verification-boundary behavior, not production answer quality.",
        "The reported provider cost is the cost recorded by the original run and is not independently recomputed here."
    ]
}
MARKDOWN

methodology.md

1,838 bytes SHA-256 8e5d7d6c22cc516f…

Methodology

1. Bounded canary selection

The pilot selected four cross-domain card blueprints rather than starting with a large generation batch.

Each blueprint was already bound to collected evidence.

2. Evidence-bound generation

Groq openai/gpt-oss-120b was used by forge-kb-v2-generator-006-groq.

Frozen evidence hashes were checked before generation.

The model's job was to create structured candidate material from the supplied evidence. Its output was not treated as a source and did not receive trust.

3. Candidate-only state

Generated objects remained:

  • verified=false
  • trust=null
  • production_ready=false

No Chroma writes were performed.

4. Human-readable inspection

The generator emitted a candidate review artifact so generated claims, retrieval text, sources, and safety flags could be inspected before trust assignment.

5. Independent claim/evidence verification

A downstream verifier evaluated candidate claims against the frozen evidence and approved scope.

This stage was allowed to fail the candidate.

Observed examples include:

  • PostgreSQL: verifier pass, still pending final policy.
  • Calculus: verifier fail because one claim was only partially supported by the available frozen evidence.
  • Diabetes: verification execution failure because claim numbering was structurally incomplete.

6. Fail-closed interpretation

A generation success was never counted as a verification success.

A partial-support finding or execution defect did not produce trusted or production-ready knowledge.

Measurement boundary

The experiment records provider usage and observed pipeline states.

It does not estimate an all-card verification pass percentage because the supplied reconstruction set does not contain a complete final-policy result for every one of the four candidates.

JSON

provenance.json

1,754 bytes SHA-256 e90242382a4e6442… 
{
    "schema": "kespa.public_provenance.v1",
    "research_id": "KESPA-EXP-007",
    "source_basis": [
        {
            "artifact": "knowledge_cards_pilot/canary-groq/generation_summary.txt",
            "role": "generation run metrics and safety state"
        },
        {
            "artifact": "knowledge_cards_pilot/canary-groq/candidate_review.txt",
            "role": "candidate-only state and four-card review record"
        },
        {
            "artifact": "knowledge_cards_pilot/canary-groq/verification-retry/verification_review.txt",
            "role": "observed downstream verifier outcomes"
        }
    ],
    "recorded_facts": {
        "generator_version": "forge-kb-v2-generator-006-groq",
        "provider": "groq",
        "model": "openai/gpt-oss-120b",
        "blueprints_selected": 4,
        "candidates_generated": 4,
        "generation_failures": 0,
        "prompt_tokens_reported": 15445,
        "completion_tokens_reported": 1942,
        "reported_cost_usd": 0.003482
    },
    "derived_facts": {
        "total_tokens": {
            "value": 17387,
            "derivation": "15445 prompt tokens + 1942 completion tokens"
        }
    },
    "verification_observations": {
        "postgresql": "VERIFIER_PASS / MODEL_VERIFIER_PASS_PENDING_FINAL_POLICY",
        "calculus": "VERIFIER_FAIL due to a PARTIALLY_SUPPORTED claim",
        "diabetes": "verification execution failure: missing_claim_number:7"
    },
    "reconstruction_boundary": "No complete final-policy outcome for all four canary candidates was present in the supplied records, so this package intentionally does not claim an aggregate post-verification pass rate.",
    "publication_boundary": "This public package summarizes measured behavior and metadata. It does not republish raw third-party evidence excerpts, credentials, private configuration, or private implementation code."
}
CSV

results.csv

882 bytes SHA-256 18055c254cc92eb7… 
phase,metric,value,unit,basis
generation,blueprints_selected,4,blueprints,recorded
generation,candidates_generated,4,candidates,recorded
generation,generation_failures,0,failures,recorded
generation,prompt_tokens,15445,tokens,recorded
generation,completion_tokens,1942,tokens,recorded
generation,total_tokens,17387,tokens,derived
generation,reported_cost,0.003482,USD,recorded
safety,evidence_hashes_verified_before_generation,YES,status,recorded
safety,old_synthetic_corpus_used_as_factual_evidence,NO,status,recorded
safety,trust_assigned_during_generation,NO,status,recorded
safety,production_ready_cards_at_generation,0,cards,recorded
safety,chroma_writes,0,writes,recorded
verification,postgresql,VERIFIER_PASS,outcome,recorded
verification,calculus,VERIFIER_FAIL_PARTIAL_SUPPORT,outcome,recorded
verification,diabetes,EXECUTION_FAILURE_MISSING_CLAIM_NUMBER_7,outcome,recorded