Technical note verified Sep 5, 2026

Public-Only External Intelligence Refinery and Budgeted Cloud Analysis

KESPA introduced an optional external-intelligence refinery that sends only explicitly classified public_external/public_benchmark tasks to Groq, retains all provider output as non-authoritative candidate analysis, enforces daily request/token budgets and rate-limit backoff, and persists queue/results/provenance locally. A desktop validation completed four Groq 120B analyses and exposed the provider's 8K TPM limit; the worker was then made rate-aware. A later Windows-service wrapper was prepared for the always-on home server but had not yet received a user-environment SCM validation at the recorded checkpoint.

external-intelligence groq privacy-boundary rate-limits retrieval-diagnostics budget-control

MARKDOWN

README.md

5,864 bytes SHA-256 624979a3890a31ce…

KESPA-NOTE-021 — Public-Only External Intelligence Refinery and Budgeted Cloud Analysis

Date: 2026-09-05 Status: Verified worker boundary; Windows-service deployment separately noted as pending user-environment validation at the recorded checkpoint Project: KESPA AI / NexLabs Studios

Purpose

KESPA's External Intelligence Refinery was built to answer a narrow architectural question:

Can KESPA use strong hosted inference for background analysis without turning the hosted model into a trust authority or sending private/internal data to it?

The refinery's answer is yes, but only behind explicit classification, budget, provenance, and promotion boundaries.

External-data boundary

The refinery permits only tasks explicitly classified as:

public_external
public_benchmark

Private/internal classes are blocked before provider execution, including categories such as:

private
personal_private
business_private
confidential
restricted
secret
PHI / health-private

Manual tasks must also be explicitly classified into one of the two allowed public classes.

Provider output is not trusted knowledge

Every successful refinery result preserves a trust block equivalent to:

``text provider output is evidence: NO provider output is verified knowledge: NO automatic promotion allowed: NO next gate: local review / future knowledge lifecycle ``

The structured diagnostic schema also requires:

do_not_auto_promote = true

That means Groq can analyze a retrieval failure, propose query rewrites, identify possible embedding/reranker issues, or suggest metadata improvements — but it cannot decide that its own output is true.

Durable queue and provenance

The refinery uses file-backed states:

``text pending/ processing/ done/ failed/ ``

Tasks receive deterministic content-derived IDs.

Successful results preserve:

input SHA-256;
result SHA-256;
provider/model identity;
latency;
token usage;
request identity where available;
provider rate-limit headers;
trust/promotion state.

Failures are also persisted.

Rate-limited tasks are returned to the pending queue rather than discarded.

First workload: retrieval diagnosis

The initial high-value workload was public benchmark retrieval diagnosis.

D v001 misses can be transformed into tasks containing:

the question;
retrieved verified claims;
locked expected gold claims;
frozen artifact hashes.

The external analyst is instructed to diagnose the miss without inventing external facts or marking anything verified.

Its recommendations are normalized into a controlled action vocabulary such as:

query rewrite;
embedding change;
reranker change;
metadata enrichment;
claim rewrite/split;
increase candidate recall;
manual review.

Spare-quota claim enrichment

A separate enrichment worker can spend otherwise-unused external quota improving retrieval aids for already verified claims.

It can propose:

alternate user queries;
aliases/synonyms;
key concepts;
retrieval keywords;
ambiguity notes;
freshness risk;
semantically equivalent retrieval text.

But it is explicitly told:

Do not verify, correct, expand, or replace the claim. Generate retrieval aids only. Do not introduce new factual assertions.

The trusted corpus remains read only.

It also skips claims that are:

high stakes;
safety sensitive;
marked for human review.

And if the real refinery queue has work, enrichment yields priority and does not run.

Rate-aware operation

The initial desktop run completed 4 Groq 120B analyses and exposed an 8K TPM provider limit.

The worker was subsequently made rate-aware.

The recorded deployment configuration preserved:

observed provider limit: 8K TPM
operating target: 6.5K TPM
daily token ceiling: 180,000
daily request ceiling: 900
proactive pacing
Retry-After handling
rate-limit cooldown
persistent usage ledger

The design response to quota pressure is therefore:

wait / queue / resume

not:

weaken safety / skip provenance / auto-promote

Windows service deployment

Because the refinery does not require the local GPU, the deployment target was moved toward the always-on Windows home server.

The service wrapper added:

delayed automatic startup;
restart-on-failure behavior;
immediate first cycle;
15-minute default queue cycles;
portable service-root inference;
status/log tooling;
Windows-to-Windows state migration.

Migration deliberately excludes .env and API keys.

At the recorded September 5 checkpoint:

worker compile: PASS
Windows-service wrapper compile: PASS
service command construction review: PASS
actual Windows SCM user-environment validation: not yet claimed

This note preserves that distinction rather than retroactively calling the service deployment verified.

Research significance

This component demonstrates an important KESPA design principle:

external model capacity != external trust authority

KESPA can borrow high-capability cloud compute for bounded public analysis while retaining:

privacy classification locally;
deterministic trust rails locally;
persistent provenance locally;
promotion authority locally.

That keeps provider choice replaceable.

Groq can later be swapped or joined by another hosted/local model without changing the rule that provider output starts as candidate intelligence, not truth.

Limitations

The original refinery workload was narrow and focused mainly on retrieval diagnosis.

Additional provider roles and full knowledge-lifecycle integration were future work at this historical checkpoint.

The recorded budget numbers are deployment settings, not permanent architectural constants.

TEXT

SHA256SUMS.txt

393 bytes SHA-256 2d2841e6baee87bc…

624979a3890a31ce57ebd7e9ed654551d5eb646ac29434c85ea9f61e36f5648b  README.md
783b35d104bdf67089b85d75e768860e0bce9a34d35b08d692919e2d4ed3e02a  note.json
0bfc8f398f60fc22cf84bc0bec146532a0cafcb3ca4bfd02118fc1888ffcb1cb  metrics.csv
a66421618e7c7271c50a00d1d853890d5a375b0cddffc2198d7fa910426d4345  methodology.md
a774a4eaf2b65833f17d6b3e0241db348b7891c7429a11a23bfe2b490e963f15  provenance.json

MARKDOWN

methodology.md

1,615 bytes SHA-256 a66421618e7c7271…

Methodology

Classification gate

Before an external request is issued, the refinery checks the task's classification.

Only public_external and public_benchmark are allowed. Private/internal classifications fail closed.

Structured external analysis

The external provider receives bounded public task data and is required to return a strict structured schema.

For retrieval diagnosis, the schema includes an explicit do_not_auto_promote=true hard rail.

Trust separation

Provider output is stored as candidate analysis with provenance and explicit trust metadata.

It is never marked as evidence or verified knowledge by the refinery itself.

Queue durability

Tasks transition through persistent pending, processing, done, and failed directories.

Rate-limited tasks are requeued.

Successful and failed executions are recorded in a usage/audit ledger.

Budget control

The worker tracks daily successful requests and token usage and stops before configured ceilings or reserved token capacity would be exceeded.

Provider rate limits trigger pacing/cooldown rather than policy bypass.

Spare-quota enrichment

Claim enrichment runs only when the primary refinery queue is idle.

It reads already verified non-high-stakes claims and creates retrieval aids without modifying the trusted corpus or introducing new factual assertions.

Deployment validation boundary

The refinery worker had real external-provider validation.

The later Windows service wrapper had compile/review validation but, at the recorded changelog checkpoint, not an actual Windows SCM user-environment PASS.

CSV

metrics.csv

880 bytes SHA-256 0bfc8f398f60fc22…

metric,value,unit_or_status
allowed_external_classifications,2,classes
provider_output_is_evidence,NO,trust
provider_output_is_verified_knowledge,NO,trust
automatic_promotion_allowed,NO,trust
desktop_groq_120b_analyses,4,successful analyses
observed_provider_tpm_limit,8000,tokens_per_minute
configured_operating_target,6500,tokens_per_minute
configured_daily_token_ceiling,180000,tokens
configured_daily_request_ceiling,900,requests
primary_queue_states,4,states
rate_limited_tasks_requeued,YES,resilience
daily_usage_ledger,YES,provenance
trusted_corpus_modified_by_enrichment,NO,boundary
high_stakes_claims_enriched,NO,boundary
real_queue_priority_over_enrichment,YES,scheduling
windows_service_default_cycle,15,minutes
windows_service_worker_compile,PASS,validation
windows_service_wrapper_compile,PASS,validation
windows_scm_user_environment_pass_at_checkpoint,NO,validation

JSON

note.json

4,025 bytes SHA-256 783b35d104bdf670…

{
    "schema": "kespa.public_technical_note.v1",
    "id": "KESPA-NOTE-021",
    "title": "Public-Only External Intelligence Refinery and Budgeted Cloud Analysis",
    "date": "2026-09-05",
    "status": "verified",
    "purpose": "Document how KESPA can use optional cloud inference for public benchmark/research analysis without allowing the external provider to become a trust authority or receive private/internal data.",
    "refinery_contract": {
        "allowed_external_classifications": [
            "public_external",
            "public_benchmark"
        ],
        "blocked_classifications": [
            "private",
            "personal_private",
            "business_private",
            "confidential",
            "restricted",
            "secret",
            "phi",
            "health_private"
        ],
        "provider_output_is_evidence": false,
        "provider_output_is_verified_knowledge": false,
        "automatic_promotion_allowed": false,
        "required_next_gate": "local_review_or_future_knowledge_lifecycle"
    },
    "queue": {
        "states": [
            "pending",
            "processing",
            "done",
            "failed"
        ],
        "task_ids_content_derived": true,
        "result_input_sha256_recorded": true,
        "result_sha256_recorded": true,
        "failure_records_persisted": true,
        "rate_limited_tasks_requeued": true
    },
    "workloads": {
        "retrieval_diagnosis": {
            "purpose": "Analyze public benchmark retrieval misses such as D v001 misses.",
            "structured_output": true,
            "do_not_auto_promote_required": true,
            "controlled_action_vocabulary": true
        },
        "claim_retrieval_enrichment": {
            "purpose": "Use spare external quota to generate retrieval aids for already verified claims.",
            "trusted_corpus_read_only": true,
            "real_refinery_queue_has_priority": true,
            "high_stakes_skipped": true,
            "safety_sensitive_skipped": true,
            "human_review_required_skipped": true,
            "new_factual_assertions_allowed": false,
            "auto_promotion": false
        }
    },
    "rate_and_budget_controls": {
        "observed_provider_tpm_limit": 8000,
        "configured_operating_target_tpm": 6500,
        "configured_daily_token_ceiling": 180000,
        "configured_daily_request_ceiling": 900,
        "retry_after_handling": true,
        "proactive_pacing": true,
        "cooldown_on_rate_limit": true,
        "daily_usage_ledger": true
    },
    "observed_validation": {
        "desktop_groq_120b_analyses_completed": 4,
        "provider_limit_exposed": "8K TPM",
        "worker_compile": "PASS",
        "windows_service_wrapper_compile": "PASS",
        "service_command_construction_review": "PASS"
    },
    "windows_service_checkpoint": {
        "deployment_target": "always-on Windows home server",
        "delayed_auto_start": true,
        "restart_on_failure": true,
        "default_queue_cycle_minutes": 15,
        "immediate_cycle_at_start": true,
        "state_migration_excludes_env_and_api_keys": true,
        "user_environment_scm_validation_at_recorded_checkpoint": false
    },
    "interpretation": [
        "Cloud inference is treated as optional analytical compute rather than as an authority over trust.",
        "Privacy classification is checked before an external request is made.",
        "Rate-limit and daily-budget controls convert provider scarcity into queued delay rather than pressure to bypass policy.",
        "Spare quota can improve retrieval metadata for already verified claims while leaving the trusted corpus unchanged.",
        "The architecture separates always-on background analysis from the interactive local GPU workstation."
    ],
    "limitations": [
        "At the recorded checkpoint, the Windows SCM deployment itself had not yet received a user-environment PASS.",
        "The refinery's original task set was narrow and centered on retrieval diagnosis; additional external-intelligence roles were still reserved.",
        "External model output remains candidate analysis and requires separate local lifecycle gates before it can influence trusted knowledge.",
        "Configured ceilings describe the recorded deployment configuration, not universal provider limits or future KESPA settings."
    ]
}

JSON

provenance.json

1,530 bytes SHA-256 a774a4eaf2b65833…

{
    "schema": "kespa.public_provenance.v1",
    "research_id": "KESPA-NOTE-021",
    "source_basis": [
        "KESPA V2 Groq External Intelligence Refinery v001 source record.",
        "KESPA Git/Audit Changelog dated 2026-09-05 for the refinery Windows-service deployment correction.",
        "KESPA verified-claim retrieval-enrichment worker source record."
    ],
    "source_observations": [
        "The refinery allows only public_external and public_benchmark and explicitly blocks private/internal classifications.",
        "Stored provider results say provider output is not evidence, not verified knowledge, and cannot auto-promote.",
        "The queue persists pending/processing/done/failed state and requeues rate-limited tasks.",
        "The recorded desktop run completed four Groq 120B analyses and exposed the 8K TPM limit.",
        "The recorded service configuration used a 6.5K TPM operating target, 180K daily token ceiling, and 900 daily request ceiling.",
        "The claim-enrichment worker gives real refinery work priority, reads the trusted corpus without modifying it, and skips high-stakes/safety-sensitive/human-review claims.",
        "At the recorded September 5 checkpoint, the Windows-service wrapper had compile/review PASS results but no user-environment Windows SCM PASS."
    ],
    "publication_note": "This public note describes safety boundaries, queue behavior, budget controls, and validation state. API keys, environment values, raw benchmark questions, private task payloads, and private implementation secrets are excluded."
}