MARKDOWN
README.md
3,265 bytes
SHA-256 d7b7bf6eff96727c…
KESPA-NOTE-020 — Always-On Orchestration Overhead in the Locked A/B/C Runtime Run
Date: 2026-09-04 Status: Verified — user environment Project: KESPA AI / NexLabs Studios
Purpose
The final A/B/C benchmark established the quality-versus-compute result.
This note preserves a narrower architecture finding from the raw runtime execution:
What did the then-current full KESPA orchestration path actually do per request, and how much compute did that behavior cost?
The answer became one of the reasons V2 moved toward adaptive inference rather than always-on orchestration.
Locked execution
The runtime experiment used:
- 54 locked evaluation cases
- 3 arms
- 162 total executions
- 162/162 successful executions
The three paths were:
- A — raw local model
- B — verified atomic retrieval fast path
- C — then-current full KESPA orchestration
Raw latency and energy
| Arm | Mean latency | Mean GPU energy | | --- | ---: | ---: | | A — raw local model | 3.757 s | 0.2450 Wh | | B — verified retrieval | 1.675 s | 0.1085 Wh | | C — full orchestration | 9.066 s | 0.5759 Wh |
Relative to B, C used approximately:
- 5.413× the latency
- 5.308× the GPU energy
B was also approximately:
- 55.4% lower latency than A
- 55.7% lower GPU energy than A
What Arm C actually did
Across 54 cases, Arm C recorded:
- total LLM passes: 65
- RAG searched: 54/54
- RAG actually used: 28/54
- planner used: 8/54
- reviewer used: 3/54
- inference avoided: 0/54
That profile is important.
The system paid for broad orchestration infrastructure on every request even though:
- retrieved knowledge was used on only about half the cases;
- planner/reviewer stages were needed on a minority of cases; and
- no evaluation case avoided inference entirely.
Later quality context
The later blinded quality stage reported:
- A quality: 20.71
- B quality: 54.22
- C quality: 27.71
- B case wins: 39/54
So the expensive C path did not recover enough quality in this workload to justify its runtime cost.
This is not a claim that orchestration is useless.
It is a claim that:
orchestration should earn its compute.
V2 architecture consequence
The raw runtime profile supported a shift away from:
every request -> full orchestration
toward:
trusted fast path -> escalate only when signals justify more compute
That is the conceptual bridge from the V1 C arm to the later D adaptive-inference experiment.
The model remains available for difficult cases.
The architectural change is that KESPA should not automatically pay for every available reasoning stage on every request.
Safety / isolation
The benchmark changed no production state:
- Production Chroma: NO
- Brain modification: NO
- database writes: NO
- private memory: OFF
- external web/cloud: OFF
- trust callback: OFF
Limitations
This was one model, one RTX 3070-class environment, one verified corpus, and one frozen 54-case workload.
The result should therefore be read as evidence against indiscriminate orchestration in this tested architecture, not as a universal result against multi-stage reasoning.
TEXT
SHA256SUMS.txt
393 bytes
SHA-256 ac56660136668b1f…
d7b7bf6eff96727ca8c76dc808d7a9b4c48f3d08e40a91452c6ffd5cdc55f879 README.md
a74583ba7a1d6014ed82310b6ee6d550fcc097914443ad7866c90dec82ef05d5 note.json
b33525a0678eebae21f3dadb6c0abfdf35697cbc755d01abae0a45d8da394361 metrics.csv
89c75a3a092ea07de1237a6ab193997b1c581009090cec9a8f38c833bf16ba2b methodology.md
6beae122e34e2244d00ce2f1ce8f93c23eb42f6975966f42b80f718ce89ecd8e provenance.json
MARKDOWN
methodology.md
1,101 bytes
SHA-256 89c75a3a092ea07d…
Methodology
Frozen workload
The raw A/B/C execution used the 54-question locked evaluation set.
All three arms executed the same cases under the same benchmark program.
Compute observation
The runtime harness recorded per-arm latency and GPU-energy measurements.
Arm C additionally exposed orchestration counters for LLM passes, RAG search/use, planner use, reviewer use, and inference avoidance.
Isolation
Private memory, external web/cloud retrieval, and trust callbacks were disabled for the locked benchmark.
No production Chroma, Brain, or database writes were performed.
Interpretation sequence
The raw runtime result was interpreted conservatively before quality judging: latency and energy alone were not treated as proof of superiority.
After blinded quality judging completed, the raw compute profile could be interpreted alongside the separately measured quality scores.
Derived values
Latency/energy multiples and percentage reductions in this public note are arithmetic derived from the recorded mean values. They are not additional benchmark measurements.
CSV
metrics.csv
797 bytes
SHA-256 b33525a0678eebae…
metric,value,unit_or_status
locked_cases,54,cases
system_executions,162,executions
successful_executions,162/162,executions
A_mean_latency,3757,ms
A_mean_energy,0.2450,Wh
B_mean_latency,1675,ms
B_mean_energy,0.1085,Wh
C_mean_latency,9066,ms
C_mean_energy,0.5759,Wh
C_total_llm_passes,65,passes
C_rag_searched,54/54,cases
C_rag_used,28/54,cases
C_planner_used,8/54,cases
C_reviewer_used,3/54,cases
C_inference_avoided,0/54,cases
C_vs_B_latency_multiple,5.413,x
C_vs_B_energy_multiple,5.308,x
B_vs_A_latency_reduction,55.4,percent
B_vs_A_energy_reduction,55.7,percent
A_quality_later_blinded_judge,20.71,composite
B_quality_later_blinded_judge,54.22,composite
C_quality_later_blinded_judge,27.71,composite
production_chroma_changed,NO,boundary
brain_modified,NO,boundary
database_writes,NO,boundary
JSON
note.json
3,025 bytes
SHA-256 a74583ba7a1d6014…
{
"schema": "kespa.public_technical_note.v1",
"id": "KESPA-NOTE-020",
"title": "Always-On Orchestration Overhead in the Locked A/B/C Runtime Run",
"date": "2026-09-04",
"status": "verified",
"purpose": "Isolate the raw runtime/compute behavior of the then-current full KESPA orchestration arm from the larger A/B/C quality benchmark and record why adaptive compute became a V2 design objective.",
"benchmark_shape": {
"locked_evaluation_cases": 54,
"arms": 3,
"system_executions": 162,
"completed_executions": "162/162"
},
"arm_definitions": {
"A": "raw local model",
"B": "verified atomic retrieval fast path",
"C": "then-current full KESPA orchestration"
},
"raw_compute": {
"A": {
"mean_latency_ms": 3757,
"mean_gpu_energy_wh": 0.245
},
"B": {
"mean_latency_ms": 1675,
"mean_gpu_energy_wh": 0.1085
},
"C": {
"mean_latency_ms": 9066,
"mean_gpu_energy_wh": 0.5759,
"total_llm_passes": 65,
"rag_searched_cases": "54/54",
"rag_used_cases": "28/54",
"planner_used_cases": "8/54",
"reviewer_used_cases": "3/54",
"inference_avoided_cases": "0/54"
}
},
"derived_comparisons": {
"C_vs_B_latency_multiple": 5.413,
"C_vs_B_energy_multiple": 5.308,
"B_vs_A_latency_reduction_percent": 55.4,
"B_vs_A_energy_reduction_percent": 55.7
},
"later_quality_context": {
"A_quality": 20.71,
"B_quality": 54.22,
"C_quality": 27.71,
"B_case_wins": "39/54",
"C_hit_at_1": "27/54",
"C_gold_micro_recall_percent": 22.02,
"note": "These quality results were produced by the later blinded judging stage; they are included only to interpret whether the measured orchestration cost bought corresponding quality."
},
"architecture_finding": [
"Searching RAG on every case did not imply using retrieved knowledge on every case.",
"The full orchestration path added multiple inference passes yet avoided inference on zero evaluation cases.",
"The verified-retrieval path was materially faster and lower-energy than both the raw model and full orchestration in this workload.",
"The result supported moving orchestration from an always-on default toward selective/adaptive escalation."
],
"safety_boundary": {
"production_chroma_changed": false,
"brain_modified": false,
"database_writes": false,
"private_memory": false,
"external_web_cloud": false,
"trust_callback": false
},
"limitations": [
"This profile reflects one local model, one GPU, one 54-case workload, and the orchestration implementation that existed on 2026-09-04.",
"Runtime/energy alone cannot establish answer quality; quality context comes from the separately completed blinded judge.",
"The result does not imply orchestration is intrinsically bad; it shows that orchestration cost must be justified selectively.",
"Later adaptive-inference work changed the compute policy and should not be conflated with Arm C."
]
}
JSON
provenance.json
1,499 bytes
SHA-256 6beae122e34e2244…
{
"schema": "kespa.public_provenance.v1",
"research_id": "KESPA-NOTE-020",
"source_basis": "Historical KESPA A/B/C raw-execution and blinded-quality benchmark records from the project changelog/Jira export.",
"source_observations": [
"The raw benchmark completed 162/162 executions across 54 locked cases and three arms.",
"Mean runtime/energy were A 3.757 s / 0.2450 Wh, B 1.675 s / 0.1085 Wh, and C 9.066 s / 0.5759 Wh.",
"Arm C executed 65 total LLM passes, searched RAG 54/54, used RAG 28/54, used planner 8/54, reviewer 3/54, and avoided inference 0/54.",
"The later blinded quality benchmark reported A=20.71, B=54.22, C=27.71 and B won 39/54 cases.",
"Production Chroma, Brain, and database state were not changed by the benchmark."
],
"derived_values": [
{
"metric": "C_vs_B_latency_multiple",
"formula": "9.066 / 1.675",
"value": 5.413
},
{
"metric": "C_vs_B_energy_multiple",
"formula": "0.5759 / 0.1085",
"value": 5.308
},
{
"metric": "B_vs_A_latency_reduction_percent",
"formula": "(1 - 1.675 / 3.757) * 100",
"value": 55.4
},
{
"metric": "B_vs_A_energy_reduction_percent",
"formula": "(1 - 0.1085 / 0.2450) * 100",
"value": 55.7
}
],
"publication_note": "This note publishes aggregate benchmark behavior and architecture interpretation. Raw private benchmark payloads, private implementation code, credentials, and user data are not included."
}