AI Systems Evidence · historical evaluations and local work
Useful answers, with evidence you can inspect.
Follow the product question from a visitor’s answer to its source. Explore how meaning shapes an answer, then inspect the comparison, implementation and costs.
Which implementation does this evidence cover?
Reproduced reference implementation: frozen source-only answering core and recovery fixtures.
Historical/private selector experiment: paired results on 120 now-consumed questions.
Experimental tool-agent work: empty-search trial; the trial stopped before fetch and selection.
Current portfolio/Guide experience: later interface and content changes, not automatically qualified by these scores.
Explore supporting evidence
Documented release · 7 October 2026
Which version does the evidence describe?
The later release uses model-selected evidence and application-controlled factual wording. The studies and downloadable reference below describe earlier candidates. Their scores belong to those experiments; current release usefulness and human comprehension and recall remain unmeasured.
Meaning in practice
Same fact. Different answer.
Check what a number means before using it to answer a question.
01 · REAL PUBLISHED ESTIMATE
Hours saved ≠ dollars saved.
Estimated 5,000+ annual work hours saved
Annualized estimate; measurement period, population, baseline and underlying operational inputs are not published.
THIS QUESTION FITS
“What outcome is published?”
The estimated annual hours above, for this workstream.
THIS NEEDS DIFFERENT EVIDENCE
“How much money did it save?”
No dollar result is published. This estimate also does not establish enterprise-wide savings or clinical outcomes.
Open the exact payer estimateSee Ola’s contribution
Definition, contribution and source contract
- Claim / evidence class
payer-annual-hours-saved· ESTIMATED OUTCOME- Entity / workstream
- Payer-response workflow automation
- Contribution
- Attributed to the payer-response automation workstream; not presented as an enterprise-wide result. Contributed to response taxonomy, workflow mapping, exception design, UAT and launch monitoring.
- Grain / unit / time basis
- Approved workstream only / work hours / annualized.
- Period / population / baseline
- Not published / Not published / Not published.
- Method / inputs
- Estimated annual work hours avoided through approved payer-response automations. Underlying operational inputs are not published. No salary or conversion rate is inferred.
- Approval / withdrawal
- Public-approved estimate. Withdrawal, invalid configuration or a changed evidence class suppresses the example.
Inspect the versioned meaning fields
Contract: HP-2026-10-07-R2/semantics-1. Reused from the governing claim, including approval.
["HP-2026-10-07-R2/semantics-1",{"value":"Estimated 5,000+","label":"annual work hours saved","finalWording":"Estimated 5,000+ annual work hours saved","status":"estimated","statusLabel":"ESTIMATED OUTCOME","productOrWorkstream":"Payer-response workflow automation","businessProblem":"Manual interpretation, documentation and routing created avoidable effort and inconsistent follow-through.","attribution":"Attributed to the payer-response automation workstream; not presented as an enterprise-wide result.","period":null,"population":null,"baseline":null,"measurementMethod":"Estimated annual work hours avoided through approved payer-response automations.","knownLimitation":"Annualized estimate; measurement period, population, baseline and underlying operational inputs are not published.","publicApproved":true}]02 · SYNTHETIC TEACHING FIXTURE
Count the whole task.
Useful answer + useful first source = useful outcome. Every original submission counts, including a failure before dispatch.
Invented submissions · fixed teaching fixture · not visitor, production or employer results.
- Submission 1Useful answer + source
- Submission 2Useful answer + source
- Submission 3Useful answer + source
- Submission 4Useful answer; wrong source
- Submission 5Failed before dispatch
Default: 3/5 = 60%. Dropping the failed submission gives 3/4 = 75%, a different denominator.
Inspect the rows and calculation rules
Fixture: synthetic:semantic-primary-five:v1. Unique original submissions in a fixed fixture; this table is its source and stays separate from the real claims registry. Filtering changes the population and denominator, not the definition. Cohorts are arbitrary labels; subgroup percentages are not averaged.
| Original submission | Cohort | Dispatched | Useful content | Useful first source | Primary useful |
|---|---|---|---|---|---|
| synthetic:01 | A | Yes | Yes | Yes | Yes |
| synthetic:02 | A | Yes | Yes | Yes | Yes |
| synthetic:03 | B | Yes | Yes | Yes | Yes |
| synthetic:04 | B | Yes | Yes | No | No |
| synthetic:05 | B | No | No | No | No |
The default includes three useful outcomes, one useful answer with the wrong source, and one pre-dispatch failure. Content alone is 4/5 = 80%; primary usefulness is 3/5. Neither the content-only nor dispatched-only result replaces the original-submission metric.
How the answer moves from meaning to evidence
A semantic model defines the entity, scope, unit, time basis and limits of a fact. The LLM, index, policy engine and JSON-LD each have different jobs.
- Define meaning
Approved business evidence establishes what is known and unknown.
- Interpret the question
Identify actor, workstream, action, scope and unit. A model proposes an interpretation.
- Select evidence
Retrieve eligible approved material for the request.
- Check applicability
Validate meaning, authority, source version, status and withdrawal.
- Render facts
The Guide compiles reviewed wording and essential qualifications.
- Optional assistant synthesis
An external assistant using MCP writes its own answer; source and qualification checks still apply.
This teaching contract does not guarantee arbitrary natural-language understanding.
Follow the A–D comparisons, Luna repairs and QA-to-production decisions →
Compare the research approaches
One visitor goal · separate approaches
Different ways to improve accurate, useful conclusions.
Two lines of work investigate this goal. Their candidates, questions and evaluation conditions are separate; their scores cannot be combined or treated as a progression.
| Approach | Question | Evidence and decision |
|---|---|---|
| Source-only retrieval versus optional model selection | Does selecting reviewed material with a model improve useful answers enough to justify its cost? | The frozen paired evaluation below supports keeping source-only as the recommended default for that frozen candidate. Inspect the paired evaluation. |
| Shared reference and answer-completeness repair | Can better reference selection preserve supported facts when a recommendation remains unconfirmed? | A separate repair program reported better local usefulness, but its 205-test candidate was rejected for two visitor-facing gaps and its later 242-check wave stopped during final-review preparation. A newer local experiment stopped before visitor answers because its evaluator referenced an earlier source checkout. Its semantic acceptance is unknown; it is separate from the hosted QA implementation. That snapshot did not establish production visitor outcomes; the later documented release does not establish human comprehension or recall. Historical reports retain their stated public-citation limits. |
My product strategy
Help visitors make sense of my work.
As a Lead PM, I’m optimizing this portfolio so visitors can choose a relevant case, understand its evidence and limitations, and decide what to investigate next. My goals are clearer comprehension, earned trust and worthwhile time examining evidence.
AI, voice, RAG and MCP are means to that goal. I assess each feature’s cost and marginal value, balancing truthful answers with useful completion, and independent assurance with delivery speed.
I apply my experience managing ambiguity and anticipating technical and operational roadblocks to turn platform constraints into clearer visitor experiences. Approved evidence and citations govern every material claim; withdrawals remove dependent claims, uncertainty stays visible, and budget, security and privacy controls remain enforced.
Implementation contribution and evaluation scope
I own product priorities, evidence standards, trade-offs and release decisions. Implementation, diagnostics and evaluation were agent-assisted; independent agent reviewers assessed answers, with separate adjudication where used.
The frozen evaluation identifies its candidate and conditions; the local repair reports disclose missing verification. Agent-proxy results establish the behavior observed in those tests; actual visitor comprehension, human screen-reader use and device speech behavior require separate validation.
Approach 1 · frozen selector evaluation
Would model selection improve useful answers?
Keep the simpler path for this experiment.
The model selector added cost and latency without improving useful answers overall.
Inspect the evaluationSource-only
Recommended80%useful answers
96 of 120 questions
Deterministic retrieval · zero model calls
Optional selector
Not preferred79.2%useful answers
95 of 120 questions
Model selects reviewed material
Previously unsuccessful answers became useful
Previously useful answers became unsuccessful
useful answer across the paired evaluation
Earlier local experiment · same 120 questions per path · independent agent scoring. The source-only path left 24 questions unresolved.
Does the model earn its place?
Visitors need answers that resolve their question and lead to the right evidence. A useful answer needs supported facts that resolve the requested question.
Bound the model’s authority.
I directed the design: reviewed sources supply facts; software controls eligibility, withdrawal, evidence limits and spending. The model selects reviewed material.
Inspect the test design and remaining work
How the test worked
Two independent authors created 60-question sets. The baseline returned 48 useful answers in each; the selector returned 48 and 47. Independent agent reviewers scored the same-question first answers, with adjudication for disagreements.
What still needs improvement
24 source-only answers were not useful. Next validation covers actual visitor outcomes, the current runtime, a completed tool-agent workflow and model comparison.
The production workflow case informed my approach to states, exceptions, ownership and controlled release. Its production history is separate from this assistant’s maturity.
Execution and authority
One workflow, bounded responsibilities.
Source and configuration verified in the 7 October 2026 release record; this diagram is not a live availability check.
- 01Visitor request
The application keeps the full question and issues stable request references.
- 02Approved evidence
Reviewed public facts, exact sources and material qualifications form the selectable catalog.
- 03Luna selects
gpt-6-luna selects evidence references, a permitted operation and request annotations. Factual prose stays in the application.
- 04Application checks
Validate references, applicability, source identity, withdrawal and required qualifications.
- 05Compile + present
Render approved factual wording, evidence limits and the first exact source.
Selection cannot be used?
Provider unavailable, spending blocked or selection rejected → use the source-only fallback. Unsupported parts remain explicit gaps with source navigation.
I own the evidence and release.
Approved facts, qualifications, withdrawals, spending limits and activation remain human-controlled. Voice was off in this release.
Expand the detailed architecture · six stages
Follow each handoff from question to answer. Open a stage to inspect its inputs, responsibilities, outputs and recovery limits.
01 · Request custody and interpretation
- Input
- The visitor’s full original question.
- Responsibility
- Application code owns request identity, exact coordinates and verbatim extraction. The selector interprets requested actors, cases, phases, conditions and deliverables using application-issued references.
- Output
- Request-bound selections and annotations for the requested tasks.
- Failure / limit
- A valid whole-question reference proves custody, not completion. Missing clauses and changed premises still require semantic evaluation.
02 · Evidence catalog and source binding
- Input
- Reviewed public evidence and exact source identities.
- Responsibility
- Build selectable evidence references from approved material. Preserve source/version binding, maturity, attribution, essential qualifications and required companions.
- Output
- Eligible evidence choices tied to reviewed wording and sources.
- Failure / limit
- Private, unknown, withdrawn or incompatible evidence cannot become a public claim. A source match alone does not establish applicability.
03 · Metered Luna selection
- Input
- The original request, application-issued references, approved catalog and permitted operations.
- Responsibility
- The protected provider path applies spending and runtime controls. Luna selects evidence, a permitted operation, focal supports and first-source support. It returns structured selections, not public factual prose.
- Output
- A proposed selection plan; it has not yet earned rendering authority.
- Failure / limit
- Provider unavailability, blocked spending or unusable output leads to the validated source-only recovery path. No additional writer or judge call is part of this diagram.
04 · Contract and applicability checks
- Input
- The proposed plan and its bound catalog/source identities.
- Responsibility
- Check known references, operation compatibility, source status and withdrawal. Enforce required qualifications and companions. Match the selected evidence to the requested actor, phase and condition.
- Output
- An accepted plan or a specific rejection.
- Failure / limit
- Structural validity cannot guarantee full task completion. Wrong-story, wrong-phase and missing-consequence failures remain separate semantic evaluation targets.
05 · Compile the first answer and source
- Input
- Accepted evidence selections, approved factual text and essential qualifications.
- Responsibility
- Application code composes the first-visible answer and selects the task-relevant exact source. Related story detail can sit behind expansion; essential requested facts and limits belong in the first answer.
- Output
- Reviewed factual wording, material limits and inspectable source navigation.
- Failure / limit
- Unsupported requested parts remain explicit gaps. Expansion does not rescue an omitted essential answer or an irrelevant first source.
06 · Recovery, observation and release authority
- Input
- Provider/validation outcomes and the source-only answer path.
- Responsibility
- Retain fallback and original evaluation outcomes. Keep provider, compiler, browser delivery and source-readback observations distinct. I control evidence approval, withdrawals, spending, activation and release.
- Output
- A bounded answer or gap, with deployment and evaluation evidence recorded separately.
- Failure / limit
- A successful source check or deployment does not prove a useful visitor answer. The release record retained rollback v56, paused QA paid calls and kept voice off.
Scope: released v57 source/configuration record, 7 October 2026. These responsibilities describe the application contract; semantic completion still needs independently scored questions and first-source review.
Separate external-assistant path and configuration
External assistant ↔ read-only MCP search / fetch ↔ approved public evidence. This transport is separate from the Guide’s application-controlled answer.
Released selector: gpt-6-luna · medium reasoning · default service tier · 8192 maximum output tokens. Configuration: owner-selector-luna-c-v6-request-meaning-v1.
The model selects; application code validates and renders. Release verification established source and configuration custody, not a new production usefulness score.
Historical evaluated architecture · earlier source-only experiment
Solid flow: source-only baseline. Separate panels: optional experiments and external access.
- 01Visitor question
Clauses, cases, exclusions and evidence limits
- 02Approved evidence
Exact + lexical retrieval; eligibility and withdrawal checks
- 03Reviewed answer
Compose approved wording; preserve limits and gaps
- 04Validation
Check coverage, citations and source status
- 05Answer + exact source
Short answer, optional story and source navigation
Between retrieval and validation
Meter reservation → model selects reviewed IDs → validate → accept or recover to baseline.
Models do not supply public factual prose.
Search → fetch → select
One search round → up to six eligible fetches → final ID selection. At most three completions; 15-second overall deadline.
Observed trial: three empty searches, then stop. Fetch and final selection were not reached.
I control evidence and release
Reviewed facts, copy, withdrawals, activation, spending and QA/production promotion.
External MCP client ↔ public tools
Initialize → discover → search / fetch approved evidence. The Guide does not call the hosted MCP transport.
Across the evaluated workflow: content-free stage observations, exact source identities and validated recovery. No persistent visitor memory. Current QA has later changes and has not been re-scored against the frozen sets.
Inspect execution steps and the authority matrix
- Interpret requested clauses, cases, exclusions and limits.
- Retrieve eligible reviewed evidence using exact and lexical paths; check source, status and visibility.
- Compose a deterministic answer and preserve specific gaps.
- In the experimental selector path, optionally select approved cards; validate before returning reviewed wording.
- Return the first answer, visible limits, exact-source navigation and optional expanded story. Record content-free stage observations.
The evaluated application and downloadable reference import the same answering core. This page preserves that frozen evaluation; later QA applications have subsequent content and interface changes and have not been re-scored against these sets. Request-local state holds obligations, evidence and remaining execution budget; there is no persistent visitor memory. Vector embeddings and multiple agents are deferred: no demonstrated retrieval benefit justified adding them.
| Responsibility | Software / model authority | Human control |
|---|---|---|
| Public facts and story copy | Compose reviewed claims and citations; no model-authored facts. | I approve evidence, copy, qualifications and withdrawals. |
| Retrieval and answer | Read approved public evidence; answer, partial completion, specific gap or refusal. | Visitors inspect sources; reviewers judge useful completion. |
| Optional model | Select reviewed IDs within budget and validation; no automatic paid retries. | I control activation and spending. Paired value evaluation determines the recommendation. |
| Experimental tool agent | One search round, up to six fetched IDs returned by search, final ID selection; at most three completions and 15 seconds. No arbitrary URLs or writes. | Private experiment only; not activated inside the public Guide. |
| MCP | Typed, read-only public search/fetch tools. Guide tools share local implementations; the Guide does not traverse hosted MCP transport. | Client connection and authentication boundaries are qualified separately. |
| Release and consequential action | No publishing, outreach, hiring or clinical decisions. | I retain editorial, QA and production approval; protected release and rollback apply. |
Independent evaluation
Count every question, including unsuccessful answers.
Frozen evaluation boundary: These results apply to the frozen evaluation build shown here. Later interface/content changes have not automatically inherited these scores.
120 questions per path; agent scoring, not human outcomes.
7 useful recoveries
Previously unsuccessful baseline answers improved.
8 degradations
Previously useful baseline answers became unsuccessful.
Net gain: −1 useful answer. The paid selector did not earn preferred-path activation.
Local p95: 224 ms → 1,858 ms
Source-only vs selector; warm private execution, excluding browser/network and cold start.
111 calls · $0.0388617
Observed selector provider cost. Engineering, review and hosting costs excluded.
| Path | Set A | Set B | Total |
|---|---|---|---|
| Source-only | 48/60 | 48/60 | 96/120 · 80% |
| Optional selector | 48/60 | 47/60 | 95/120 · 79.2% |
My acceptance threshold was at least 41/60 useful answers in each set. Both paths passed that threshold. The selector failed its additional preferred-path criterion: positive combined gain and no lower score in either set. It recovered seven answers and degraded eight.
How usefulness was scored
Two separate authors sealed 60 ordinary-language questions each before execution. Two reviewers who did not author the sets or implement the candidate scored all 240 masked choices against independent expectations and exact passages. They assessed the first visible answer, separately visible limits and actual first source. Expanded story content cannot rescue an omitted first-answer requirement.
A useful answer must resolve the requested actions and clauses with supported facts and essential limits. Generic gaps, related but unresponsive prose, wrong case ownership and material unsupported claims fail. Every question remains in the denominator, including zero-call fallbacks. The scorers agreed on 232/240 decisions; a separate adjudicator resolved eight disagreements. These are agent reviews, not human validation.
Read the rubric and scoring method · Inspect all questions, observed answers, source passages and scores · Inspect provenance and summary
Recorded selector answers and original scores can be inspected. Re-running a current model or independently rescoring need not reproduce them. The credential-free runner reproduces source-only behavior; exact parity alone does not reproduce the judgment of usefulness.
Failure gallery: what failed and how it was detected
- Tenure attribution: an answer gave platform age and role but omitted the requested November 2023 tenure distinction. Paired blind scoring caught it; a reviewed fact alone was insufficient.
- Wrong case: an agent-count essay replaced a question about the performance case and production changes. Scorers checked entity and requested action against sources.
- Clause omission: hours savings did not complete a question asking whether a dollar conversion was supported. First-answer review retained the missing financial boundary.
- Privacy/configuration conflict: the export supplied static provider-off privacy text with a private provider-on answer. The contradiction remains in the original scores. Frozen code diagnosis identified omitted runtime source context; canonical public activation was not established. Even crediting that one answer would yield zero net selector gain.
- Empty agent search: the 20-task experiment stopped after three empty searches and three baseline fallbacks. No fetch or final selection executed. Offline diagnosis exposed compound-query discovery limitations; actual model query text was not retained, so its exact causal role is unknown.
- Challenger access: three HTTP 403 responses stopped the attempted two-model comparison. The comparison remains incomplete because challenger access failed.
All 120 questions, useful answers and failures are now consumed regression material. They remain regression material for future repairs. Recovery examples below cover simulated timeout, malformed selection and actual partial execution.
Historical V169 provider-off results were 43/60 and 44/60 on different sets. They are separate observations, not a same-question improvement or production trend.
Run the historical reference
Runnable implementation and contracts
Start without a credential.
Download the reference, extract it, and follow its README. It includes the typed client, contracts, exact source manifest and locked dependencies. The file-byte manifest identifies the packaged core, not a production deployment.
npm ci npm test npm run answer -- source-only "What demonstrates production workflow leadership?" npm run answer -- fixture-invalid-selection npm run answer -- fixture-timeout
Source-only mode and recovery fixtures need no key. Fixtures simulate failures and do not call a provider. To reproduce all 120 baseline answers, follow the reproduction instructions and use the credential-free runner with the consumed evaluation dataset.
HTTP and tool contracts
The same-origin Guide request is POST /api/evidence-guide with {"question":"…"}. Successful responses expose an accepted answer with reviewed claims, sources, limitations and optional opaque request identity. HTTP failures are distinct from successful responses containing a gap or partial answer. The downloadable TypeScript client defines the response union and error handling.
MCP search accepts {"query":"payer automation"}; fetch accepts an evidence id returned by discovery. Both return approved public material and canonical source routes. Initialization and tool discovery use MCP JSON-RPC; inspect the contracts and examples.
Production MCP endpoint: sampled initialization, discovery, search and fetch were observed October 2. Those HTTP requests were accepted without authentication; Authenticated client and ChatGPT plugin sessions require separate qualification. Source/version correspondence and client authentication are separate evidence boundaries.
Inspect the underlying artifacts
Evidence you can open
Inspect the artifacts, not just the claims.
Runnable reference
Frozen shared core, typed client, locked dependencies and source manifest. Run source-only and simulated recovery without an API key.
Why inspect it: Review how grounding and fallback are implemented. Download to run the historical source-only path without an API key.
Preview runnable reference
Evaluation dataset
Observed paired answers, expected passages, two independent agent scores and adjudication. Includes unsuccessful answers.
Why inspect it: Check the questions behind the scores; including unsuccessful answers. Download to inspect all source passages, reviewer scores and adjudication.
Preview evaluation dataset
Execution traces
Six sanitized examples. Inspect stage outcomes and evidence identities in the trace explorer below.
Why inspect it: Follow evidence through an execution and distinguish actual observations from simulated failures. Download for a closer trace-by-trace audit.
Preview execution traces
API / tool contracts
Guide request/response and read-only MCP search/fetch examples; transport and authentication limits.
Why inspect it: See what the Guide and public tools accept and return. Download when reviewing or building a compatible client.
Preview api / tool contracts
Decision + economics
Candidate identity, usefulness, calls, cost and local latency. Source-only recommended after negative selector gain.
Why inspect it: Check the denominators, cost boundaries and candidate identity behind the charts. Download the structured record for independent analysis.
Preview decision + economics
Reproduction runner
Reproduce every source-only answer with network calls forbidden. Use alongside the reference and dataset.
Why inspect it: Inspect what the runner checks before using it. Download to reproduce the recorded source-only answers alongside the reference and dataset.
Preview reproduction runner
Preview a contract: a question enters; reviewed evidence comes back
POST /api/evidence-guide
Content-Type: application/json
{"question":"What demonstrates production workflow leadership?"}
Response contract: accepted answer or explicit HTTP error.
Answer includes reviewed claims, sources and limitations.
An evidence gap is distinct from a transport failure.Open full contractsPreview the evidence packet: what you can independently inspect
- 120 authored questions with same-question source-only and selector answers.
- Expected source passages and requested actions, original scores and disagreements.
- Frozen implementation identity; reproduction compares full baseline answer objects.
- 24 unsuccessful source-only answers remain visible; this is not a repaired golden set.
Interactive walkthrough
Follow the evidence. See the product decision.
Step 1 of 5 · Goal
Start with a useful visitor decision.
Choose a relevant case, understand what its evidence supports, and decide what to investigate next.
What demonstrates production workflow leadership?
Success means a useful conclusion supported by the right case, with its qualifications visible.
Prefer a single document? Download the written walkthrough.
Read the failure analysis
Inspect the frozen experiment post-mortem
Approach 1 · frozen candidate 06226e4
The experiment improved the decision; the selector did not improve the result.
Source-only delivered 96/120 useful answers; optional selection delivered 95/120, with seven recoveries and eight degradations. The conclusion is no demonstrated incremental benefit on these sets. The conclusion applies to these questions and conditions. Inspect the answers and scoring rationales.
Where completion broke down
- 01 · REQUEST
A specific question
The visitor asks for a fact, relationship or next action.
- 02 · EVIDENCE
A valid source
Related approved material passes source checks.
- 03 · COMPLETION
The task can still fail
A requested clause is missing, the case is wrong, or the first source leads elsewhere.
The strongest supported shared failure was that source-valid selection and composition did not reliably preserve question completion through the first visible answer and first inspection route. Topic relevance, complete answers and useful citations require separate checks.
Trace the observed failure mechanisms
- Missing clauses: IA003 omitted the tenure distinction; SB010 omitted the dollar-conversion boundary; SB037 omitted revenue evidence.
- Wrong case or relationship: IA014 substituted an agent essay for the performance case; SB028 omitted the requested healthcare connection.
- Wrong first inspection route: IA016 opened ROI limits for a recovery question; SB009 opened a workflow model for savings; SB055 opened an outcome summary for video provenance.
- Generic gaps despite available evidence: IA011 concerned payer integration; SB043 concerned classification of agentic writing.
- Qualifications replacing explanation: IA027 supplied prototype restrictions while omitting requested mechanisms; SB032 did not resolve a specific renter-preference inference.
- Configuration mismatch: SB049 paired a provider-on answer with provider-off privacy text.
These observed mechanisms guide investigation. Exact code-path causes across the 24 baseline failures require separate diagnosis. Find these question IDs in the frozen dataset.
Keep three separate failures distinct
Privacy export: Static provider-off text conflicted with a private provider-on answer. No matching hosted HTML was captured; the original score remains. Crediting this answer would bring selector gain only to zero.
Bounded agent: Three empty searches triggered termination. Fetch and final selection never ran. Offline discovery limitations do not establish the actual model query’s causal role because that text was not retained.
Challenger access: Three HTTP 403 responses prevented comparison. Access failure does not establish model-quality inferiority.
Proposed corrections and the historical decision
What I would change
Preserve requested obligations through rendering, bind the first source to the primary requested action, protect essential baseline clauses during selection, and answer specific inference boundaries explicitly. Configuration-dependent disclosures need matching runtime identity; tool discovery should be checked offline before paid execution.
Historical proposal at this experiment’s freeze. First verify consumed failures and preserved useful-answer controls. Then use a wholly new independent challenge to assess the implemented change. Answer reproduction establishes implementation parity, not independent validation of meaning.
Decision: Retain source-only as the recommended mode, keep paid selection experimental and stop the unsuccessful agent attempt. Further paid acceptance should follow evidence that the shared completion mechanism improves without regressions.
This historical experiment remains separate from the local repair history. Subsequent private QA publication did not re-score the current answering runtime or close these 24 failures. Inspect frozen provenance, cost and measurement conditions.
Open the experiment history
Approach 2 · local experiment
A useful answer needs more than passing source checks.
- SOURCE CHECKS
332/332 passed
The six-question evaluation stopped at a source mismatch, before visitor answers.
Usefulness unmeasured - SEPARATE 205-TEST SNAPSHOT
Two visitor gaps
The supplied review rejected selection completeness and contradictory qualification handling.
Candidate rejected - SEPARATE 242-CHECK WAVE
Context exceeded its limit
Final-review preparation stopped before paired answers or usefulness judgments.
Evaluation not run
Separate historical snapshots, not a progression or combined score. The 205/242 reports retain their unverified public-citation limits; configuration reflects each recorded snapshot.
Inspect the experiment status and verification limits →
Inspect this experiment snapshot and three historical repair waves
Local experiment snapshot and historical repair limits
Local experiment snapshot · evaluator blocked
The preserved experiment snapshot identifies source 1e6a8a5dd1af864eef7a03ada0bc9970fba0ab01. It is not the implementation running in hosted QA and does not inherit the historical frozen selector scores.
332/332 local source checks passed. A fresh six-question evaluation contained 20 essential requirements. First decisions were preserved, but visitor answers, blind usefulness scoring and failure controls were NOT_RUN after an evaluator source mismatch was found. Source checks alone do not establish useful answers, improvement or release readiness.
First decisions are preserved and this question suite is retired. A corrected evaluator must be frozen before wholly fresh questions are authored. Inspect the sanitized experiment report. Configured-provider, hosted answering and voice qualification remain NOT_RUN.
Report SHA-256: 2f5d844e77e30f9cbbe9c2e5c0f5e01809058fc5420c5775b9c8dcbc77da418b.
Earlier repair snapshot · 197 tests · self-reported report
Local preview draft: These figures were supplied for this page revision. The candidate identifier and underlying review artifacts have not yet been verified for public citation. They do not update the frozen selector results.
- 197/197 affected implementation tests passed.
- Fresh blind review passed 5/6 tasks and 17/20 essential units.
- The report found no unsupported claims or citation failures.
QA held: An incorrect reference prevented a mixed control from running; another control lost a supported Nike adoption fact. No deployment occurred.
Next repair: Improve shared reference selection and retain accepted facts when recommendations fail. A fresh blind review must assess the repair; reusing failed questions provides regression evidence, not fresh proof of improvement.
Earlier evaluated repair snapshot · 205 tests · self-reported report
Local preview draft: This separate snapshot was supplied for frozen candidate 44f60493a9b254cb439bbd450de697a8f9df869d. Supporting source and evaluation artifacts await verification for public citation. Its six-task comparison is separate from the 197-test snapshot, the 242-check preparation wave and the frozen 120-question selector evaluation.
What was completed
Visitor tasks assess appropriate case selection, understanding of published evidence and qualifications, and identification of uncertainty. Review scores the visible answer, rather than retrieval alone.
Local safeguards preserve original question references, require approved source support, retain independently supported facts when recommendations fail review, remove withdrawn evidence and prevent duplicate factual claims.
The reported implementation passed 205 affected tests, TypeScript checking, and independent source and evaluation-harness review. Evidence withdrawal and restoration passed; selective rejection retained supported facts while withholding unsupported recommendation links.
| Measure | Candidate | Baseline |
|---|---|---|
| Visitor tasks passed | 5/6 | 0/6 |
| Essential requirements passed | 15/16 | 3/16 |
Independent agents authored questions and assessed anonymized answers against source-backed expectations. All three baseline-supported requirements were retained. Zero material unsupported claims or citation failures were identified in this review.
First valid fresh evaluation: rejected. Source and harness remained unchanged; first decisions, answers and independent judgments were preserved. The first blind judgment was frozen before unblinding with SHA-256 c720dc8a2845dc0bddec7c4774b18e42cfc8a1d3ddabfc34660b2303a390340c. This supplied identity is not a substitute for inspecting the judgment artifact.
Reported controls passed withdrawal/restoration, selective target rejection and conservation of supported own-source facts during a mixed final-review fault. One false qualification-gap answer remained. Typed references in that mixed fault were NOT_ELICITED, not a zero-failure pass.
Why the candidate was rejected
- CR06: the first binder selected only unit.renter-human-choice-model, covering ready/tie/abstain but omitting the apartment-comparison domain. Adequate public body cards were offered. Independent final review rejected the identity; no unsupported identity was published. This was selection completeness, not missing corpus evidence or a citation failure.
- CR01: after successful retrieval, fallback framed every static answer as unconfirmed requested evidence, including an independently answerable published qualification. The visible answer stated that qualification and then contradicted it. Mechanical checks had tested presence and empty fallback keys, rather than useful combined resolution.
Successor status at this snapshot: designed, not implemented. The proposed shared original-unit witness/resolution contract requires selected public bodies to cover explicit descriptive obligations; comparison metadata cannot supply missing facts. Independent fallback qualifications require complete live approved original-axis coverage, while choice, identity and dependent relationships remain unconfirmed. Independent design and capacity review precede implementation; no second candidate was admitted in this report.
Execution boundary: Zero actual provider API calls; first-answer execution used 48 mocked fetches. Configured-model reliability, 256-token fit, live budget durability, hosted QA and voice were NOT_RUN. QA remains held.
What remains, in order
- Independently check the design against existing response, token and evidence limits.
- Implement both corrections while preserving safeguards.
- Test known failures, varied questions, negative cases and evidence withdrawal.
- Review and freeze the source and evaluation harness.
- Run a wholly new blind evaluation; previous questions serve only as regressions.
- After local acceptance, complete applicable live-model and hosted validation before QA advancement.
- Measure actual visitor task completion, understanding of limitations and ability to choose a useful next investigation.
Historical wave · 242 checks · final review blocked
Self-reported local draft: Frozen candidate daed71a2694c482ab60ab2215c49323ce2048e98. The supplied wave report names supporting receipts, but their contents and public citation locations have not been verified here. This is separate from the earlier six-task results and frozen selector experiment.
What completed
242/242 affected implementation checks, TypeScript and whitespace checks passed. Independent source and harness reviews preceded freeze. Six fresh questions contained 19 essential units. First planner decisions, request reviews and evidence selections were sealed; schema, character-envelope and component checks passed.
The shared identity-component admission and selected-body accounting contract was implemented. Corpus, stage count, provider limits and fail-closed authority remained unchanged. Ordinary supported facts were retained while exact original-request completion remained visibly unverified where semantic completion failed.
Why final review stopped
CR03’s final-review context was 124,328 bytes, exceeding the unchanged 120,000-byte limit by 4,328 bytes. A 79,174-byte comparison inventory and repeated source/provenance content dominated the payload. The adapter stopped at semantic-input-too-large; no provider request was sent for that rejected context. Complete capture was not persisted because output was saved only after the loop.
A separate read-only audit found approved performance details in provenance spans but absent from offered card text. The binder preserved uncertainty rather than treating provenance as an answerable source body. This indicates a representation gap; it does not establish missing public evidence.
Evaluation stages still to run
NOT_RUN: paired visitor answers, pre-unblinding usefulness judgment, withdrawal/restoration and fault controls, configured-provider qualification, hosted QA and voice. Actual provider API calls: zero. Preparation failure is not a passing score or a measured fresh blind recurrence. QA and production advancement remain held.
Next bounded work
- Preserve approved original detail in usable source bodies with explicit custody and accurate limits.
- Normalize review transport around one canonical source inventory and indexed references. Preserve selected bodies, comparisons and permission relationships; independently verify equivalence and worst-case sizes within existing limits.
- Freeze a new candidate and use wholly new questions and first decisions. This wave’s decisions remain diagnostic evidence.
- Run first paired answers, independent usefulness judgment and every declared failure control.
- After local acceptance, separately qualify 256-token fit, provider reliability, latency, cost, durable metering, hosted QA and voice.
Proposed prevention: Test the complete final-review comparison and witness payload, rather than admission alone, with representative mixed requirements and approved-body coverage accounting. No prevention code was added to this frozen candidate.
These are agent UAT simulations, not human user research or external independent assurance. Limited disk space was a separate environment condition, not the cause of the deterministic context overflow.
Three different questions to test
Does the software behave?
Implementation checks test references, source eligibility and controls.
Does the answer complete the task?
Independent agent reviewers judge supported facts, missing clauses and the first source.
Does a visitor understand?
Human task studies and delayed recall assess comprehension. These outcomes remain unmeasured.
Judge methodology, trade-offs and next investment
How LLM-as-a-judge applies
I use LLM-based reviewers to assess answers against explicit visitor-task criteria and source-backed expectations. Fresh blind evaluation asks whether the visible answer completes the task, cites supporting evidence, preserves qualifications and avoids unsupported claims. Anonymized candidate and baseline answers reduce identification cues.
Implementation tests check software behavior; judge scores assess answer usefulness. An answer can contain supported facts and still fail through an omitted requirement or contradictory uncertainty label. This explains why zero unsupported claims can coexist with candidate rejection.
LLM judges can miss omissions and share model biases. Fresh questions, independent review and adjudication where used improve assurance. These judgments remain distinct from actual visitor validation. Judge models and procedures should be identified in the supporting reports; the historical repair snapshots await verified public citations.
The trade-offs
Truthfulness and answer completion must be assessed together. Strict evidence checks prevent invented connections but can also withhold an answer that adequate public evidence supports. Independent review improves assurance while adding delivery time and coordination cost; richer evidence contracts must fit existing token and request budgets.
Every material claim requires approved public evidence and supporting citations. Missing or withdrawn evidence removes dependent claims. Uncertain recommendations and relationships remain visibly unconfirmed. Security, privacy and spending controls apply to each approach.
Compare the next investment. Next validation covers typical visitors, the configured model and the hosted experience. Further repair should be compared with clearer case navigation and stronger source-only answers, using fresh visitor-task evidence.
Follow an execution trace
Sanitized trace explorer
Inspect execution, not just the answer.
6 examples. These are authored test executions, not visitor traffic.
Source-only answer · actual-source-only
Actual local source-only execution.
Terminal: answered. Network calls: 0. Structural coverage: no_explicit_gap; this is not a usefulness score.
Stage timings may overlap; do not add them to infer total duration.
- retrieval: ready-for-private-selection · 45 ms
- validation: source-valid · duration unmeasured
- terminal: answered · 347 ms
Inspect identities, evidence and validation
{
"schemaVersion": "guide-trace-v1",
"identity": {
"sourceCommit": "reference-sha256:80708175c1908a71ce985d708fc1c9f89d963d2c30a5da88ab9f8be962a5b3f7",
"corpusId": "8a7962dc10994703e3a21ff573c6f3527d59327cf89a85bcbda7c72521cf97ed",
"configurationId": "private-reviewed-case-card-sets-v20"
},
"model": "gpt-4o-mini-2024-07-18",
"durationMs": 346.86579199999994,
"events": [
{
"stage": "retrieval",
"outcome": "ready-for-private-selection",
"cardKeys": [
"unit.workflow-production",
"unit.career-healthcare-renter-contrast",
"unit.career-hca-about-tenure",
"unit.impact-platform-on-time-enhancements",
"unit.workflow-person-journey",
"workflow.pending-sync-followup",
"unit.workflow-first-read",
"unit.workflow-variants",
"workflow.variant-unity",
"unit.workflow-demo-data",
"unit.case-library-maturity",
"unit.workflow-platform-model",
"unit.impact-workflow-uat-compliance",
"unit.performance-worked-example",
"unit.performance-release-claim-status",
"unit.workflow-event-path",
"unit.performance-human-approval",
"workflow.order",
"workflow.permitted_action",
"unit.case-library-flagships",
"unit.workflow-uat",
"unit.performance-learning",
"unit.performance-problem",
"unit.career-about-foundation",
"unit.min-max-founder-fit-boundary",
"unit.min-max-product-os-boundary",
"unit.workflow-commercial-limit",
"unit.workflow-migration",
"unit.workflow-role",
"unit.workflow-tenure",
"workflow.commercial-limit",
"workflow.migration-methods",
"unit.partner-data-boundary",
"unit.performance-tradeoff",
"unit.renter-next-validation-gate",
"unit.workflow-rollout",
"unit.career-hca",
"unit.payer-ownership-boundary",
"unit.performance-release-notes",
"partner.pilot-study-gate",
"unit.career-current-role",
"unit.career-decade",
"unit.career-management-start",
"unit.career-people",
"unit.career-roles",
"unit.partner-pilot",
"unit.performance-inputs",
"unit.performance-maturity",
"unit.performance-release-audience",
"unit.performance-release-correction",
"unit.performance-tests",
"unit.renter-cost-rules",
"unit.renter-invalidation",
"unit.renter-prototype",
"unit.renter-save",
"unit.workflow-confidentiality",
"unit.workflow-eligibility",
"unit.workflow-expansion",
"unit.workflow-governance-tradeoff",
"unit.workflow-migration-dates",
"unit.workflow-migration-measurement",
"unit.workflow-onboarding-gate",
"unit.workflow-partial-sync",
"unit.workflow-precedence-limit",
"unit.workflow-problem",
"unit.workflow-resolution",
"workflow.governance-evidence-limit",
"workflow.migration-activation",
"workflow.migration-destination",
"workflow.migration-disruption-limit"
],
"durationMs": 45.1697079999999
},
{
"stage": "validation",
"durationMs": null,
"outcome": "source-valid",
"evidenceIds": [
"facet:workflow-production",
"facet:career-hca-about-tenure",
"facet:workflow-problem",
"facet:workflow-platform-model",
"facet:workflow-governance-tradeoff",
"facet:workflow-migration-measurement",
"facet:workflow-commercial-limit"
],
"cardKeys": []
},
{
"stage": "terminal",
"durationMs": 346.86579199999994,
"outcome": "answered"
}
],
"meteredAttempts": 0,
"providerCalls": 0,
"fallbackReason": "provider-runtime-unavailable",
"validationErrors": [],
"coverageStatus": "no_explicit_gap",
"terminal": "answered"
}Partial answer · actual-source-only
Actual local source-only execution.
Terminal: partial. Network calls: 0. Structural coverage: partial; this is not a usefulness score.
Stage timings may overlap; do not add them to infer total duration.
- retrieval: ready-for-private-selection · 39 ms
- validation: source-valid · duration unmeasured
- terminal: partial · 313 ms
Inspect identities, evidence and validation
{
"schemaVersion": "guide-trace-v1",
"identity": {
"sourceCommit": "reference-sha256:80708175c1908a71ce985d708fc1c9f89d963d2c30a5da88ab9f8be962a5b3f7",
"corpusId": "8a7962dc10994703e3a21ff573c6f3527d59327cf89a85bcbda7c72521cf97ed",
"configurationId": "private-reviewed-case-card-sets-v20"
},
"model": "gpt-4o-mini-2024-07-18",
"durationMs": 313.353167,
"events": [
{
"stage": "retrieval",
"outcome": "ready-for-private-selection",
"cardKeys": [
"unit.integration-outcome-boundary",
"unit.integration-result-limit",
"unit.integration-role",
"unit.min-max-wedge-proof",
"unit.integration-standards",
"unit.career-hiring-public-passage-limit",
"unit.integration-contract",
"unit.integration-recovery",
"unit.min-max-product-os-boundary",
"unit.renter-revenue-boundary",
"unit.integration-problem",
"unit.integration-learning",
"unit.clinical-review-maturity",
"unit.integration-model-limit",
"unit.case-library-maturity",
"unit.integration-users",
"unit.integration-uat",
"unit.integration-edit-history",
"unit.integration-hold",
"unit.career-api",
"unit.delivery-prototype-proof",
"unit.workflow-production",
"unit.career-positioning-title",
"unit.data-quality-users",
"unit.performance-release-boundary",
"unit.payer-users",
"unit.writing-bottleneck-outcome-flow",
"integration.failure-file-trace-reconcile",
"unit.integration-confidentiality",
"unit.payer-rollout-scope",
"unit.integration-validation",
"unit.case-library-reading-key",
"payer.completion-rule-limit",
"integration.role-edit-review-history",
"unit.integration-duplicate",
"unit.integration-late-event",
"unit.integration-replay-contract",
"unit.min-max-incumbent-response",
"unit.payer-public-contract",
"unit.workflow-precedence-limit",
"unit.career-hca-about-tenure",
"unit.career-hiring-process",
"unit.clinical-review-outcome",
"unit.provider-collaborators",
"unit.provider-role",
"workflow.no_auto_override",
"unit.demo-evidence-boundary",
"unit.renter-thesis",
"unit.writing-bottleneck-lesson",
"unit.performance-outcome",
"unit.provider-users",
"unit.performance-public-boundary",
"unit.renter-moat",
"delivery.bottleneck-decision-card",
"delivery.revise-allocation-when-evidence-changes",
"performance.inspect-test-regress-approve",
"unit.delivery-model",
"unit.impact-statuses",
"unit.performance-ai-role",
"unit.performance-evidence-gap",
"unit.performance-inputs",
"unit.performance-worked-example",
"unit.writing-bottleneck-decision-card",
"unit.case-library-flagships",
"unit.payer-learning",
"unit.payer-next-signals",
"unit.workflow-first-read",
"unit.career-decade",
"unit.career-people-detail",
"unit.career-resume"
],
"durationMs": 38.62491699999998
},
{
"stage": "validation",
"durationMs": null,
"outcome": "source-valid",
"evidenceIds": [
"facet:integration-replay-contract",
"facet:integration-outcome-boundary",
"facet:integration-result-limit",
"facet:integration-contract",
"facet:integration-validation",
"facet:integration-problem",
"facet:integration-hold",
"facet:integration-recovery",
"facet:integration-uat"
],
"cardKeys": []
},
{
"stage": "terminal",
"durationMs": 313.353167,
"outcome": "partial"
}
],
"meteredAttempts": 0,
"providerCalls": 0,
"fallbackReason": "provider-runtime-unavailable",
"validationErrors": [],
"coverageStatus": "partial",
"terminal": "partial"
}Invalid selection recovery · simulated-provider-fixture
Simulated provider failure from the credential-free reference; not a hosted incident.
Terminal: source_only_fallback. Network calls: 0. Structural coverage: no_explicit_gap; this is not a usefulness score.
Stage timings may overlap; do not add them to infer total duration.
- retrieval: ready-for-private-selection · 33 ms
- provider: failed · 0 ms
- validation: source-valid · duration unmeasured
- terminal: source_only_fallback · 286 ms
Inspect identities, evidence and validation
{
"schemaVersion": "guide-trace-v1",
"identity": {
"sourceCommit": "reference-sha256:80708175c1908a71ce985d708fc1c9f89d963d2c30a5da88ab9f8be962a5b3f7",
"corpusId": "8a7962dc10994703e3a21ff573c6f3527d59327cf89a85bcbda7c72521cf97ed",
"configurationId": "private-reviewed-case-card-sets-v20"
},
"model": "gpt-4o-mini-2024-07-18",
"durationMs": 286.417791,
"events": [
{
"stage": "retrieval",
"outcome": "ready-for-private-selection",
"cardKeys": [
"unit.workflow-production",
"unit.career-healthcare-renter-contrast",
"unit.career-hca-about-tenure",
"unit.impact-platform-on-time-enhancements",
"unit.workflow-person-journey",
"workflow.pending-sync-followup",
"unit.workflow-first-read",
"unit.workflow-variants",
"workflow.variant-unity",
"unit.workflow-demo-data",
"unit.case-library-maturity",
"unit.workflow-platform-model",
"unit.impact-workflow-uat-compliance",
"unit.performance-worked-example",
"unit.performance-release-claim-status",
"unit.workflow-event-path",
"unit.performance-human-approval",
"workflow.order",
"workflow.permitted_action",
"unit.case-library-flagships",
"unit.workflow-uat",
"unit.performance-learning",
"unit.performance-problem",
"unit.career-about-foundation",
"unit.min-max-founder-fit-boundary",
"unit.min-max-product-os-boundary",
"unit.workflow-commercial-limit",
"unit.workflow-migration",
"unit.workflow-role",
"unit.workflow-tenure",
"workflow.commercial-limit",
"workflow.migration-methods",
"unit.partner-data-boundary",
"unit.performance-tradeoff",
"unit.renter-next-validation-gate",
"unit.workflow-rollout",
"unit.career-hca",
"unit.payer-ownership-boundary",
"unit.performance-release-notes",
"partner.pilot-study-gate",
"unit.career-current-role",
"unit.career-decade",
"unit.career-management-start",
"unit.career-people",
"unit.career-roles",
"unit.partner-pilot",
"unit.performance-inputs",
"unit.performance-maturity",
"unit.performance-release-audience",
"unit.performance-release-correction",
"unit.performance-tests",
"unit.renter-cost-rules",
"unit.renter-invalidation",
"unit.renter-prototype",
"unit.renter-save",
"unit.workflow-confidentiality",
"unit.workflow-eligibility",
"unit.workflow-expansion",
"unit.workflow-governance-tradeoff",
"unit.workflow-migration-dates",
"unit.workflow-migration-measurement",
"unit.workflow-onboarding-gate",
"unit.workflow-partial-sync",
"unit.workflow-precedence-limit",
"unit.workflow-problem",
"unit.workflow-resolution",
"workflow.governance-evidence-limit",
"workflow.migration-activation",
"workflow.migration-destination",
"workflow.migration-disruption-limit"
],
"durationMs": 33.143249999999966
},
{
"stage": "provider",
"durationMs": 0.04679199999998218,
"outcome": "failed"
},
{
"stage": "validation",
"durationMs": null,
"outcome": "source-valid",
"evidenceIds": [
"facet:workflow-production",
"facet:career-hca-about-tenure",
"facet:workflow-problem",
"facet:workflow-platform-model",
"facet:workflow-governance-tradeoff",
"facet:workflow-migration-measurement",
"facet:workflow-commercial-limit"
],
"cardKeys": []
},
{
"stage": "terminal",
"durationMs": 286.417791,
"outcome": "source_only_fallback"
}
],
"meteredAttempts": 1,
"providerCalls": 1,
"fallbackReason": "provider-unknown",
"validationErrors": [],
"coverageStatus": "no_explicit_gap",
"terminal": "source_only_fallback"
}Provider timeout recovery · simulated-provider-fixture
Simulated provider failure from the credential-free reference; not a hosted incident.
Terminal: source_only_fallback. Network calls: 0. Structural coverage: no_explicit_gap; this is not a usefulness score.
Stage timings may overlap; do not add them to infer total duration.
- retrieval: ready-for-private-selection · 33 ms
- provider: deadline · 25 ms
- validation: source-valid · duration unmeasured
- terminal: source_only_fallback · 305 ms
Inspect identities, evidence and validation
{
"schemaVersion": "guide-trace-v1",
"identity": {
"sourceCommit": "reference-sha256:80708175c1908a71ce985d708fc1c9f89d963d2c30a5da88ab9f8be962a5b3f7",
"corpusId": "8a7962dc10994703e3a21ff573c6f3527d59327cf89a85bcbda7c72521cf97ed",
"configurationId": "private-reviewed-case-card-sets-v20"
},
"model": "gpt-4o-mini-2024-07-18",
"durationMs": 305.46633299999996,
"events": [
{
"stage": "retrieval",
"outcome": "ready-for-private-selection",
"cardKeys": [
"unit.workflow-production",
"unit.career-healthcare-renter-contrast",
"unit.career-hca-about-tenure",
"unit.impact-platform-on-time-enhancements",
"unit.workflow-person-journey",
"workflow.pending-sync-followup",
"unit.workflow-first-read",
"unit.workflow-variants",
"workflow.variant-unity",
"unit.workflow-demo-data",
"unit.case-library-maturity",
"unit.workflow-platform-model",
"unit.impact-workflow-uat-compliance",
"unit.performance-worked-example",
"unit.performance-release-claim-status",
"unit.workflow-event-path",
"unit.performance-human-approval",
"workflow.order",
"workflow.permitted_action",
"unit.case-library-flagships",
"unit.workflow-uat",
"unit.performance-learning",
"unit.performance-problem",
"unit.career-about-foundation",
"unit.min-max-founder-fit-boundary",
"unit.min-max-product-os-boundary",
"unit.workflow-commercial-limit",
"unit.workflow-migration",
"unit.workflow-role",
"unit.workflow-tenure",
"workflow.commercial-limit",
"workflow.migration-methods",
"unit.partner-data-boundary",
"unit.performance-tradeoff",
"unit.renter-next-validation-gate",
"unit.workflow-rollout",
"unit.career-hca",
"unit.payer-ownership-boundary",
"unit.performance-release-notes",
"partner.pilot-study-gate",
"unit.career-current-role",
"unit.career-decade",
"unit.career-management-start",
"unit.career-people",
"unit.career-roles",
"unit.partner-pilot",
"unit.performance-inputs",
"unit.performance-maturity",
"unit.performance-release-audience",
"unit.performance-release-correction",
"unit.performance-tests",
"unit.renter-cost-rules",
"unit.renter-invalidation",
"unit.renter-prototype",
"unit.renter-save",
"unit.workflow-confidentiality",
"unit.workflow-eligibility",
"unit.workflow-expansion",
"unit.workflow-governance-tradeoff",
"unit.workflow-migration-dates",
"unit.workflow-migration-measurement",
"unit.workflow-onboarding-gate",
"unit.workflow-partial-sync",
"unit.workflow-precedence-limit",
"unit.workflow-problem",
"unit.workflow-resolution",
"workflow.governance-evidence-limit",
"workflow.migration-activation",
"workflow.migration-destination",
"workflow.migration-disruption-limit"
],
"durationMs": 32.68591599999996
},
{
"stage": "provider",
"durationMs": 25.473874999999964,
"outcome": "deadline"
},
{
"stage": "validation",
"durationMs": null,
"outcome": "source-valid",
"evidenceIds": [
"facet:workflow-production",
"facet:career-hca-about-tenure",
"facet:workflow-problem",
"facet:workflow-platform-model",
"facet:workflow-governance-tradeoff",
"facet:workflow-migration-measurement",
"facet:workflow-commercial-limit"
],
"cardKeys": []
},
{
"stage": "terminal",
"durationMs": 305.46633299999996,
"outcome": "source_only_fallback"
}
],
"meteredAttempts": 1,
"providerCalls": 1,
"fallbackReason": "provider-timeout",
"validationErrors": [],
"coverageStatus": "no_explicit_gap",
"terminal": "source_only_fallback"
}Recorded private selector execution · actual-private-selector
One execution from the completed private acceptance run. Validated reviewed selection does not establish that this answer is useful.
Terminal: answered. Network calls: 1. Structural coverage: no_explicit_gap; this is not a usefulness score.
Stage timings may overlap; do not add them to infer total duration.
- retrieval: ready-for-private-selection · 23 ms
- provider: parsed-json · duration unmeasured
- provider: selection-returned · 1857 ms
- validation: source-valid · duration unmeasured
- terminal: answered · 2109 ms
Inspect identities, evidence and validation
{
"schemaVersion": "guide-trace-v1",
"identity": {
"sourceCommit": "8918e7eb747e23b413000ec3e2f117345f322c6727560a4454b6093f15ac2af5",
"corpusId": "8a7962dc10994703e3a21ff573c6f3527d59327cf89a85bcbda7c72521cf97ed",
"configurationId": "private-reviewed-case-card-sets-v20"
},
"model": "gpt-4o-mini-2024-07-18",
"durationMs": 2109.412292,
"events": [
{
"stage": "retrieval",
"outcome": "ready-for-private-selection",
"cardKeys": [
"unit.workflow-platform-model",
"unit.workflow-expansion",
"unit.workflow-feedback",
"unit.impact-uat-adoption-boundary",
"unit.integration-model-limit",
"unit.performance-worked-example",
"unit.workflow-operating-changes",
"unit.career-tool-monitoring",
"unit.workflow-next-signals",
"unit.workflow-production",
"unit.workflow-rollout",
"unit.writing-agents-lesson",
"unit.provider-concept",
"unit.nike-confidentiality",
"unit.integration-recovery",
"unit.leadership-toolkit-summary",
"unit.product-prioritization",
"provider.concept",
"renter.ai-delivery-boundary",
"unit.career-resume",
"unit.career-role-fit",
"unit.integration-learning",
"unit.workflow-governance-tradeoff",
"unit.workflow-migration",
"workflow.governance-evidence-limit",
"unit.workflow-collaborators",
"unit.workflow-confidentiality",
"unit.workflow-eligibility",
"unit.workflow-first-read",
"unit.workflow-migration-dates",
"unit.workflow-partial-sync",
"unit.workflow-precedence-limit",
"unit.workflow-resolution",
"unit.workflow-users",
"workflow.commercial-limit",
"provider.model-gates-trace",
"provider.model-no-match",
"provider.model-review",
"provider.no_live_release",
"provider.override",
"provider.override_denial",
"unit.career-current-role",
"unit.career-hca",
"unit.career-hca-about-tenure",
"unit.career-management-start",
"unit.career-people",
"unit.leadership-tool-release",
"unit.payer-lifecycle-ownership",
"unit.performance-human-approval",
"unit.performance-tradeoff",
"unit.product-exception-method",
"unit.provider-model",
"unit.provider-uat",
"unit.renter-next-validation-gate",
"unit.renter-prototype",
"unit.renter-validation",
"unit.workflow-commercial-limit",
"unit.workflow-current-status",
"unit.workflow-demo-data",
"unit.workflow-dependency-readiness",
"unit.workflow-migration-measurement",
"unit.workflow-onboarding-gate",
"unit.workflow-person-journey",
"unit.workflow-problem",
"unit.workflow-risk-visibility",
"unit.workflow-role",
"unit.workflow-tenure",
"unit.workflow-variants",
"unit.writing-bottleneck-lesson",
"workflow.migration-activation"
],
"durationMs": 23.02045800000019
},
{
"stage": "provider",
"outcome": "parsed-json",
"inputTokens": 2784,
"outputTokens": 16,
"durationMs": null
},
{
"stage": "provider",
"durationMs": 1857.3782079999999,
"outcome": "selection-returned"
},
{
"stage": "validation",
"durationMs": null,
"outcome": "source-valid",
"evidenceIds": [
"proposition:unit.workflow-platform-model",
"proposition:case-status.workflow",
"proposition:unit.workflow-migration",
"proposition:unit.workflow-expansion",
"proposition:unit.workflow-operating-changes",
"proposition:unit.workflow-rollout"
],
"cardKeys": [
"unit.workflow-platform-model",
"unit.workflow-expansion",
"unit.workflow-operating-changes",
"unit.workflow-rollout"
]
},
{
"stage": "terminal",
"durationMs": 2109.412292,
"outcome": "answered"
}
],
"meteredAttempts": 1,
"providerCalls": 1,
"fallbackReason": null,
"validationErrors": [],
"coverageStatus": "no_explicit_gap",
"terminal": "answered"
}Empty search and source-only recovery · actual-private-agent-experiment
The first agent trial search returned no results. No fetch or final model selection executed; the 20-task trial stopped after three tasks.
Terminal: source_only_fallback. Network calls: 1. Structural coverage: not-a-usefulness-score; this is not a usefulness score.
Stage timings may overlap; do not add them to infer total duration.
- search-usage: observed · 1164 ms
- search: completed · 1165 ms
- search-tool: completed · 49 ms
- terminal: no-search-results · 0 ms
Inspect identities, evidence and validation
{
"schemaVersion": "agent-stage-export-v1",
"identity": {
"sourceCommit": "8918e7eb747e23b413000ec3e2f117345f322c6727560a4454b6093f15ac2af5",
"corpusId": "8a7962dc10994703e3a21ff573c6f3527d59327cf89a85bcbda7c72521cf97ed",
"configurationId": "private-bounded-agent-experiment"
},
"terminal": "source_only_fallback",
"coverageStatus": "not-a-usefulness-score",
"events": [
{
"stage": "search-usage",
"outcome": "observed",
"durationMs": 1164,
"model": "gpt-4o-mini-2024-07-18",
"inputTokens": 186,
"outputTokens": 11,
"meteredAttempts": 1,
"providerCalls": 1
},
{
"stage": "search",
"outcome": "completed",
"durationMs": 1165,
"meteredAttempts": 1,
"providerCalls": 1
},
{
"stage": "search-tool",
"outcome": "completed",
"durationMs": 49,
"evidenceIds": [],
"meteredAttempts": 1,
"providerCalls": 1
},
{
"stage": "terminal",
"outcome": "no-search-results",
"durationMs": 0,
"meteredAttempts": 1,
"providerCalls": 1
}
]
}Download sanitized traces. Questions, audio, transcripts, fingerprints, credentials and provider request IDs are absent.
Inspect the delivery map
Historical delivery map
The experiment, then the production release.
- 1
Frame
Useful resolution with inspectable sources; I set limits.
- 2
Build
Reviewed evidence + deterministic baseline + optional selector.
- 3
Challenge
Two independent 60-question sets, blind scores and adjudication.
- 4
Decide
96 baseline vs 95 selector; keep source-only as recommendation.
- 5
Expose
Reference, failures, traces, economics and contracts on this page.
- 6
Historical QA
Hosted evidence page for the earlier selector experiment.
The 96/120 and 95/120 results belong to the earlier paired experiment.
Later release · 7 October 2026
The repaired Guide configuration reached production v57 through a separate release decision. The release record documents its evaluation, deployed source and configuration.
Read the production release record ↗Visitor research is the next step for understanding how people use the Guide.
Inspect model access and selection decisions
Model and architecture decision
Keep a model only when it earns its place.
- CONSUMED DEVELOPMENT BANK
32/40 vs 28/40
The selector looked promising on development questions.
- SEPARATE FRESH BANK AT FREEZE
95/120 vs 96/120
The selector did not produce a net gain in acceptance.
- MODEL ACCESS TEST
Three HTTP 403s
The challenger comparison was incomplete; access failure is not a model-quality result.
Different question banks and conditions. These scores are not a trend or a current-release model ranking.
Model identities, conditions and the decision
The pinned incumbent was gpt-4o-mini-2024-07-18. A challenger, gpt-4.1-mini-2025-04-14, returned three HTTP 403s; the head-to-head comparison remained incomplete. Incumbent operational availability is not a model-quality win.
A 40-question consumed development check favored the selector, 32/40 versus 28/40. Fresh acceptance did not confirm positive incremental value: 95 versus 96 useful answers. The measured decision for that frozen candidate was therefore source-only as the recommended default, with paid selection and the bounded tool agent experimental in that comparison.
This preserves low-cost task completion while forgoing seven observed model recoveries. It also avoids eight observed degradations. The 24 source-only failures remain open. A future paid path needs a traced shared repair and a fresh paired-value test; another keyword patch or replay of consumed questions is insufficient.
Inspect provider cost and local latency
Cost and latency
What did the extra step add?
Earlier frozen experiment · 120 questions per path · independent agent scoring
Seven recoveries, eight degradations: one fewer useful answer overall.
Provider cost excludes hosting, engineering, review and rework. Local timings are not production SLOs.
Open the dated source record →All figures, timing conditions and spending context
| Measure | Source-only | Selector acceptance |
|---|---|---|
| Useful answers | 96/120 | 95/120 |
| Metered attempts | 0 | 111 |
| Observed provider cost | $0 | $0.0388617 |
| Cost / useful answer | $0 provider cost | $0.000409 provider cost |
| Local p50 / p95 | 138 / 224 ms | 1,172 / 1,858 ms |
The source-only measurement reproduced all 120 baseline answers with provider and network invocation forbidden. It is one sequential warm-process run. Selector timing includes all 120 questions, including fallbacks. Both exclude browser rendering, network transit to a deployed site and cold startup. No statistical superiority, production reliability or adoption is inferred.
All 111 acceptance attempts had token usage observed. Cost per net additional useful answer is undefined because net gain was negative. Provider cost excludes hosting, engineering, review and rework. A cheaper call is not necessarily better product economics.
The agent experiment cost $0.00010395 for three attempts before stopping; it demonstrated no incremental agent value. Protected spending used an 800/month local attempt cap and separate $20 QA ceiling; hosted and local request ledgers were kept separate. No meter was reset. Download the dated economics and provenance record.
Inspect security and privacy controls
Inspect security controls and residual risks
Threat model and residual risk
Controls have an explicit boundary.
Protect approved facts, private material, credentials, visitor privacy and spending.
Prompt injection / fabricated claims
- Control
- Reviewed factual wording, eligibility and claim-source validation; models select IDs.
- Remaining limit
- Source validity does not guarantee relevance or complete clauses; independent meaning evaluation caught failures.
Unauthorized tools / exfiltration
- Control
- Read-only approved evidence, fetched IDs restricted to search returns in the agent experiment; reject unknown tools, arbitrary URLs and writes.
- Remaining limit
- Public MCP/client authentication is a separate boundary. Sampled unauthenticated protocol access is not an authenticated-client qualification.
Private content / visitor tracking
- Control
- Current workflow traces omit question text, transcripts, audio, fingerprints and question hashes. Authored evaluation datasets are explicitly separate.
- Remaining limit
- Optional provider processing has its own retention terms. Legacy trace paths require separate review.
Overspend / meter failure
- Control
- Durable reservation before calls, attempt/cost caps, exclusive evaluation lease, scope fence and no automatic paid retries. Meter failure prevents paid execution.
- Remaining limit
- Local and hosted ledgers are distinct; operator-attested fences are not automated global coordination.
Outage / malformed output / timeout
- Control
- Validated source-only recovery or specific gap; observer failures do not change answers.
- Remaining limit
- Circuit breaker is process-local, not fleet-wide. Production uptime requires hosted measurement.
Withdrawn or mismatched source
- Control
- Withdrawal and source/configuration identity checks.
- Remaining limit
- Runtime-dependent text must be exported with its configuration; the privacy export finding is preserved.
Unsafe release / misleading maturity
- Control
- Human editorial/release decisions, protected promotion and retained rollback; prototypes and simulations labelled.
- Remaining limit
- This review covers the Guide’s retrieval, answer construction and privacy controls.
Evaluation coverage
Did we retrieve the right evidence, and answer the question?
Explore four distinct dimensions. Existing visitor-task judgments cover relevance and support; this portfolio does not report four independently measured RAG scores.
Explore the four dimensions and failure patterns
Choose an evaluation dimension
Answer relevance
Does the visible answer resolve the question and its requirements?
Represented in task and essential-requirement reviewWrong-case answers and omitted clauses fail usefulness review.
Inspect: Compare each requested action, case and qualification with the first visible answer.
Supported but off-topic
Question → relevant source? → responsive answer?
Inspect case identity, reference selection and requested clauses. If the right evidence was retrieved, inspect answer composition.
Example: Frozen review: an agent-count essay answered a question about the performance case.
These paths suggest investigations; they do not identify a proven root cause. Faithfulness is grounding in context, distinct from factual correctness. Precision and recall depend on the chosen reference and scoring method. See the Ragas metric definitions.
Inspect the frozen rubric and dataset · Read the separate repair reports and their draft boundaries