# Evaluation custody and usefulness rubric ## Custody Independent authors receive approved public sources and frozen corpus identity, not candidate answers or another author's fresh packet. Fresh packets remain outside the reference export, mode 0600, and are identified by file-byte SHA-256. Authors record provenance and no-execution status. Implementers and orchestrator receive only metadata before freeze; a designated runner opens packets only after candidate/configuration/model/budget are frozen. This is independent agent authorship, not independent human research. Authors do not score their own set. A packet becomes consumed when execution exposes its questions or answers to implementation work; retain the packet, delivered answers, both scores and adjudication. Do not rebrand consumed or paraphrased repaired prompts as fresh acceptance. No prompt or raw visitor content enters production telemetry. Evaluation questions are authored research material, with separate retention and publication permission. ## Unit and denominator The unit is one authored visitor question, including all requested clauses. Score the first-visible answer and accessible claim-level source links. Story expansion cannot repair a material omission in that answer. Preserve all 60 questions per set, including refusals, evidence gaps, timeouts and fallbacks; no selective deletion or denominator reduction. ## Useful first-visible answer Mark useful only when the visible answer: 1. Responds to every material supported part of the actual question, preserving requested actors, cases, exclusions, comparisons, actions and routes. 2. Makes the requested facts or decision explicit without requiring inference from merely adjacent story. 3. Uses approved active sources that support the actual material claim, with exact passage links where available. An approved more precise alias for the same passage can satisfy a route oracle; exact URL string equality is not a substitute for source meaning. 4. Includes essential maturity, attribution, numerical measurement and uncertainty limits when they materially change interpretation. 5. Identifies any requested private, unsupported or unavailable part specifically, and offers a useful supported route or next step when the source allows one. 6. Contains no material unsupported assertion or contradictory qualification. A grounded but irrelevant answer fails. A specific bounded refusal can pass when it directly resolves an unsupported request and supplies appropriate supported navigation. Mechanical status or source validity alone cannot determine usefulness. Do not require verbatim oracle prose, incidental adjacent facts, fixed word counts, or an unsolicited hiring verdict. ## Source and boundary scoring For each delivered arm, record useful boolean, requested-clause coverage, source-support assessment, first-source relevance/decisiveness, essential-limit completeness, and reason. First-source decisiveness is a separate metric except where exact-source navigation is explicitly requested or its failure materially prevents the requested resolution. Score provider-enabled privacy statements against the frozen activation configuration; conditional oracle statements must retain their condition. Classify defects as omission, wrong actor/case, false gap, relevance/compression, wrong requested action, attribution/measurement, maturity, exclusion, privacy, withdrawal, citation or infrastructure. Separate oracle error from product error by actual public source meaning; preserve both original scores and the adjudicated disposition. A material source or boundary failure makes that answer non-useful. Report invariant exercise outcomes separately; do not introduce a new zero-defect usefulness-only gate. ## Blind scoring and adjudication Two reviewers separately score randomized arm labels without model or baseline identity and without viewing the other score. Reviewers receive authored questions, source-grounded oracle and delivered first-visible answers/sources, not implementation internals or development repairs. A separate adjudicator resolves disagreements using the same rubric, records the rationale and keeps initial judgments immutable. Disclose that agent scorers are proxies; human comprehension, adoption and recall remain distinct evidence. ## Acceptance and paired gain Each independently authored 60-question set must reach at least **41/60 adjudicated useful first-visible answers**, matching the owner's revised >40/60 threshold. The provider/agent candidate additionally requires a positive combined useful gain over the same-question deterministic baseline and no lower useful count in either set. Report useful in both, recoveries, losses and not useful in either. Use the same frozen corpus and constraints in paired arms. This acceptance establishes usefulness on these agent-authored sets; it does not establish universal performance, production reliability, real-user value or a 56/60 target. Existing protected release requirements remain separate. If acceptance fails, preserve and consume the questions, diagnose a shared cause, use a bounded development experiment and an unsealed falsification challenge before any new sealed attempt. ## Cost and latency Reserve per completion, not per question. Preflight the canonical ledger, cumulative QA cost evidence, planned maximum attempts and retained acceptance reserve before execution. No retries outside the predeclared bound, meter reset or output overwrite. Record all failures and unknown usage, preserving unknown totals or conservative bounds. Report baseline/candidate useful counts, recoveries/losses, cost per useful task, cost per net useful gain only when positive, and p50/p95 end-to-end latency with measured denominator. ## Public export status The two authored 60-question sets are now consumed and published. Authors are anonymized as ReviewerA and ReviewerB; the separate agent scorers are ScorerA and ScorerB. They are agent proxies, not independent human research. The JSON retains both initial judgments, eight adjudication rationales, source passages, source oracle, and each recorded first-visible answer with separately visible limitations and source links. Expanded claims cannot repair a first-visible omission. Re-execution reproduces deterministic answers only; it does not reproduce semantic scoring or paid model responses.