Focus the investigation.
Protect the user’s task.

Evidence modules

Detailed product evidence Evidence that separates AI assistance from accountable decisions. Performance decision system Signals, AI assistance and human judgment remained separate. The workflow made the evidence path explicit so an AI-generated suggestion could not become a production change without review, controlled testing and approval. Performance signals Response-time distributions, error rates, throughput, database resource pressure, query plans, timeouts and user-visible workflow delay were reviewed as diagnostic inputs; the July UAT comparison has its own published values and conditions. AI-agent or LLM role Organize approved diagnostic evidence, compare patterns, identify missing information, propose hypotheses and draft test or communication artifacts with source references. Human role Validate the problem, inspect source evidence, decide which hypothesis merits testing, approve query or index changes, evaluate results and retain release and rollback authority. Query and indexing decisions Consider query shape, selective predicates, pagination, bounded reads and targeted indexes while checking redundancy, maintenance cost and downstream behavior. Read/write trade-offs Treat read improvement, write amplification, storage, index maintenance and operational complexity as one product decision rather than optimizing a single signal. Controlled testing Compare approved candidates in a controlled environment, use representative scenarios, check regressions and reject results that cannot be reproduced or traced to the tested version. Approval boundaries The LLM could recommend and draft. It could not execute production queries, create production indexes, change infrastructure, approve a release or suppress a failed guardrail. Monitoring Monitor the user-visible workflow, response distribution, errors, resource signals and rollback indicators after an approved release. Outcome status QUALITATIVE OUTCOME: improved diagnostic structure and release-decision confidence; the separate July team UAT comparison is published below. Generalized example A slow filtered worklist became a controlled query-and-index decision. In a generalized example, a filtered worklist showed inconsistent response times as data and filter combinations changed. The workflow assembled approved telemetry and a generalized query plan, then used an LLM to identify missing evidence and propose bounded-query and composite-index hypotheses. Decision path Engineering and product owners reviewed the hypotheses, selected the smallest credible candidates, compared them under representative read and write conditions and rejected any option that could not meet the defined guardrails. Boundary No autonomous query, index or production change occurred. Exact schemas, values, thresholds and implementation details remain confidential. Supporting use case AI-assisted release intelligence for stakeholder-specific communication. The same governed pattern transformed approved work items, acceptance criteria, UAT results and final release scope into audience-specific release communication without generating unsupported claims. Source traceability Every release statement referenced an approved work item, acceptance criterion, UAT result or version-controlled scope decision. Claim-status validation Drafts distinguished delivered behavior, validated results, known limitations and intended follow-up work. Confidentiality review Internal identifiers, sensitive workflow details and unsupported implementation specifics were removed before distribution. Human approval Product and release owners reviewed the communication for accuracy, audience fit and operational risk before publication. Version control The communication was tied to the approved release scope and revised when that scope changed. Correction workflow A discovered mismatch returned the draft to source review, correction, revalidation and renewed approval rather than silent editing. The LLM drafted from approved evidence; it did not determine release scope or approve the communication.

Validate the problem, inspect source evidence, decide which hypothesis merits testing, approve query or index changes, evaluate results and retain release and rollback authority.

Treat read improvement, write amplification, storage, index maintenance and operational complexity as one product decision rather than optimizing a single signal.

Compare approved candidates in a controlled environment, use representative scenarios, check regressions and reject results that cannot be reproduced or traced to the tested version.

The LLM could recommend and draft. It could not execute production queries, create production indexes, change infrastructure, approve a release or suppress a failed guardrail. The team’s UAT comparison recorded mean request time of 0.910s on 28 June and 0.228s on 6 July 2026, 74.9% lower, with 1,000 virtual users and a configuration recorded as 80 CPUs. These are reported team UAT results. They do not isolate AI’s contribution or establish production latency, tail latency or an independently repeated result.

The team’s UAT comparison recorded mean request time of 0.910s on 28 June and 0.228s on 6 July 2026, 74.9% lower, with 1,000 virtual users and a configuration recorded as 80 CPUs.

These are reported team UAT results. They do not isolate AI’s contribution or establish production latency, tail latency or an independently repeated result.

The team’s UAT comparison recorded mean request time of 0.910s on 28 June and 0.228s on 6 July 2026, 74.9% lower, with 1,000 virtual users and a configuration recorded as 80 CPUs.

These are reported team UAT results. They do not isolate AI’s contribution or establish production latency, tail latency or an independently repeated result.

Measurement

Measurement framework Define success before development. The documented result is qualitative. Product maturity and limitations are explicit; AI’s isolated contribution is not measured. The team’s UAT comparison recorded mean request time of 0.910s on 28 June and 0.228s on 6 July 2026, 74.9% lower, with 1,000 virtual users and a configuration recorded as 80 CPUs. These are reported team UAT results. They do not isolate AI’s contribution or establish production latency, tail latency or an independently repeated result.

Confidentiality

This case study has been generalized to protect confidential information. Application names, schemas, actual query plans and query text, index names, data distributions, production latency values and monitoring thresholds, infrastructure details and production identifiers are generalized or omitted.

Reviewed product decisions

Require controlled comparisons and regression checks before accepting a performance recommendation.

Keep production approval, rollback readiness and monitoring thresholds under accountable human ownership.

Product decisions The decisions behind the workflow. Product maturity: Delivered AI-assisted product-operations workflow; implementation details are generalized; the July team UAT comparison is published Use the LLM to organize approved evidence and propose testable hypotheses, never to execute production changes or approve its own recommendation. Evaluate query and indexing changes as read/write trade-offs rather than treating faster reads as a free optimization. Require controlled comparisons and regression checks before accepting a performance recommendation. Keep production approval, rollback readiness and monitoring thresholds under accountable human ownership. Generate release communication only from approved work items, acceptance criteria, UAT evidence and the final release scope.

Context

Context and business problem Why the problem mattered. Application performance can degrade through the interaction of query shape, data growth, indexing, read patterns, write cost and release scope. Diagnostic evidence was distributed across telemetry, query plans, approved work items, acceptance criteria and UAT results. Business problem What had to change Teams needed a faster, more consistent way to connect performance signals to testable product and technical decisions without allowing an LLM to change queries, indexes, infrastructure or production behavior on its own.

Outcome summary

My role Product leader for performance prioritization, evidence requirements, test boundaries and release approval Product maturity Delivered AI-assisted product-operations workflow; implementation details are generalized; the July team UAT comparison is published Architecture Human-governed AI-assisted performance and release workflow Business problem Teams needed a faster, more consistent way to connect performance signals to testable product and technical decisions without allowing an LLM to change queries, indexes, infrastructure or production behavior on its own. Strategic decision Use the LLM to organize approved evidence and propose testable hypotheses, never to execute production changes or approve its own recommendation. QUALITATIVE OUTCOME AI-assisted analysis supported structured application-performance investigation and release communication while keeping query, test and production decisions under human control. Relevant capabilities AI-Assisted Product Operations · Application Performance · Data Products · Release Intelligence · Human-in-the-Loop

Delivered AI-assisted product-operations workflow; implementation details are generalized; the July team UAT comparison is published The team’s UAT comparison recorded mean request time of 0.910s on 28 June and 0.228s on 6 July 2026, 74.9% lower, with 1,000 virtual users and a configuration recorded as 80 CPUs. These are reported team UAT results. They do not isolate AI’s contribution or establish production latency, tail latency or an independently repeated result.

Users and stakeholders

Users and stakeholders The people making, supporting and governing the decision. Primary users Product and engineering teams Application support Operational users Release stakeholders Partnered with engineering, data, QA, application support and release stakeholders .

Model

Workflow, rules and product model The visible experience and the logic beneath it. 1 Performance-signal intake 2 LLM-assisted evidence synthesis 3 Query-plan and indexing hypothesis review 4 Read/write trade-off assessment 5 Controlled test comparison 6 Human approval boundaries 7 Post-release monitoring 8 Traceable release intelligence AI-assisted performance decision 01 Observe 02 Synthesize 03 Hypothesize 04 Test 05 Approve 06 Monitor Production boundary Human approval required · no autonomous changes

Lessons

Lessons and next decisions Use evidence to decide what expands next. “ AI assistance is most useful when it reduces evidence-fragmentation and speeds hypothesis formation while controlled testing and accountable people retain decision authority. ” Product lesson Current product maturity: Delivered AI-assisted product-operations workflow; implementation details are generalized; the July team UAT comparison is published 1 Improve longitudinal comparison of query and release signals 2 Strengthen automated evidence completeness checks 3 Evaluate diagnostic usefulness without collecting sensitive production content

Leadership

Cross-functional leadership Align domain expertise with technical delivery. Ola's published role: Product leader for performance prioritization, evidence requirements, test boundaries and release approval . Defined the performance problem, required evidence, AI-assistance boundaries, query and indexing decision criteria, controlled test plan, approval gates, monitoring expectations and stakeholder-specific release communication Aligned product and engineering on the user-visible performance problem Defined the signals and controlled evidence required for a decision Reviewed AI-generated hypotheses before they entered testing Connected acceptance criteria and UAT evidence to release readiness Required human approval and version-controlled correction paths

Constraints

Constraints and complexity The happy path was not the product. Timing, partial outcomes, missing data, conflicting state and system boundaries shaped the product strategy from the beginning. CONSTRAINT 01 A faster read increases write or maintenance cost CONSTRAINT 02 A query-plan change helps one workflow but harms another CONSTRAINT 03 Test data does not represent production distribution CONSTRAINT 04 An LLM recommendation lacks source evidence CONSTRAINT 05 UAT passes while an operational performance guardrail regresses CONSTRAINT 06 Release scope changes after communication is drafted

Uat source

Average request response time 0.910s → 0.228s 74.9% lower · June 28 to July 6, 2026 1,000 virtual users. Test configuration recorded as 80 CPUs. Source: July 2026 internal performance update. The team’s July performance update records the release result. My contribution was to focus the investigation on the highest-impact queries, challenge unnecessary query behavior against how people searched, and keep the next change testable. Engineering refactored stored procedures. The update specifies a UAT environment with 1,000 virtual users and an 80-CPU configuration. These figures describe the June 28 and July 6 test runs. They measure the team’s release changes in that environment.

A generalized product story about using an AI-assisted workflow to organize application-performance evidence, generate testable hypotheses and improve release intelligence while product and engineering owners retained every query, test and production decision.