# Production AI System Service Review

## Review identity

| Field | Value |
| --- | --- |
| Workflow/system and versions | — |
| Review period | — |
| Supported segment and automation/autonomy | — |
| Operational, technical, risk, and service owners | — |
| Business sponsor or owner and independent backup | — |
| Last release and incident | — |
| Continuation mechanism, owner, and decision date | External renewal / internal funding or sponsorship / other |
| Decision required | Expand, continue, improve, constrain, pause, or retire |

## Operating responsibility

Record who can make each decision and the latest evidence that the role can exercise it. Shared platform or delivery capacity can support these owners; it does not silently absorb their authority.

| Role | Named owner and backup | Decision authority | Last exercised evidence | Open gap or temporary substitution | Action and due date |
| --- | --- | --- | --- | --- | --- |
| Workflow owner | — | Process boundary, accepted outcome, exclusions, and exceptions | — | — | — |
| Business metric owner and independent verifier | — | Baseline, denominator, attribution, acceptance, and value readback | — | — | — |
| AI service owner | — | Service lifecycle, SLO, support, change, and retirement | — | — | — |
| Operational owner | — | Day-to-day operation, on-call, incident coordination, reviewer capacity, and degraded-service decisions | — | — | — |
| Technical owner | — | Architecture, implementation, dependencies, and recovery | — | — | — |
| Data, policy, security, and risk owners | — | Source authority, permitted use, control, loss, and escalation decisions | — | — | — |
| Release, rollback, and kill-switch authorities | — | Exact release promotion, containment, and recovery | — | — | — |
| Operator, reviewer, and support route | — | Correction, exception, escalation, and stop path | — | — | — |

One person may hold the service and operational roles only when both assignments are explicit and every required independent verifier, release, risk, and receiving-service approval remains separate.

## Current sponsor readout

Generate this view from dated interaction receipts, effect or evaluation evidence, and the current governed records. Freeze the evidence cutoff and source digests. Do not maintain a parallel status narrative.

| Question | Current answer | Source receipt, record, or digest | Owner |
| --- | --- | --- | --- |
| What outcome and workflow boundary are currently accepted? | — | — | — |
| What changed since the last review? | — | — | — |
| What is measured, estimated, contested, or still unknown? | — | — | — |
| Which consequential conflict or decision is open? | — | — | — |
| What is the next field or operating move, by whom, and by when? | — | — | — |
| What remains excluded or unproved? | — | — | — |

This readout is a projection, not a new source of truth, approval, or acceptance record. Reconcile it with the cited source when a receipt is missing, stale, disputed, or outside the engagement boundary.

## Outcome and value

| Metric | Baseline | Target | Current | Eligible denominator | Confidence | Owner |
| --- | ---: | ---: | ---: | ---: | --- | --- |
| Accepted outcome | — | — | — | — | — | — |
| Business value | — | — | — | — | — | — |
| Realized residual loss | — | — | — | — | — | — |
| Cost per accepted outcome | — | — | — | — | — | — |
| Guardrail metric | — | — | — | — | — | — |

Explain attribution, non-adoption, downstream rework, avoided loss evidence, realized residual loss, fixed cost, and any rebaseline. Deduct residual loss separately only when it is not already netted from avoided loss or unit value.

### Agent-work cost drivers when applicable

Keep full cost per accepted outcome as the decision measure. Use the factor chain only to explain model-mediated spend changes; reconcile it with tool, compute, storage, wait, retry, human-review, recovery, and allocated-service cost.

| Driver | Prior | Current | Explained effect | Evidence / fixed comparison | Owner |
| --- | ---: | ---: | ---: | --- | --- |
| Eligible users | — | — | — | — | — |
| Sessions per user | — | — | — | — | — |
| Turns per session | — | — | — | — | — |
| Requests per turn | — | — | — | — | — |
| Tokens per request | — | — | — | — | — |
| Price per token | — | — | — | — | — |
| Other service cost and unexplained residual | — | — | — | — | — |

Record whether comparisons held the representative workload, model, route, and enforced budget constant. Do not attribute a mixed change to one lever.

## Declared adoption contract

Compare the current result with the pre-pilot contract; do not silently redefine the metric after observing performance.

| Metric revision | Eligible denominator and exclusions | Baseline value / as-of / status | Target | Guardrail and limit | Measurement window | Authoritative event/query source and revision | Owner | Contract drift |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — | — | — | None / approved rebaseline / unapproved |

## Reliability, safety, and recovery

| SLI or event | Objective | Current | Budget/breach | Segment | Action |
| --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — |

Include unauthorized, prohibited, duplicate, effect-unknown, and readback-mismatch events even when the count is zero.

## Adoption and human review

| Funnel or guardrail measure | Current | Trend | Slice | Observed cause from representative cases | Capacity, product, or risk decision |
| --- | ---: | --- | --- | --- | --- |
| Eligible workflow opportunities | — | — | — | — | — |
| Exposed in the usable surface | — | — | — | — | — |
| Completed or explicitly dispositioned | — | — | — | — | — |
| Independently accepted | — | — | — | — | — |
| Verified business effect | — | — | — | — | — |
| Override/edit/reject and abandonment | — | — | — | — | — |
| Approval acceptance and unsafe sample | — | — | — | — | — |
| Reviewer wait and workload | — | — | — | — | — |
| Support contacts and training gaps | — | — | — | — | — |

Assign the first broken transition to an owning layer and dated response. Do not prescribe training until access, routing, integration, data quality, product friction, behavior quality, policy, and reviewer capacity have been checked.

## Evaluation and behavior

| Suite/claim | Version manifest | Trials/metric | Result and uncertainty | Saturation/contamination | Decision |
| --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — |

List production behavior clusters, new regressions, evaluator calibration changes, and open gaps.

### Human-AI deployment qualification

Use this section when service reliability depends on an oversight or routing policy. Compare the live operating point with the exact qualified policy; a changed reviewer path, policy, behavior, context, tool, environment, or population may require requalification.

| Policy/version and routing signal | Qualification partition and sampling unit | Reliability target / observed / lower bound | Autonomous coverage / review burden | Reviewer effectiveness basis / latency / capacity | Total cost per case | Drift and decision |
| --- | --- | --- | --- | --- | ---: | --- |
| — | — | — | — | — | — | — |

State whether evidence came from terminal replay or a policy-in-loop rerun. Assumed reviewer effectiveness remains a limitation until measured in the declared operating population. [R26-85](https://github.com/davidahmann/applied-ai-field-guide/blob/main/research/2026-09-03--workflow-proof-and-deployment-qualification.md#r26-85)

### Scorer and observation review

| Scorer/claim/version | Eligible runs, sample, and exclusions | Trace and human-interaction sources | Label authority, calibration, and disagreement | Cost and reviewer burden | Accepted-outcome effect / decision |
| --- | --- | --- | --- | ---: | --- |
| — | — | — | — | — | — |

Automated scores, reviewer comments, corrections, and overrides can identify a failure or improvement candidate. They do not authorize effects, become anonymous ground truth, or prove improvement without a matched evaluation and the ordinary release path. Record privacy and reuse authority for human-interaction evidence. [R26-81](https://github.com/davidahmann/applied-ai-field-guide/blob/main/research/2026-08-28--warp-self-improving-software-factories.md#r26-81)

### Direct and downstream contract health

Use this section when a prediction, classification, ranking, retrieval result, or model proposal feeds another component or customer-visible decision.

| Component and version | Direct contract / metric / result | Downstream decision / metric / result | Calibration or drift by slice | Replay or experiment evidence | Decision |
| --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — |

### Zero-value work and improvement candidates

| Signal | Volume and cost | Accepted-outcome or reliability effect | Containment | Candidate change | Evaluation / canary / rollback |
| --- | ---: | --- | --- | --- | --- |
| Repeated searching, unnecessary turns, oversized results, model-visible polling, unused tool schemas, avoidable cache misses, or overpowered routing | — | — | — | — | — |

Trace-derived papercuts may create sanitized replay cases and candidate skill, prompt, routing, or tool changes. They do not update production behavior, protected evaluators, or release thresholds without independent evaluation and ordinary change approval. [R26-77](https://github.com/davidahmann/applied-ai-field-guide/blob/main/research/2026-08-28--uber-production-ai-operating-lessons.md#r26-77)

## Changes and dependencies

| Change/dependency | Version or lifecycle date | Evidence | Canary/soak | Rollback | Owner |
| --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — |

## Receiving-team operating capability

| Capability | Receiving owner | Last exercise/evidence | Open gap | Exit or remediation decision |
| --- | --- | --- | --- | --- |
| Support and escalation | — | — | — | — |
| Incident response and reconciliation | — | — | — | — |
| Evaluation and release | — | — | — | — |
| Policy, data-context manifest, source/schema/permission/quality/lineage/drift, capability, and selected-mechanism change | — | — | — | — |
| Rollback and recovery | — | — | — | — |
| Cost, value, and capacity review | — | — | — | — |
| Retirement and state disposition | — | — | — | — |

Compare open gaps with the [customer enablement handoff](https://davidahmann.github.io/applied-ai-field-guide/study/templates/customer-enablement-handoff/). A recurring gap needs an owner, exercise, and due date; delivery-team availability is not a substitute for the receiving customer's or internal team's operating capability.

## Continuation and sponsor resilience

| Signal or dependency | Current evidence | Trend | Owner / backup | Decision or action |
| --- | --- | --- | --- | --- |
| Business outcome still matters | — | — | — | — |
| Sponsor or business-owner continuity | — | — | — | — |
| Receiving service ownership | — | — | — | — |
| External renewal or internal funding readiness | — | — | — | — |
| Delivery-team dependency and exit readiness | — | — | — | — |

Continuation signals route attention and planning. A contract, budget, sponsor, reference, or high usage level does not independently prove an accepted outcome or realized value. If this workflow is part of a multi-service program, link its evidence into the [workflow portfolio review](https://davidahmann.github.io/applied-ai-field-guide/study/templates/workflow-portfolio-review/) without replacing this service-level decision.

## Incidents and reconciliation

| Incident/signal | First divergent state or invariant | Root cause/owner | Reconciliation | Regression | Product disposition |
| --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — |

## Field learning

| Learning ID | Incident/signal | Recurrence | Evidence/regression | Confidentiality | Destination | Product owner | Disposition | Validation status |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| — | — | — | — | — | — | — | — | — |

Maintain detailed records in the [field-learning register](https://davidahmann.github.io/applied-ai-field-guide/study/templates/field-learning-register/). Link the incident and reconciliation record instead of duplicating sensitive evidence here.

## Decisions and actions

| Decision/action | Evidence | Owner | Due | Verification |
| --- | --- | --- | --- | --- |
| — | — | — | — | — |

Record the final scope and automation/autonomy decision, improve-or-retire disposition, conditions, dissent, next review, and automatic rollback triggers. A retire decision starts the [owned retirement sequence](https://davidahmann.github.io/applied-ai-field-guide/study/playbooks/03-operate-and-scale/#10-run-the-improve-expand-or-retire-sequence) and is not complete until the target software release record carries verified shutdown evidence. For a model/agent release, record that evidence in a new solution release under [`solution-release.retirement_evidence`](https://github.com/davidahmann/applied-ai-field-guide/blob/main/schemas/solution-release.schema.json); deterministic, optimization, or classical-ML-only systems use equivalent target software retirement evidence.