AgentOps／AI-SRE

AgentはHTTP成功/失敗だけでは品質を測れない。モデル呼出し自体が成功しても、誤ったtool選択、partial completion、business-rule違反、長期trajectory崩壊、コスト暴走が起こり得る。公開後更新では、従来SREを捨てずにAI固有のquality/trajectory/evaluation signalsを重ねる運用モデルが必要になる。

Enterprise AI Core / operations / governance / observability / benchmark-evaluation / implementation / incident response

AgentOps/AI-SREは従来SREと別物なのか、それともSREのservice reliability control loopにAI固有SLIとevaluationを追加する拡張として設計すべきか。production agentの可靠性を単一success rateではなく何で管理すべきか。

• source class: primary operations guidance exact title: Implementing SLOs author/org: Google SRE canonical URL: https://sre.google/workbook/implementing-slos/ publication/update/access: Google SRE Workbook; accessed 2026-09-14 version/revision: web edition locator: SLI specification vs implementation; success ratios; SLO and error budget construction evidence label: VERIFIED_SPEC / SOURCE_REPORT relation: support/context

• source class: primary operations guidance exact title: Example Error Budget Policy author/org: Google SRE / Steven Thurgood canonical URL: https://sre.google/workbook/error-budget-policy/ publication/update/access: policy dated 2018-02-19; accessed 2026-09-14 version/revision: web edition locator: release freeze/reliability work when error budget is exhausted; postmortem triggers evidence label: SOURCE_REPORT relation: support/context

• source class: primary operations guidance exact title: Alerting on SLOs author/org: Google SRE canonical URL: https://sre.google/workbook/alerting-on-slos/ publication/update/access: accessed 2026-09-14 version/revision: web edition locator: burn-rate based alerting and budget-consumption model evidence label: SOURCE_REPORT relation: support/context

• source class: author-maintained AI telemetry specification exact title: OpenInference Specification / Semantic Conventions author/org: Arize AI / OpenInference canonical URL: https://arize-ai.github.io/openinference/spec/ publication/update/access: accessed 2026-09-14 version/revision: current docs locator: traces across LLM/tool/retrieval/agent operations; human/LLM/code evaluator feedback; evaluation/annotation attachment evidence label: VERIFIED_SPEC / OSS_PRACTICE relation: support

• source class: author-maintained implementation-gap report exact title: Add metrics support to Anthropic & Groq GenAI instrumentations #98 author/org: OpenTelemetry opentelemetry-python-genai contributors canonical URL: https://github.com/open-telemetry/opentelemetry-python-genai/issues/98 publication/update/access: opened 2026-05-14; accessed 2026-09-14 version/revision: issue #98 locator: cited signal coverage differs across provider instrumentations evidence label: SOURCE_REPORT / OSS_PRACTICE / PRODUCT_VOLATILE relation: limit

• source class: research / reproducible benchmark proposal exact title: ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions author/org: Aayush Gupta canonical URL: https://arxiv.org/abs/2601.06112 publication/update/access: 2026-01-03; accessed 2026-09-14 version/revision: arXiv 2601.06112 locator: repeated-run consistency, semantic perturbation robustness, tool/API fault injection; reliability surface R(k, ε, λ) evidence label: RESEARCH relation: support/context

• source class: research exact title: How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making author/org: Shubhra Mittal canonical URL: https://arxiv.org/abs/2609.01660 publication/update/access: 2026-08-31; accessed 2026-09-14 version/revision: arXiv 2609.01660 locator: long-horizon degradation; step-count reliability framing; benchmark-to-production horizon gap evidence label: RESEARCH relation: support/counter

• source class: research exact title: Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability author/org: Harsh Raj et al. canonical URL: https://arxiv.org/abs/2605.10516 publication/update/access: 2026-05-11; accessed 2026-09-14 version/revision: arXiv 2605.10516 locator: distinction between capability and execution robustness; trajectory-level consistency under semantically preserving perturbations evidence label: RESEARCH relation: support/context

• AgentOps/AI-SRE should be framed as an extension of SRE, not a replacement. Classical availability, latency, saturation, dependency health, change control, SLOs and error budgets remain necessary because agents still run on ordinary distributed systems.
• AI-specific reliability needs additional outcome layers: task/end-state success, business-rule compliance, tool/retrieval correctness, refusal/safety behavior, trajectory consistency, human escalation rate, token/cost, and evaluation quality. A successful HTTP/model response is only transport/service success.
• Keep SLI specification separate from SLI implementation. “Correctly completed customer task” is an outcome specification; an LLM judge score is only one possible measurement implementation and can itself drift or fail.
• Evaluation telemetry should attach to the execution trace/session so operators can move from a degraded aggregate metric to the exact model/tool/retrieval trajectory. EA-051 provides the interoperability substrate for that linkage.
• Reliability cannot be represented by one pass@1 or aggregate success metric. The cited 2026 research motivates repeated-run consistency, perturbation robustness, controlled dependency-failure tests and task-horizon sensitivity as separate axes.
• Long-horizon workflows need explicit reliability budgets or step/trajectory risk analysis. Even when per-step reliability is high, dependent multi-step execution compounds failure risk; benchmark horizons must not be assumed representative of production horizons.
• Error-budget logic can be adapted to AI outcomes, but only after an auditable SLI exists. Example: “percentage of in-scope claims-processing runs ending in policy-compliant terminal state within 120 s” is budgetable; “average judge score looks good” is insufficient without threshold, population and measurement reliability.
• Incident response must preserve the full provenance needed for replay/diagnosis: prompt/config/model revision, tools and tool versions, retrieved evidence identifiers, policy version, trace, evaluator version and external dependency state. Exact content retention remains privacy-governed.

• Google SRE guidance is general service-operations guidance, not an AI-agent standard. Its value is the control-loop model, not direct proof of AI-specific thresholds.
• 2026 agent reliability papers are research evidence; they do not establish universal production failure rates or product guarantees.
• LLM-as-judge or automated evaluator output is measurement, not ground truth. Evaluator model/version/prompt changes can create apparent SLI movement without product behavior change.
• Offline benchmark gains may not survive real permissions, data freshness, rate limits, UI drift, business rules, or long task horizons.
• Provider telemetry gaps can make nominally identical SLO calculations non-comparable across stacks.
• Retry can improve user-visible success while increasing latency/cost and masking per-attempt instability; report logical-run and attempt-level metrics separately.

• Production trace/evaluation linkage requires sufficient instrumentation coverage from EA-051.
• Ground-truth or high-quality adjudication may require human review samples, domain validators or deterministic end-state checks.
• Organization policy determines whether prompts/outputs/tool payloads can be retained for incident analysis.
• Provider/model version changes require re-baselining when they materially alter quality or cost distributions.

Status: operational pattern is implementable now; universal AI-SRE standard is not established. Recommended minimum control loop: 1. Define user/business outcome SLIs first. 2. Add classical service SLIs: availability, latency, dependency/tool error, queue/saturation. 3. Add AI execution SLIs: valid terminal state, tool/retrieval correctness, policy compliance, escalation/refusal, trajectory length/retry, token/cost. 4. Attach offline/online evaluations to traces with evaluator provenance. 5. Set SLOs/error budgets only for measures with defensible denominators and measurement procedures. 6. Alert on burn/degradation; freeze or canary risky model/prompt/tool changes when agreed reliability budget is exhausted. 7. Run postmortems with trajectory/provenance evidence and convert findings into regression/fault-injection tests.

Use a portfolio rather than one score: repeated execution; semantically equivalent perturbations; dependency fault injection (timeouts/rate limits/partial responses/schema drift); long-horizon tasks; cost/latency distribution; policy/end-state validators. Report benchmark version, task horizon and environment. Do not turn research benchmark numbers into production SLO targets without local measurement.

Security/privacy: incident traces can contain the most sensitive AI data. Apply least-retention, redaction and access control; preserve hashes/IDs/version metadata when payloads cannot be retained. Audit: evaluator identity/version/prompt, policy version and model/tool/retrieval versions must accompany quality decisions. Cost: treat token/tool/retry spend as operational SLIs where economically material; reconcile telemetry-derived costs against billing sources. Operability: distinguish model-quality degradation from platform outage, provider outage, tool failure, retrieval/data failure, evaluator drift and policy change so the response owner is clear.

“AgentOps/AI-SRE is best treated as conventional SRE extended with AI-specific outcome and trajectory measurements. Availability and latency remain necessary but are insufficient: production agents also need task/end-state correctness, policy compliance, dependency behavior, repeated-run consistency, evaluation provenance and cost visibility. Automated evaluators are measurement instruments rather than ground truth, and benchmark pass rates should not be converted directly into production SLOs. Define auditable SLIs first, then apply SLO/error-budget, alerting, incident and continuous-improvement practices.”

• Use EA-051 to define the trace/evaluation event model required for operational SLIs.
• Create an SLI catalog template separating outcome specification, measurement implementation, denominator/population, evaluator version, threshold and owner.
• Add TEST_PROFILEs for repeated-run consistency, semantic perturbation, dependency fault injection and long-horizon regression.
• For each provider mapping, document telemetry blind spots before claiming comparable SLO coverage.
• Add incident taxonomy: platform/model/tool/retrieval/data/policy/evaluator/credential/GUI-browser dependency.

consensus terminology for AgentOps vs AI-SRE; validated local thresholds for quality SLOs; evaluator drift controls; how to budget compounded long-horizon risk; provider-equivalent telemetry coverage.

bounded initial packet complete. Reopen when local production data exists, EA-051 schema mapping is finalized, or a provider-specific operational profile is built.

---

AgentOps/AI-SRE is retained as an evidence-backed dictionary operating-model proposal extending conventional SRE with AI-specific outcome/trajectory/evaluation signals; no universal production thresholds are asserted. Automated evaluators are measurement instruments, not ground truth. Research benchmark numbers are not production SLO targets. Preserve telemetry dependency on EA-051 and all OPEN_UNKNOWN.