{ "format": "agent-article-v0", "article_id": "EA-052", "canonical_url": "/tools/ea-052-agentops-ai-sre", "page_type": "reference", "claim_ids": [], "evidence_ids": [ "EVD-EA-052-URL-06C934AF218E", "EVD-EA-052-URL-0744228B978B", "EVD-EA-052-URL-01FA52816728", "EVD-EA-052-URL-6ACAC33977CC", "EVD-EA-052-URL-0F5EEBDD1974", "EVD-EA-052-URL-9C2E21D90560", "EVD-EA-052-URL-56E7946F43C6", "EVD-EA-052-URL-DA8E50A714DD", "EA-052-R11-RESEARCH", "EA-052-R11-WEB-READBACK" ], "unknown_ids": [ "EA-052-UNKNOWN-01", "EA-052-UNKNOWN-CURRENTNESS" ], "operational_object_ids": [], "provider": [], "models": [], "plans": [], "surfaces": [], "regions": [], "current_status": "UNKNOWN", "observation_status": "SPEC_ONLY", "freshness": "UNKNOWN", "last_verified": null, "supersedes": [], "superseded_by": [], "provenance": [ "COMM-052-EA0052", "SRC-EA-052-URL-06C934AF218E", "SRC-EA-052-URL-0744228B978B", "SRC-EA-052-URL-01FA52816728", "SRC-EA-052-URL-6ACAC33977CC", "SRC-EA-052-URL-0F5EEBDD1974", "SRC-EA-052-URL-9C2E21D90560", "SRC-EA-052-URL-56E7946F43C6", "SRC-EA-052-URL-DA8E50A714DD", "WEBBODY-EA0052-R8-3C2539DA" ], "article_sources": [ { "id": "EVD-EA-052-URL-06C934AF218E", "evidence_class": "RESEARCH", "title": "author/org: Google SRE", "href": "https://sre.google/workbook/implementing-slos/", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-06C934AF218E" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://sre.google/workbook/implementing-slos/" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-0744228B978B", "evidence_class": "RESEARCH", "title": "author/org: Google SRE / Steven Thurgood", "href": "https://sre.google/workbook/error-budget-policy/", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-0744228B978B" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://sre.google/workbook/error-budget-policy/" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-01FA52816728", "evidence_class": "RESEARCH", "title": "author/org: Google SRE", "href": "https://sre.google/workbook/alerting-on-slos/", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-01FA52816728" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://sre.google/workbook/alerting-on-slos/" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-6ACAC33977CC", "evidence_class": "RESEARCH", "title": "author/org: Arize AI / OpenInference", "href": "https://arize-ai.github.io/openinference/spec/", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-6ACAC33977CC" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://arize-ai.github.io/openinference/spec/" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-0F5EEBDD1974", "evidence_class": "RESEARCH", "title": "author/org: OpenTelemetry opentelemetry-python-genai contributors", "href": "https://github.com/open-telemetry/opentelemetry-python-genai/issues/98", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-0F5EEBDD1974" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://github.com/open-telemetry/opentelemetry-python-genai/issues/98" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-9C2E21D90560", "evidence_class": "RESEARCH", "title": "author/org: Aayush Gupta", "href": "https://arxiv.org/abs/2601.06112", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-9C2E21D90560" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://arxiv.org/abs/2601.06112" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-56E7946F43C6", "evidence_class": "RESEARCH", "title": "author/org: Shubhra Mittal", "href": "https://arxiv.org/abs/2609.01660", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-56E7946F43C6" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://arxiv.org/abs/2609.01660" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-052-URL-DA8E50A714DD", "evidence_class": "RESEARCH", "title": "author/org: Harsh Raj et al.", "href": "https://arxiv.org/abs/2605.10516", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-052-URL-DA8E50A714DD" ], "supporting_locators": [ "EVIDENCE TABLE", "canonical URL: https://arxiv.org/abs/2605.10516" ], "scope": "EVIDENCE TABLE", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EA-052-R11-RESEARCH", "evidence_class": "RESEARCH", "title": "EA-052 監査済みresearch / canonical revision binding", "supports": [ "- AgentOps/AI-SRE should be framed as an extension of SRE, not a replacement. Classical availability, latency, saturation, dependency health, change control, SLOs and error budgets remain necessary because agents still run on ordinary distributed systems.\n- AI-specific reliability needs additional outcome layers: task/end-state success, business-rule compliance, tool/retrieval correctness, refusal/safety behavior, trajectory consistency, human escalation rate, token/cost, and evaluation quality. A successful HTTP/model response is only transport/service success.\n- Keep SLI specification separate from SLI implementation. “Correctly completed customer task” is an outcome specification; an LLM judge score is only one possible measurement implementation and can itself drift or fail.\n- Evaluation telemetry should attach to the execution trace/session so operators can move from a degraded aggregate metric to the exact model/tool/retrieval trajectory. EA-051 provides the interoperability substrate for that linkage.\n- Reliability cannot be represented by one pass@1 or aggregate success metric. The cited 2026 research motivates repeated-run consistency, perturbation robustness, controlled dependency-failure tests and task-horizon sensitivity as separate axes.\n- Long-horizon workflows need explicit reliability budgets or step/trajectory risk analysis. Even when per-step reliability is high, dependent multi-step execution compounds failure risk; benchmark horizons must not be assumed representative of production horizons.\n- Error-budget logic can be adapted to AI outcomes, but only after an auditable SLI exists. Example: “percentage of in-scope claims-processing runs ending in policy-compliant terminal state within 120 s” is budgetable; “average judge score looks good” is insufficient without threshold, population and measurement reliability.\n- Incident response must preserve the full provenance needed for replay/diagnosis: prompt/config/model revision, tools and tool versions, retrieved evidence identifiers, policy version, trace, evaluator version and external dependency state. Exact content retention remains privacy-governed." ], "limit": "canonical監査・revision bindingの存在を示す。外部ソースcurrentnessが本文またはcontrol-planeで確定していない場合は、currentnessをUNKNOWNのまま保持する。", "origin_class": "AUDITED_CANONICAL_RESEARCH_PACKET", "source_or_experiment_ids": [ "COMM-052-EA0052" ], "supporting_locators": [ "COMM-052-EA0052", "EA-052-R5-LEGACY-MIGRATION-20261008-R1", "content/staging/EA-052.md" ], "scope": "監査済みEA本文・canonical revision・既存Source Graph relationの範囲。欠損currentnessは推測で閉じない。", "provenance_status": "AUDIT_PASS_CANONICAL_BINDING" }, { "id": "EA-052-R11-WEB-READBACK", "evidence_class": "IMPL", "title": "EA-052 Web body identity / canonical readback", "supports": [ "canonical staging bodyのSHA-256一致とR9 canonical ledger readbackをProduction projectionの入力として検証する。" ], "limit": "body identity/readbackは本文claim自体の真偽や外部ソースの最新性を追加保証しない。", "checked_at": null, "origin_class": "IMPLEMENTATION_READBACK", "source_or_experiment_ids": [ "WEBBODY-EA0052-R8-3C2539DA" ], "supporting_locators": [ "WEBBODY-EA0052-R8-3C2539DA", "content/staging/EA-052.md", "EA-052-R5-LEGACY-MIGRATION-20261008-R1" ], "scope": "R9/R10 canonical readback後のProduction生成時body identity。", "provenance_status": "CANONICAL_READBACK_BODY_HASH_MATCH" } ], "research_sections": [ { "heading": "WHY_THIS_MATTERS_TO_UPDATE", "text": "AgentはHTTP成功/失敗だけでは品質を測れない。モデル呼出し自体が成功しても、誤ったtool選択、partial completion、business-rule違反、長期trajectory崩壊、コスト暴走が起こり得る。公開後更新では、従来SREを捨てずにAI固有のquality/trajectory/evaluation signalsを重ねる運用モデルが必要になる。" }, { "heading": "SCOPE_ROUTING", "text": "Enterprise AI Core / operations / governance / observability / benchmark-evaluation / implementation / incident response" }, { "heading": "CLAIM_OR_RESEARCH_QUESTION", "text": "AgentOps/AI-SREは従来SREと別物なのか、それともSREのservice reliability control loopにAI固有SLIとevaluationを追加する拡張として設計すべきか。production agentの可靠性を単一success rateではなく何で管理すべきか。" }, { "heading": "EVIDENCE TABLE", "text": "1. source class: primary operations guidance\n exact title: Implementing SLOs\n author/org: Google SRE\n canonical URL: https://sre.google/workbook/implementing-slos/\n publication/update/access: Google SRE Workbook; accessed 2026-09-14\n version/revision: web edition\n locator: SLI specification vs implementation; success ratios; SLO and error budget construction\n evidence label: VERIFIED_SPEC / SOURCE_REPORT\n relation: support/context\n\n\n2. source class: primary operations guidance\n exact title: Example Error Budget Policy\n author/org: Google SRE / Steven Thurgood\n canonical URL: https://sre.google/workbook/error-budget-policy/\n publication/update/access: policy dated 2018-02-19; accessed 2026-09-14\n version/revision: web edition\n locator: release freeze/reliability work when error budget is exhausted; postmortem triggers\n evidence label: SOURCE_REPORT\n relation: support/context\n\n\n3. source class: primary operations guidance\n exact title: Alerting on SLOs\n author/org: Google SRE\n canonical URL: https://sre.google/workbook/alerting-on-slos/\n publication/update/access: accessed 2026-09-14\n version/revision: web edition\n locator: burn-rate based alerting and budget-consumption model\n evidence label: SOURCE_REPORT\n relation: support/context\n\n\n4. source class: author-maintained AI telemetry specification\n exact title: OpenInference Specification / Semantic Conventions\n author/org: Arize AI / OpenInference\n canonical URL: https://arize-ai.github.io/openinference/spec/\n publication/update/access: accessed 2026-09-14\n version/revision: current docs\n locator: traces across LLM/tool/retrieval/agent operations; human/LLM/code evaluator feedback; evaluation/annotation attachment\n evidence label: VERIFIED_SPEC / OSS_PRACTICE\n relation: support\n\n\n5. source class: author-maintained implementation-gap report\n exact title: Add metrics support to Anthropic & Groq GenAI instrumentations #98\n author/org: OpenTelemetry opentelemetry-python-genai contributors\n canonical URL: https://github.com/open-telemetry/opentelemetry-python-genai/issues/98\n publication/update/access: opened 2026-05-14; accessed 2026-09-14\n version/revision: issue #98\n locator: cited signal coverage differs across provider instrumentations\n evidence label: SOURCE_REPORT / OSS_PRACTICE / PRODUCT_VOLATILE\n relation: limit\n\n\n6. source class: research / reproducible benchmark proposal\n exact title: ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions\n author/org: Aayush Gupta\n canonical URL: https://arxiv.org/abs/2601.06112\n publication/update/access: 2026-01-03; accessed 2026-09-14\n version/revision: arXiv 2601.06112\n locator: repeated-run consistency, semantic perturbation robustness, tool/API fault injection; reliability surface R(k, ε, λ)\n evidence label: RESEARCH\n relation: support/context\n\n\n7. source class: research\n exact title: How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making\n author/org: Shubhra Mittal\n canonical URL: https://arxiv.org/abs/2609.01660\n publication/update/access: 2026-08-31; accessed 2026-09-14\n version/revision: arXiv 2609.01660\n locator: long-horizon degradation; step-count reliability framing; benchmark-to-production horizon gap\n evidence label: RESEARCH\n relation: support/counter\n\n\n8. source class: research\n exact title: Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability\n author/org: Harsh Raj et al.\n canonical URL: https://arxiv.org/abs/2605.10516\n publication/update/access: 2026-05-11; accessed 2026-09-14\n version/revision: arXiv 2605.10516\n locator: distinction between capability and execution robustness; trajectory-level consistency under semantically preserving perturbations\n evidence label: RESEARCH\n relation: support/context" }, { "heading": "FINDINGS", "text": "- AgentOps/AI-SRE should be framed as an extension of SRE, not a replacement. Classical availability, latency, saturation, dependency health, change control, SLOs and error budgets remain necessary because agents still run on ordinary distributed systems.\n- AI-specific reliability needs additional outcome layers: task/end-state success, business-rule compliance, tool/retrieval correctness, refusal/safety behavior, trajectory consistency, human escalation rate, token/cost, and evaluation quality. A successful HTTP/model response is only transport/service success.\n- Keep SLI specification separate from SLI implementation. “Correctly completed customer task” is an outcome specification; an LLM judge score is only one possible measurement implementation and can itself drift or fail.\n- Evaluation telemetry should attach to the execution trace/session so operators can move from a degraded aggregate metric to the exact model/tool/retrieval trajectory. EA-051 provides the interoperability substrate for that linkage.\n- Reliability cannot be represented by one pass@1 or aggregate success metric. The cited 2026 research motivates repeated-run consistency, perturbation robustness, controlled dependency-failure tests and task-horizon sensitivity as separate axes.\n- Long-horizon workflows need explicit reliability budgets or step/trajectory risk analysis. Even when per-step reliability is high, dependent multi-step execution compounds failure risk; benchmark horizons must not be assumed representative of production horizons.\n- Error-budget logic can be adapted to AI outcomes, but only after an auditable SLI exists. Example: “percentage of in-scope claims-processing runs ending in policy-compliant terminal state within 120 s” is budgetable; “average judge score looks good” is insufficient without threshold, population and measurement reliability.\n- Incident response must preserve the full provenance needed for replay/diagnosis: prompt/config/model revision, tools and tool versions, retrieved evidence identifiers, policy version, trace, evaluator version and external dependency state. Exact content retention remains privacy-governed." }, { "heading": "LIMITS / COUNTEREVIDENCE / FAILURE ENVELOPE", "text": "- Google SRE guidance is general service-operations guidance, not an AI-agent standard. Its value is the control-loop model, not direct proof of AI-specific thresholds.\n- 2026 agent reliability papers are research evidence; they do not establish universal production failure rates or product guarantees.\n- LLM-as-judge or automated evaluator output is measurement, not ground truth. Evaluator model/version/prompt changes can create apparent SLI movement without product behavior change.\n- Offline benchmark gains may not survive real permissions, data freshness, rate limits, UI drift, business rules, or long task horizons.\n- Provider telemetry gaps can make nominally identical SLO calculations non-comparable across stacks.\n- Retry can improve user-visible success while increasing latency/cost and masking per-attempt instability; report logical-run and attempt-level metrics separately." }, { "heading": "AVAILABILITY_GATES", "text": "- Production trace/evaluation linkage requires sufficient instrumentation coverage from EA-051.\n- Ground-truth or high-quality adjudication may require human review samples, domain validators or deterministic end-state checks.\n- Organization policy determines whether prompts/outputs/tool payloads can be retained for incident analysis.\n- Provider/model version changes require re-baselining when they materially alter quality or cost distributions." }, { "heading": "IMPLEMENTATION_STATE", "text": "Status: operational pattern is implementable now; universal AI-SRE standard is not established.\nRecommended minimum control loop:\n1. Define user/business outcome SLIs first.\n2. Add classical service SLIs: availability, latency, dependency/tool error, queue/saturation.\n3. Add AI execution SLIs: valid terminal state, tool/retrieval correctness, policy compliance, escalation/refusal, trajectory length/retry, token/cost.\n4. Attach offline/online evaluations to traces with evaluator provenance.\n5. Set SLOs/error budgets only for measures with defensible denominators and measurement procedures.\n6. Alert on burn/degradation; freeze or canary risky model/prompt/tool changes when agreed reliability budget is exhausted.\n7. Run postmortems with trajectory/provenance evidence and convert findings into regression/fault-injection tests." }, { "heading": "BENCHMARK_OR_EVAL", "text": "Use a portfolio rather than one score: repeated execution; semantically equivalent perturbations; dependency fault injection (timeouts/rate limits/partial responses/schema drift); long-horizon tasks; cost/latency distribution; policy/end-state validators. Report benchmark version, task horizon and environment. Do not turn research benchmark numbers into production SLO targets without local measurement." }, { "heading": "SECURITY / PRIVACY / AUDIT / COST / OPERABILITY", "text": "Security/privacy: incident traces can contain the most sensitive AI data. Apply least-retention, redaction and access control; preserve hashes/IDs/version metadata when payloads cannot be retained.\nAudit: evaluator identity/version/prompt, policy version and model/tool/retrieval versions must accompany quality decisions.\nCost: treat token/tool/retry spend as operational SLIs where economically material; reconcile telemetry-derived costs against billing sources.\nOperability: distinguish model-quality degradation from platform outage, provider outage, tool failure, retrieval/data failure, evaluator drift and policy change so the response owner is clear." }, { "heading": "SAFE_WORDING_FOR_FUTURE_BOOK", "text": "“AgentOps/AI-SRE is best treated as conventional SRE extended with AI-specific outcome and trajectory measurements. Availability and latency remain necessary but are insufficient: production agents also need task/end-state correctness, policy compliance, dependency behavior, repeated-run consistency, evaluation provenance and cost visibility. Automated evaluators are measurement instruments rather than ground truth, and benchmark pass rates should not be converted directly into production SLOs. Define auditable SLIs first, then apply SLO/error-budget, alerting, incident and continuous-improvement practices.”" }, { "heading": "UPDATE_WORK_HANDOFF", "text": "1. Use EA-051 to define the trace/evaluation event model required for operational SLIs.\n2. Create an SLI catalog template separating outcome specification, measurement implementation, denominator/population, evaluator version, threshold and owner.\n3. Add TEST_PROFILEs for repeated-run consistency, semantic perturbation, dependency fault injection and long-horizon regression.\n4. For each provider mapping, document telemetry blind spots before claiming comparable SLO coverage.\n5. Add incident taxonomy: platform/model/tool/retrieval/data/policy/evaluator/credential/GUI-browser dependency." }, { "heading": "OPEN_UNKNOWN", "text": "consensus terminology for AgentOps vs AI-SRE; validated local thresholds for quality SLOs; evaluator drift controls; how to budget compounded long-horizon risk; provider-equivalent telemetry coverage." }, { "heading": "STOP_CONDITION", "text": "bounded initial packet complete. Reopen when local production data exists, EA-051 schema mapping is finalized, or a provider-specific operational profile is built.\n\n---" }, { "heading": "AUDITED_BOUNDARY", "text": "AgentOps/AI-SRE is retained as an evidence-backed dictionary operating-model proposal extending conventional SRE with AI-specific outcome/trajectory/evaluation signals; no universal production thresholds are asserted. Automated evaluators are measurement instruments, not ground truth. Research benchmark numbers are not production SLO targets. Preserve telemetry dependency on EA-051 and all OPEN_UNKNOWN." } ], "human_url": "/tools/ea-052-agentops-ai-sre", "markdown_url": "/agent/article.md?path=%2Ftools%2Fea-052-agentops-ai-sre", "json_url": "/agent/article.json?path=%2Ftools%2Fea-052-agentops-ai-sre" }