Canary／A-B／Feature Flag for AI

AI変更は、通常ソフトウェアのbinary差分だけでなく、model ID/revision、prompt/instruction、generation parameters、tool schema、retrieval/index、policy/guardrail、response schema等の組合せで実行挙動が変わる。新構成を一括切替すると品質劣化・安全性低下・tool side effect・latency/cost増大のblast radiusが大きくなる。公開後アップデートでは、AI構成をversioned assetとして固定したうえで、誰に／何％に／いつまで新構成を露出するかを別のprogressive-delivery controlとして管理する設計を扱う価値が高い。

Enterprise AI Core: progressive delivery / deployment governance / rollback gate Interoperability: OpenFeature vendor-neutral flag evaluation API Implementation: Argo Rollouts等のcanary/progressive delivery、provider prompt version/variant機能 Provider mapping: Amazon Bedrock Prompt Management等 Governance: cohort definition、promotion/abort authority、change provenance Benchmark/Eval: offline eval + online canary/A-Bの分離

• AIのcanary/A-B/feature flagは同義ではなく、異なる目的を持つcontrolであるか。
• 新model／prompt／tool等を安全に限定投入するため、どの構成をimmutableに固定し、何をruntime routingで切り替えるべきか。
• promotion/abortの判定に通常SRE指標だけでなくAI固有指標をどこまで組み込めるか。
• feature flag control plane自体のsecurity/privacy/failure modeは何か。

E1 | class: official specification | title: Flag Evaluation API | author/org: OpenFeature | canonical URL: https://openfeature.dev/specification/sections/flag-evaluation/ | publication/update: current specification, accessed 2026-09-14 | version: OpenFeature Specification v0.9.0 context | locator: Overview; abnormal/default evaluation requirements; shutdown | label: VERIFIED_SPEC | relation: support/context | Evidence: flag evaluation API is vendor-independent from the flag control plane; no-op/default behavior exists, establishing routing control and fallback as separate from the AI payload itself. E2 | class: official specification | title: Evaluation Context | author/org: OpenFeature | canonical URL: https://openfeature.dev/specification/sections/evaluation-context/ | accessed 2026-09-14 | version: v0.9.0 context | locator: §3 overview, Requirement 3.1.1, context merging | label: VERIFIED_SPEC | relation: support/limit | Evidence: targeting context can drive rule, subject override, and fractional evaluation; targeting key may be required for deterministic fractional rollout. E3 | class: official specification | title: Hooks | author/org: OpenFeature | canonical URL: https://openfeature.dev/specification/sections/hooks/ | accessed 2026-09-14 | version: v0.9.0 context | locator: Overview; before/after/error/finally; hook data | label: VERIFIED_SPEC | relation: support | Evidence: flag lifecycle exposes validation, telemetry, and error hooks, enabling rollout-decision observability without coupling it to model logic. E4 | class: project release/update | title: OpenFeature Mid-2026 Update: Platform-Native Feature Flags, a New TC Member, and Spec v0.9.0 | author/org: OpenFeature | canonical URL: https://openfeature.dev/blog/openfeature-mid-2026-update/ | publication: 2026-08-12 | version: Specification v0.9.0 | locator: Specification v0.9.0 section | label: SOURCE_REPORT / OSS_PRACTICE | relation: support/limit | Evidence: v0.9.0 is first spec release since v0.8.0 (2024-03) and stabilizes evaluation/provider sections while including accumulated changes; SDK/provider compliance can lag specification change. E5 | class: author-maintained OSS / release | title: Argo Rollouts — Progressive Delivery for Kubernetes; Releases | author/org: Argo Project | canonical URLs: https://github.com/argoproj/argo-rollouts ; https://github.com/argoproj/argo-rollouts/releases | release: v1.10.0 dated 2026-08-05, release assets updated 2026-08-27 | tag/version: v1.10.0, commit d90700a | locator: project README features; v1.10.0 release | label: OSS_PRACTICE / OBSERVED | relation: support/context | Evidence: canary, weighted traffic shifting, metric analysis, automated promotion/rollback are implemented progressive-delivery primitives. E6 | class: public security advisory | title: Argo Rollouts dashboard through 1.10.0 binds to all interfaces and exposes mutating Rollout operations without authentication | author/org: GitHub Advisory Database / NVD reference | canonical URL: https://github.com/advisories/GHSA-366v-5xmx-36vh | publication: 2026-08-28 | affected: through 1.10.0 per advisory | CVE: CVE-2026-82277 | locator: Description | label: SOURCE_REPORT | relation: counter/limit | Evidence: rollout control plane can itself become a critical attack surface; same-network attacker could invoke promote/abort/restart/image/undo/retry functions under described conditions. E7 | class: provider official documentation | title: Construct and store reusable prompts with Prompt management in Amazon Bedrock | author/org: AWS | canonical URL: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management.html | accessed: 2026-09-14 | service revision: current docs | locator: Key definitions / Prompt variant | label: PRODUCT_VOLATILE / SOURCE_REPORT | relation: support/context | Evidence: a prompt variant can vary message, model, or inference configuration and can be tested/compared. E8 | class: provider official documentation | title: Deploy a prompt to your application using versions in Prompt management | author/org: AWS | canonical URL: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management-deploy.html | accessed: 2026-09-14 | locator: version definition | label: PRODUCT_VOLATILE / SOURCE_REPORT | relation: support | Evidence: a prompt version is a point-in-time snapshot of prompt/configuration for production integration. E9 | class: provider official documentation | title: Compare versions of a prompt in Prompt management | author/org: AWS | canonical URL: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management-version-compare.html | accessed: 2026-09-14 | locator: compare JSON definitions and model responses | label: PRODUCT_VOLATILE / SOURCE_REPORT | relation: support/limit | Evidence: provider supports configuration/output comparison, but this feature alone is not a production traffic-splitting or statistical A/B system.

F1 [VERIFIED_SPEC / INFERENCE]: Feature flag, canary, and A/B must not be collapsed. Feature flag is a runtime selection primitive. Canary is staged exposure intended to bound blast radius and support promote/abort decisions. A/B is simultaneous assignment intended primarily for comparative inference/experimentation. One flag system may implement routing for either, but does not by itself supply an evaluation methodology. F2 [OSS_PRACTICE]: Progressive delivery is mature outside AI. Argo Rollouts implements weighted traffic shifting, metric-driven analysis, manual judgement, promotion and rollback. Therefore AI does not require inventing a wholly new rollout control plane; existing progressive-delivery mechanisms can carry AI-specific gates. F3 [INFERENCE / PROPOSAL]: AI rollout should separate two layers: (A) an immutable execution-configuration bundle identifying the exact model/prompt/tool/retrieval/policy/schema configuration and (B) a traffic/cohort selector deciding which requests use which bundle. This prevents a mutable alias or prompt draft from changing underneath an experiment. F4 [VERIFIED_SPEC]: OpenFeature evaluation context supports deterministic subject/cohort targeting and fractional rollout patterns via targeting keys and provider evaluation rules. Hooks provide appropriate attachment points for telemetry and validation. F5 [INFERENCE / PROPOSAL]: Promotion gates should include both conventional SRE signals (availability, errors, latency, saturation, unit cost) and AI-specific signals where applicable (task success/quality, policy violation/refusal rates, tool/retrieval failure, harmful side effects). Evaluator/judge version and sampling method must be recorded because the measurement instrument itself can drift. F6 [PRODUCT_VOLATILE]: Amazon Bedrock Prompt Management currently provides variants and version snapshots. This is useful as a provider-specific configuration primitive, but official docs cited here do not establish automatic percentage traffic splitting across prompt versions; do not describe it as a Bedrock canary controller without separate evidence. F7 [INFERENCE]: A rollout record needs at least resolved immutable configuration ID, flag/rule revision, cohort/target key derivation, exposure timestamp, metrics/eval revision, decision (continue/promote/abort), decision actor, and rollback target.

• A canary reduces blast radius; it does not prove correctness or safety. Rare failures, long-tail workflows, delayed side effects, and population shifts can survive a small canary.
• LLM stochasticity means a one-request comparison can be misleading. Repeated/sample-aware evaluation is required where variance matters.
• A/B experimentation may be unethical or operationally unacceptable for high-risk decisions if one arm may cause material harm. Offline/synthetic evaluation and stricter approval gates can be required before any live exposure.
• Feature flag service outage or mis-evaluation can route traffic incorrectly. Stable/fail-safe defaults must be deliberate rather than accidental.
• The Argo CVE-2026-82277 advisory demonstrates that deployment-control interfaces can themselves permit high-impact mutating actions if insufficiently protected. This advisory is evidence about the described Argo dashboard versions, not proof that all progressive-delivery systems share the vulnerability.
• OpenFeature v0.9.0 remains pre-1.0; spec evolution and SDK/provider conformance lag must be version-gated.

• OpenFeature specifies an API abstraction; actual percentage rollout, persistence, targeting, audit, and control-plane functionality depend on the selected provider/product.
• Argo Rollouts requires Kubernetes and integration assumptions; traffic routing/metrics support varies by ingress, service mesh and provider integration.
• Amazon Bedrock Prompt Management availability depends on supported regions/models, IAM permissions and current service configuration. Public docs do not prove availability in a specific tenant.
• Product/release claims above are as-of 2026-09-14 and require revalidation at book-update time.

OpenFeature: implementable vendor-neutral evaluation abstraction; current spec context v0.9.0. Argo Rollouts: production OSS progressive-delivery implementation; current GitHub release observed v1.10.0. Amazon Bedrock Prompt Management: provider-managed prompt variants/version snapshots exist; use as configuration/version evidence, not as automatic canary proof. Core candidate implementation pattern: versioned configuration registry -> cohort/flag resolver -> execution -> telemetry/eval -> promotion/abort controller -> stable rollback target.

No universal AI canary threshold was established. Future Work should define organization- and use-case-specific guardrails. Recommended evaluation sequence is offline regression/contract evaluation -> shadow or replay where feasible -> bounded canary -> staged increase -> broad rollout. This sequence is PROPOSAL/INFERENCE, not an industry standard established by the cited sources. For A/B, statistical design, power, interference, repeat exposure, and stochastic output variance must be handled separately from deployment safety.

Security: rollout promotion/abort and flag administration are privileged control-plane actions; enforce strong authn/authz, network restriction and audit. Do not expose unauthenticated mutation endpoints. Privacy: evaluation context may contain user attributes. OpenFeature documentation itself warns that providers may handle/persist context; minimize, pseudonymize/hash where suitable, and avoid unnecessary sensitive content. Audit: preserve resolved config revision and flag/rule revision, not only mutable alias/flag name. Cost: canary/A-B duplicates evaluation and may temporarily increase model/tool spend; cost per successful task should be a rollout signal where material. Operability: default/fallback route should be tested. A fail-open default to an unsafe/new AI configuration is an avoidable control error.

「AIの段階投入では、modelやprompt等の構成をversionで固定する層と、その構成へtrafficを割り当てるflag/canary層を分ける。Feature flagはroutingの仕組み、canaryはblast radiusを制限して段階的に昇格・中止する運用、A/Bは比較実験であり、同じものではない。既存のprogressive-delivery基盤はAIにも利用できるが、昇格判定には通常の可用性・遅延・costだけでなく、task品質・policy違反・tool失敗等のAI固有指標を追加し、評価器自体のversionも記録する。限定投入は安全性の証明ではなく、被害範囲を限定して検出可能性を上げるcontrolである。」

• 本編アップデート時はEA-058のexecution-configuration bundleと直接接続する。
• provider別mappingでは、prompt/model versioning機能とtraffic splitting機能を必ず別claimとして採集する。
• 代表的なOpenFeature provider 2〜3種についてpercentage rollout、sticky assignment、audit/fallback挙動を再確認すると実装比較が閉じる。
• Argo Rolloutsを例示する場合、v1.10.0以後のsecurity statusとCVE-2026-82277修正状況を更新時に再確認する。
• TEST_PROFILE（未実施）: synthetic/non-sensitive workloadでstable configとcandidate configを固定し、1%→5%→25%の段階投入、sticky cohort、abort threshold、audit record、fallback時のstable復帰、cohort汚染の不存在を確認。外部副作用を伴うtoolはmock/stub化する。

provider横断でAI-specific rollout metric schemaを統一する標準は未確立。特定tenantでのflag provider／Bedrock機能のavailabilityは未確認。LLM judgeをonline promotion gateにした場合の組織別許容誤差も未確立。

progressive-delivery primitive、AI configuration versioning、targeting/telemetry、control-plane counterevidenceの一次証拠が揃い、初回Packetとしては閉じた。tenant/provider固有挙動は将来mappingで再採集する。

---

---

OpenFeature v0.9.0 remains the current announced pre-1.0 line. Argo Rollouts v1.10.0 is the latest release, but CVE-2026-82277/GHSA-366v-5xmx-36vh describes dashboard exposure through 1.10.0 with no known patched version. Bedrock Prompt Management variants/versions are not by themselves traffic-splitting canary control.