{ "format": "agent-article-v0", "article_id": "EA-016", "canonical_url": "/tools/ea-016-ai-evals", "page_type": "guide", "claim_ids": [], "evidence_ids": [ "EVD-EA-016-URL-B9A542AD09B5", "EVD-EA-016-URL-6C790A16DE0A", "EVD-EA-016-URL-89540EC431B8", "EVD-EA-016-URL-86506DE6773B", "EVD-EA-016-URL-1B94BEF68087", "EVD-EA-016-URL-2B3F10F1A571", "EVD-EA-016-URL-42468EB1C49E", "EVD-EA-016-URL-9AC163575FA8", "EA-016-R11-RESEARCH", "EA-016-R11-WEB-READBACK" ], "unknown_ids": [ "EA-016-UNKNOWN-01", "EA-016-UNKNOWN-CURRENTNESS" ], "operational_object_ids": [], "provider": [], "models": [], "plans": [], "surfaces": [], "regions": [], "current_status": "UNKNOWN", "observation_status": "SPEC_ONLY", "freshness": "UNKNOWN", "last_verified": null, "supersedes": [], "superseded_by": [], "provenance": [ "COMM-016-EA0016", "SRC-EA-016-URL-B9A542AD09B5", "SRC-EA-016-URL-6C790A16DE0A", "SRC-EA-016-URL-89540EC431B8", "SRC-EA-016-URL-86506DE6773B", "SRC-EA-016-URL-1B94BEF68087", "SRC-EA-016-URL-2B3F10F1A571", "SRC-EA-016-URL-42468EB1C49E", "SRC-EA-016-URL-9AC163575FA8", "WEBBODY-EA0016-R8-E4B33B2A" ], "article_sources": [ { "id": "EVD-EA-016-URL-B9A542AD09B5", "evidence_class": "RESEARCH", "title": "NIST TEVV-Athlon Framework, NIST AI 200-2 Initial Public Draft, announced 2026-08-07 / page updated 2026-08-14, comment period through 2026-10-06 at source-packet time. Use as orga", "href": "https://doi.org/10.6028/NIST.AI.200-2.ipd", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-B9A542AD09B5" ], "supporting_locators": [ "Evidence and provenance", "- NIST TEVV-Athlon Framework, NIST AI 200-2 Initial Public Draft, announced 2026-08-07 / page updated 2026-08-14, comment period through 2026-10-06 at source-packet time. Use as organizational TEVV context; do not present as a final standard. Locator: https://doi.org/10.6028/NIST.AI.200-2.ipd" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-6C790A16DE0A", "evidence_class": "RESEARCH", "title": "Anthropic, “Demystifying evals for AI agents”, 2026-01-09. Use for task/trial/grader/transcript/outcome/harness separation, capability vs regression, isolated trials and grader cla", "href": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-6C790A16DE0A" ], "supporting_locators": [ "Evidence and provenance", "- Anthropic, “Demystifying evals for AI agents”, 2026-01-09. Use for task/trial/grader/transcript/outcome/harness separation, capability vs regression, isolated trials and grader classes. Provider engineering practice, not universal standard. Locator: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-89540EC431B8", "evidence_class": "RESEARCH", "title": "OpenAI Evals API “Create eval”, current at source-packet access 2026-09-11. Label VERIFIED_SPEC + PRODUCT_VOLATILE. API, pricing, availability and tenant entitlement require public", "href": "https://developers.openai.com/api/reference/java/resources/evals/methods/create", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-89540EC431B8" ], "supporting_locators": [ "Evidence and provenance", "- OpenAI Evals API “Create eval”, current at source-packet access 2026-09-11. Label VERIFIED_SPEC + PRODUCT_VOLATILE. API, pricing, availability and tenant entitlement require publication-time recheck. Locator: https://developers.openai.com/api/reference/java/resources/evals/methods/create" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-86506DE6773B", "evidence_class": "RESEARCH", "title": "OpenAI, “信頼できる第三者評価のための共通プレイブック”, 2026-05-29. Use for harness/budget/tool/scoring/contamination/shortcut/broken-task validity hazards. Locator:", "href": "https://openai.com/ja-JP/index/trustworthy-third-party-evaluations-foundations/", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-86506DE6773B" ], "supporting_locators": [ "Evidence and provenance", "- OpenAI, “信頼できる第三者評価のための共通プレイブック”, 2026-05-29. Use for harness/budget/tool/scoring/contamination/shortcut/broken-task validity hazards. Locator: https://openai.com/ja-JP/index/trustworthy-third-party-evaluations-foundations/" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-1B94BEF68087", "evidence_class": "RESEARCH", "title": "Google Cloud, “Evaluate a judge model”, source-packet state Preview / Pre-GA. Label VERIFIED_SPEC + PRODUCT_VOLATILE. Use for judge calibration against human-rating ground truth an", "href": "https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluate-judge-model", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-1B94BEF68087" ], "supporting_locators": [ "Evidence and provenance", "- Google Cloud, “Evaluate a judge model”, source-packet state Preview / Pre-GA. Label VERIFIED_SPEC + PRODUCT_VOLATILE. Use for judge calibration against human-rating ground truth and balanced accuracy/F1/confusion-matrix examples. Do not infer GA/SLA/region availability. Locator: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluate-judge-model" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-2B3F10F1A571", "evidence_class": "RESEARCH", "title": "Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge”, arXiv:2406.07791, 2024-06-12. Use as counterevidence for pairwise judge position bias; do n", "href": "https://arxiv.org/abs/2406.07791", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-2B3F10F1A571" ], "supporting_locators": [ "Evidence and provenance", "- Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge”, arXiv:2406.07791, 2024-06-12. Use as counterevidence for pairwise judge position bias; do not generalize one quantitative result to all judges/tasks. Locator: https://arxiv.org/abs/2406.07791" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-42468EB1C49E", "evidence_class": "RESEARCH", "title": "Anthropic, “Eval awareness in Claude Opus 4.6’s BrowseComp performance”, 2026-03-06. Use as contamination / answer-leakage / shortcut / eval-awareness failure-envelope evidence; do", "href": "https://www.anthropic.com/engineering/eval-awareness-browsecomp", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-42468EB1C49E" ], "supporting_locators": [ "Evidence and provenance", "- Anthropic, “Eval awareness in Claude Opus 4.6’s BrowseComp performance”, 2026-03-06. Use as contamination / answer-leakage / shortcut / eval-awareness failure-envelope evidence; do not generalize to every benchmark/model. Locator: https://www.anthropic.com/engineering/eval-awareness-browsecomp" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EVD-EA-016-URL-9AC163575FA8", "evidence_class": "RESEARCH", "title": "NIST Generative Artificial Intelligence Evaluation Program (GenAI), updated 2026-05-07 at source-packet access. Use as measurement-science context. Locator:", "href": "https://www.nist.gov/programs-projects/generative-artificial-intelligence-evaluation-program-genai", "supports": [], "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。", "origin_class": "SOURCE_PACKET_REFERENCE", "source_or_experiment_ids": [ "SRC-EA-016-URL-9AC163575FA8" ], "supporting_locators": [ "Evidence and provenance", "- NIST Generative Artificial Intelligence Evaluation Program (GenAI), updated 2026-05-07 at source-packet access. Use as measurement-science context. Locator: https://www.nist.gov/programs-projects/generative-artificial-intelligence-evaluation-program-genai" ], "scope": "Evidence and provenance", "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED" }, { "id": "EA-016-R11-RESEARCH", "evidence_class": "RESEARCH", "title": "EA-016 監査済みresearch / canonical revision binding", "supports": [ "EvalsはAIの「点数表」ではなく、業務上の成功条件を実行可能なテストへ変換し、変更による改善と回帰を検知する運用機構として扱う。生成AIやagentでは最終発話だけでなく、成果物、環境state、tool side effect、権限違反を評価する。Golden setは固定された万能正解集ではなく、provenance、version、coverage、criticalityを持つliving artifactとして管理する。高いscoreは当該suite内の結果であり、未知の実運用分布や安全性全体の保証へ外挿しない。" ], "limit": "canonical監査・revision bindingの存在を示す。外部ソースcurrentnessが本文またはcontrol-planeで確定していない場合は、currentnessをUNKNOWNのまま保持する。", "origin_class": "AUDITED_CANONICAL_RESEARCH_PACKET", "source_or_experiment_ids": [ "COMM-016-EA0016" ], "supporting_locators": [ "COMM-016-EA0016", "EA-016-R5-LEGACY-MIGRATION-20261008-R1", "content/staging/EA-016.md" ], "scope": "監査済みEA本文・canonical revision・既存Source Graph relationの範囲。欠損currentnessは推測で閉じない。", "provenance_status": "AUDIT_PASS_CANONICAL_BINDING" }, { "id": "EA-016-R11-WEB-READBACK", "evidence_class": "IMPL", "title": "EA-016 Web body identity / canonical readback", "supports": [ "canonical staging bodyのSHA-256一致とR9 canonical ledger readbackをProduction projectionの入力として検証する。" ], "limit": "body identity/readbackは本文claim自体の真偽や外部ソースの最新性を追加保証しない。", "checked_at": null, "origin_class": "IMPLEMENTATION_READBACK", "source_or_experiment_ids": [ "WEBBODY-EA0016-R8-E4B33B2A" ], "supporting_locators": [ "WEBBODY-EA0016-R8-E4B33B2A", "content/staging/EA-016.md", "EA-016-R5-LEGACY-MIGRATION-20261008-R1" ], "scope": "R9/R10 canonical readback後のProduction生成時body identity。", "provenance_status": "CANONICAL_READBACK_BODY_HASH_MATCH" } ], "research_sections": [ { "heading": "Public claim", "text": "EvalsはAIの「点数表」ではなく、業務上の成功条件を実行可能なテストへ変換し、変更による改善と回帰を検知する運用機構として扱う。生成AIやagentでは最終発話だけでなく、成果物、環境state、tool side effect、権限違反を評価する。Golden setは固定された万能正解集ではなく、provenance、version、coverage、criticalityを持つliving artifactとして管理する。高いscoreは当該suite内の結果であり、未知の実運用分布や安全性全体の保証へ外挿しない。" }, { "heading": "Scope", "text": "EA-011のSUT_MANIFEST / FIXTURE_MANIFEST / RESULT_RECORDを再利用し、EA-016ではCASE/SUITE/GRADER lifecycle metadataを上乗せする。公開benchmark比較そのものを重複して主題化しない。\n- evaluation lifecycle / release gate / regression governance\n- golden / regression / capability / critical / holdout suite\n- deterministic grader / bounded programmatic metric / LLM judge / human eval\n- agent task completion / tool execution / final state\n- eval dataset governance / contamination / judge bias / harness defects\n- production-derived case governance" }, { "heading": "Evidence and provenance", "text": "- NIST TEVV-Athlon Framework, NIST AI 200-2 Initial Public Draft, announced 2026-08-07 / page updated 2026-08-14, comment period through 2026-10-06 at source-packet time. Use as organizational TEVV context; do not present as a final standard. Locator: https://doi.org/10.6028/NIST.AI.200-2.ipd\n- Anthropic, “Demystifying evals for AI agents”, 2026-01-09. Use for task/trial/grader/transcript/outcome/harness separation, capability vs regression, isolated trials and grader classes. Provider engineering practice, not universal standard. Locator: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\n- OpenAI Evals API “Create eval”, current at source-packet access 2026-09-11. Label VERIFIED_SPEC + PRODUCT_VOLATILE. API, pricing, availability and tenant entitlement require publication-time recheck. Locator: https://developers.openai.com/api/reference/java/resources/evals/methods/create\n- OpenAI, “信頼できる第三者評価のための共通プレイブック”, 2026-05-29. Use for harness/budget/tool/scoring/contamination/shortcut/broken-task validity hazards. Locator: https://openai.com/ja-JP/index/trustworthy-third-party-evaluations-foundations/\n- Google Cloud, “Evaluate a judge model”, source-packet state Preview / Pre-GA. Label VERIFIED_SPEC + PRODUCT_VOLATILE. Use for judge calibration against human-rating ground truth and balanced accuracy/F1/confusion-matrix examples. Do not infer GA/SLA/region availability. Locator: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluate-judge-model\n- Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge”, arXiv:2406.07791, 2024-06-12. Use as counterevidence for pairwise judge position bias; do not generalize one quantitative result to all judges/tasks. Locator: https://arxiv.org/abs/2406.07791\n- Anthropic, “Eval awareness in Claude Opus 4.6’s BrowseComp performance”, 2026-03-06. Use as contamination / answer-leakage / shortcut / eval-awareness failure-envelope evidence; do not generalize to every benchmark/model. Locator: https://www.anthropic.com/engineering/eval-awareness-browsecomp\n- NIST Generative Artificial Intelligence Evaluation Program (GenAI), updated 2026-05-07 at source-packet access. Use as measurement-science context. Locator: https://www.nist.gov/programs-projects/generative-artificial-intelligence-evaluation-program-genai" }, { "heading": "Evaluation model", "text": "Evalの最小単位をprompt/outputだけに置かず、`task → trial → evidence → grade` として扱う。\n- TASK_SPEC: input、initial state、allowed tools/permissions、success condition、forbidden effects\n- TRIAL: model/version、prompt/config、runtime/harness、開始条件\n- TRACE: tool call、retrieval、state transition、error/retryの保存可能部分\n- OUTCOME: DB、document、table、send queue等の最終state\n- GRADERS: deterministic assertion、state check、schema/diff、human rubric、LLM rubric\n- RESULT: dimension別score、critical failures、cost/latency\nagentの自己申告をtask completionの証拠としない。" }, { "heading": "Suite separation", "text": "- CAPABILITY_SUITE: 新能力・改善余地を測る。\n- REGRESSION_SUITE: 過去のaccepted behaviorを保護する。\n- CRITICAL_SAFETY_PERMISSION_SUITE: aggregate平均で相殺せず、重大違反をhard gate化できる。\n- HOLDOUT_ADAPTIVE_SUITE: 頻繁に開発へ露出させず、overfit / shortcut / contaminationを検出する。\n- PRODUCTION_SENTINEL: 統制された実運用failure patternを再現可能なcaseへ変換し、必要に応じregressionへ昇格する。" }, { "heading": "EVAL_CASE_RECORD", "text": "以下は調査済みimplementation structureであり、既存標準の必須schemaとして扱わない。\n- CASE_ID / SUITE_ID / SUITE_VERSION\n- scenario / source_origin / business_owner / criticality\n- input_fixture_ref / initial_state_hash\n- allowed_tools / allowed_permissions / forbidden_effects\n- expected_outcome / acceptable_variation\n- grader_ids / grader_versions\n- provenance / sensitivity / holdout_flag\n- added_reason / review_date / retirement_state" }, { "heading": "EVAL_RUN_RECORD", "text": "- RUN_ID\n- candidate SUT manifest / baseline SUT manifest\n- case / suite version\n- n_trials / environment version\n- per-trial outputs / outcome refs\n- dimension scores / critical failures\n- grader outputs / grader version\n- latency / token / cost / error\n- aggregate / confidence or variance representation\n- release_decision / reviewer / exception rationale" }, { "heading": "Suite lifecycle", "text": "incident or user failure → reproduce → minimize/sanitize → assign expected outcome and invariants → validate grader with positive/negative controls → add to capability/regression/critical suite → version → CI/pre-release run → production sentinel → periodic retire/rebalance。" }, { "heading": "Grader hierarchy", "text": "検証可能な業務ではdeterministicな判定を優先する。\n1. deterministic: schema、normalized field、unit test、artifact diff、permission log、DB state、tool side effect\n2. bounded programmatic metric: precision/recall、数値許容差、citation relation等\n3. LLM grader: tone、説明品質、複数妥当解など\n4. human/domain expert: rubric定義、gold calibration、曖昧case、重大判断\n\nLLM judgeをhumanの代替正解器としない。pairwise評価ではA/B order swap、匿名化、human-labeled sample等による校正を行う。human evalにもrubric ambiguity、inter-rater disagreement、疲労、順序効果があり得るため、人間判定自体も無謬としない。" }, { "heading": "Agent outcome and invariants", "text": "通常のtask completionはfinal state/outputを中心に評価し、有効な別解を固定tool-call sequenceで不当に落とさない。一方、credential scope超過、禁止tool、未承認write、外部送信、duplicate side effect等はpath-independent invariantとして別途fail可能にする。" }, { "heading": "Release result vector", "text": "- task_success / partial_completion\n- critical_failure_count\n- permission_violation\n- unintended_write / duplicate_side_effect\n- factual / grounding / citation error\n- schema / structure failure\n- human rubric dimensions\n- retry / timeout / error rate\n- latency / token / monetary cost\n- trial variance / repeated-trial pass rate\n単一aggregate scoreのみでreleaseを判断しない。閾値とtrial数は業務criticality依存であり、一律値を推測しない。" }, { "heading": "Eval health", "text": "Eval suite自体もテスト対象とする。\n- reference solutionがgraderを通るか\n- intentionally-bad solutionを落とせるか\n- A/B order swapでjudge verdictが不自然に反転しないか\n- task textとgrader assumptionsが一致するか\n- fixture initial stateがtrial間でcleanか\n- hidden state/cache/previous artifactが漏れていないか\n- public answer leakage / benchmark contaminationがないか\n- eval version変更前後のscore continuityを説明できるか\nscore低下だけでmodel regression、高得点だけで能力向上と断定しない。" }, { "heading": "Security / privacy / audit / cost / operability", "text": "本番credential/secretをfixtureへ埋め込まない。permission/tool evalは最小権限のisolated environmentで行い、実外部送信や不可逆writeを避ける。holdout answer key / grader criteriaへのSUT accessを遮断する。\nproduction failureをcase化するときはminimize / anonymize / synthetic化を優先する。raw transcriptが不要なら必要なoutcome / structured evidenceのみ保持する。\nsuite / case / grader / harness / SUT versionと変更主体を追跡する。failed release overrideはreason / approver / expiryを残す。\neval costはinference、judge、tool/API、environment、human review、rerun、maintenanceを含む。fast deterministic smoke → regression → expensive agent/human/holdoutの段階化は可能だが、critical permission/safety gateをcost削減目的で外さない。\nflaky environment failureとSUT failureを別codeにする。timeout/retryによる二重side effectを避けるisolated fixtureを用いる。" }, { "heading": "Test profiles", "text": "以下はすべて `UNEXECUTED`。実行済みと表示しない。\n- TEST_PROFILE-016-A — GOLDEN/REGRESSION RELEASE GATE\n- TEST_PROFILE-016-B — AGENT TASK COMPLETION / TOOL EXECUTION\n- TEST_PROFILE-016-C — LLM JUDGE CALIBRATION\n- TEST_PROFILE-016-D — EVAL HEALTH / NEGATIVE CONTROL\n- TEST_PROFILE-016-E — HOLDOUT / CONTAMINATION" }, { "heading": "Failure envelope / OPEN_UNKNOWN", "text": "- Evals導入は品質保証そのものではない。\n- 高いregression pass率はcase bank内での維持を示すだけで、全業務分布の安全性を証明しない。\n- LLM-as-judgeにはbiasがあり、human calibrationなしの絶対判定器としない。\n- deterministic graderも仕様が誤っていれば誤判定する。\n- public benchmarkはcontamination / shortcut / eval-awarenessの影響を受け得る。\n- provider eval serviceのpricing、region、retention、ZDR適合、enterprise entitlementの完全横断表は `OPEN_UNKNOWN`。\n- 業種別human agreement threshold、trial数、release thresholdは一律に決めない。\n- NIST AI 200-2のfinal statusはpublication時に再確認する。\n- PRODUCT_VOLATILE provider surfacesは公開直前に再確認する。" } ], "human_url": "/tools/ea-016-ai-evals", "markdown_url": "/agent/article.md?path=%2Ftools%2Fea-016-ai-evals", "json_url": "/agent/article.json?path=%2Ftools%2Fea-016-ai-evals" }