# CLAUDE：評価データと公開前の評価契約

CURRENT_STATUS: UNKNOWN
OBSERVATION: UNOBSERVED
FRESHNESS: UNKNOWN
PAGE_TYPE: reference
CANONICAL_URL: /providers/pm-claude-core-07
CLAIM_IDS: none
EVIDENCE_IDS: EVD-PM-CLAUDE-CORE-07-URL-6727386D515F, EVD-PM-CLAUDE-CORE-07-URL-C49DB9436032, EVD-PM-CLAUDE-CORE-07-URL-DA45DE5646CB, EVD-PM-CLAUDE-CORE-07-URL-D11C2860A404, EVD-PM-CLAUDE-CORE-07-URL-DBCB362197DB, EVD-PM-CLAUDE-CORE-07-URL-1BA226019836, EVD-PM-CLAUDE-CORE-07-URL-1F49D0119CED, EVD-PM-CLAUDE-CORE-07-URL-8E2E37051F9B, EVD-PM-CLAUDE-CORE-07-URL-827ACF165DA2, EVD-PM-CLAUDE-CORE-07-URL-16D2741870AD, PM-CLAUDE-CORE-07-DOCUMENT-READBACK
UNKNOWN_IDS: PM-CLAUDE-CORE-07-CURRENTNESS
OPS_IDS: none

## Summary
CORE-07のdataset、scorer/judge、holdout、slice gate、release evidenceを、Anthropicの評価設計guidance、Console prompt improver/evaluator、Message Batches、structured outputs、Agent SDK/Claude Codeおよびauthor-maintained cookbookへ対応づける。公開仕様・固定履歴・限定issueだけを対象とし、実organization、非公開dataset、Claude/model judgeの実性能、組織閾値はOrganization Overlayへ残す。

## Current judgment
CORE-07のdataset、scorer/judge、holdout、slice gate、release evidenceを、Anthropicの評価設計guidance、Console prompt improver/evaluator、Message Batches、structured outputs、Agent SDK/Claude Codeおよびauthor-maintained cookbookへ対応づける。公開仕様・固定履歴・限定issueだけを対象とし、実organization、非公開dataset、Claude/model judgeの実性能、組織閾値はOrganization Overlayへ残す。

## Scope
CORE-07のdataset、scorer/judge、holdout、slice gate、release evidenceを、Anthropicの評価設計guidance、Console prompt improver/evaluator、Message Batches、structured outputs、Agent SDK/Claude Codeおよびauthor-maintained cookbookへ対応づける。公開仕様・固定履歴・限定issueだけを対象とし、実organization、非公開dataset、Claude/model judgeの実性能、組織閾値はOrganization Overlayへ残す。

## Confirmed

## UNKNOWN
- PM-CLAUDE-CORE-07-CURRENTNESS: 原資料の確定版から収録しています。外部根拠の最新性を今回の取込日で確認済みと扱いません。本文の未解決事項・残余制約を保持します。

## Evidence
- EVD-PM-CLAUDE-CORE-07-URL-6727386D515F (RESEARCH): Anthropic, Define success criteria and build evaluations: <[ — https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
- EVD-PM-CLAUDE-CORE-07-URL-C49DB9436032 (RESEARCH): Anthropic, Prompt engineering overview / prompting best practices: <[ — https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview
- EVD-PM-CLAUDE-CORE-07-URL-DA45DE5646CB (RESEARCH): Anthropic, Prompt engineering overview / prompting best practices: <[https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview](https://platform.claude.com/ — https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
- EVD-PM-CLAUDE-CORE-07-URL-D11C2860A404 (RESEARCH): Anthropic, Improve your prompts in the developer console: <[ — https://claude.com/blog/prompt-improver
- EVD-PM-CLAUDE-CORE-07-URL-DBCB362197DB (RESEARCH): Anthropic, Batch processing / Structured outputs: <[ — https://platform.claude.com/docs/en/build-with-claude/batch-processing
- EVD-PM-CLAUDE-CORE-07-URL-1BA226019836 (RESEARCH): Anthropic, Batch processing / Structured outputs: <[https://platform.claude.com/docs/en/build-with-claude/batch-processing](https://platform.claude.com/docs/en/build-with-claude/ba — https://platform.claude.com/docs/en/build-with-claude/structured-outputs
- EVD-PM-CLAUDE-CORE-07-URL-1F49D0119CED (RESEARCH): Anthropic, claude-cookbooks: <[ — https://github.com/anthropics/claude-cookbooks
- EVD-PM-CLAUDE-CORE-07-URL-8E2E37051F9B (RESEARCH): Anthropic, claude-cookbooks: <[https://github.com/anthropics/claude-cookbooks](https://github.com/anthropics/claude-cookbooks)>; issue #761: <[ — https://github.com/anthropics/claude-cookbooks/issues/761
- EVD-PM-CLAUDE-CORE-07-URL-827ACF165DA2 (RESEARCH): Shi et al., Judging the Judges: <[ — https://aclanthology.org/2025.ijcnlp-long.18/
- EVD-PM-CLAUDE-CORE-07-URL-16D2741870AD (RESEARCH): Dekoninck et al., ConStat: <[ — https://arxiv.org/abs/2405.16281
- PM-CLAUDE-CORE-07-DOCUMENT-READBACK (IMPL): 原資料との収録照合

## Practice
CORE-07のdataset、scorer/judge、holdout、slice gate、release evidenceを、Anthropicの評価設計guidance、Console prompt improver/evaluator、Message Batches、structured outputs、Agent SDK/Claude Codeおよびauthor-maintained cookbookへ対応づける。公開仕様・固定履歴・限定issueだけを対象とし、実organization、非公開dataset、Claude/model judgeの実性能、組織閾値はOrganization Overlayへ残す。
## 資料の識別情報
Source SHA-256 87d70f34057257f0c115f9ec6b6456f8c459c75d6068ad3dc28aaa5523f391f3

## Product Mapping — Claude / CORE-07
PRODUCT_MAPPING_ID: PM-CLAUDE-CORE-07-v0.1

PROVIDER: CLAUDE / ANTHROPIC

CORE_PACKET: PKT-GEMINI-A-CORE-07-v0.1

AS_OF: 2026-09-10 JST

STATUS: PUBLIC_MAPPING_BOUNDED_COMPLETE / CONSOLE_TEMPORAL_BOUNDARY / LIVE_ORG_PENDING

LAYER: PRODUCT_MAPPING (not Enterprise AI Core; not Organization Overlay)

## Scope
CORE-07のdataset、scorer/judge、holdout、slice gate、release evidenceを、Anthropicの評価設計guidance、Console prompt improver/evaluator、Message Batches、structured outputs、Agent SDK/Claude Codeおよびauthor-maintained cookbookへ対応づける。公開仕様・固定履歴・限定issueだけを対象とし、実organization、非公開dataset、Claude/model judgeの実性能、組織閾値はOrganization Overlayへ残す。

## Surface and lifecycle separation
| SURFACE | CURRENT EVIDENCE | DECISION | LIMIT |
| --- | --- | --- | --- |
| evaluation design guidance | success criteriaをspecific/measurableにし、task-specific・実分布・edge caseを含む評価、code/human/LLM gradingを使い分ける | provider guidanceをevaluation contract設計へ利用 | hosted registry、artifact immutability、retention、release authorityの保証ではない |
| Console prompt improver / evaluator | 2024-10-14発表ではprompt/example改善、ideal output、5段階評価、反復feedbackを提供 | historical/current-accessed Console surfaceとして記録し、candidate作成面と評価面を分離 | 発表時のprefill手法はClaude 4.6以降のcurrent prompting guidanceと互換でない。現行のversion/export/retention保証は公開証拠から確定しない |
| Message Batches | independent requestsをcustom_idで結合し、結果は順不同・部分成功、24時間expiry、29日availability | bulk eval実行carrierとして利用可能 | hosted eval registry・判定器・release decisionではない。batch完了を評価受入としない |
| structured outputs | JSON schemaに沿う出力を強制し、schema cacheを持つ | grader/result envelopeの構文検証に利用 | schema conformanceはrubric妥当性、意味品質、judge calibrationを証明しない |
| Agent SDK / Claude Code | tool、permission、hooks、session、result、OTel等のtrajectory carrier | agent evalではconfig、tool receipt、artifact/readback、外部最終状態を別join | ResultMessage/traceだけでは業務成果・side effectの完全性を証明しない |
| anthropics/claude-cookbooks | automated evaluation、RAG等のauthor-maintained recipes | external/self-managed harness例として扱う | sample notebookはhosted product、SLA、current-model互換性の保証ではない |

## Core → Claude mapping rows
| ROW | CORE CONTRACT | CLAUDE / ANTHROPIC FINDING | DECISION / UNKNOWN |
| --- | --- | --- | --- |
| PM-C07-CLAUDE-001 | evaluation artifact separation | current guidanceはsuccess criteria、test cases、graderを説明するが、candidate/dataset/scorer/judge/run/releaseを一つのimmutable objectとして保証しない | Core contract IDsで外部結合し、Console/UI名を正本identityにしない |
| PM-C07-CLAUDE-002 | representative and edge distribution | task-specific、real-world distribution、edge cases、held-out testsを推奨 | representative/failure/challenge/sealed holdoutを分離し、各sliceのprovenanceとaccessを記録 |
| PM-C07-CLAUDE-003 | deterministic-first scoring | code-based gradingは高速・信頼性、人手は柔軟だが高コスト、LLM gradingはscalableだが信頼性確認が必要とされる | exact/schema/rule/tool receiptを先行し、LLM judgeはversioned補助判定に限定 |
| PM-C07-CLAUDE-004 | judge registry and calibration | LLM graderには明確なrubricと指定出力が必要。独立研究はposition biasがjudge/taskごとに変動することを示す | judge model/config/prompt/rubric/order/sampling/abstention、人手不一致率を保存し、swap/permutationを行う |
| PM-C07-CLAUDE-005 | candidate optimization / holdout split | Console prompt improverはprompt・examples・feedbackを反復変更し得る | improver/evaluatorへ提示したcaseをoptimization corpusへ分類し、sealed holdoutから失効。発表時機能の現行永続性はUNKNOWN |
| PM-C07-CLAUDE-006 | temporal compatibility | 2024発表のprompt improverはassistant prefillを手法に含むが、current guidanceはClaude 4.6以降のlast-assistant prefill非対応を示す | candidate、model、API behavior、improver生成時点を固定し、生成promptをcurrent modelで回帰評価。古い機能説明を現行互換保証へ昇格しない |
| PM-C07-CLAUDE-007 | evaluator infrastructure failure | cookbook issue #761は、Sonnet 4.6で拒否されたprefill例外をbroad catchがis_correct=Falseへ変換し、全件0%に見せ得ると報告 | API/harness/parser/timeoutをINFRA_ERRORとしてcandidate failureから分離し、評価器異常時はfail closed。issueは対象script/version限定 |
| PM-C07-CLAUDE-008 | batch/output carrier boundary | Batchは順不同・部分状態、structured outputsは構文契約を提供 | custom_id、missing/errored/canceled/expired、retry、schema/parser errorを別記録し、valid N不足を平均へ混ぜない |
| PM-C07-CLAUDE-009 | bounded contamination and release gate | ConStatはpublic benchmarkと同分布fresh/rephrased casesのgapを測る限定反証を提供 | named Claude modelの汚染断定には使わず、critical slice AND gate、fresh challenge、human adjudication、artifact readbackでreleaseを判定 |

## Safe verification profile
非機密synthetic fixtureだけを用い、今回はAnthropic API、Console、organization、model、Batch、SDK/CLI、tool、dataset、judgeを操作していない。

- candidateにmodel ID、effort/thinking、system prompt、tools、permissions、SDK/CLI版、settings/source snapshotを含める。

- dev/regression/challenge/failure corpus/sealed holdoutを分離し、prompt improver・example generation・人手feedbackへの露出をleak ledgerへ記録する。

- deterministic checksを先行し、LLM judgeはorder swap、反復、abstention、人手calibrationを伴う補助判定にする。

- prefillを含む旧prompt/harnessをcurrent modelへ移すnegative compatibility testを置き、API例外を0点へ変換しない。

- Batchではcustom_idと各statusをread backし、missing/expired/erroredをINFRA_ERRORとして再実行可否と分離する。

- agent taskはtrajectory、tool receipt、artifact digest、external readback、acceptanceを別証拠として結合する。

- overall平均がpassでもcritical slice、known regression、Japanese、safety/over-refusal、tail、variance、valid Nのいずれかがfailならreleaseをblockする。

既存EXP-C07-001の12/12はCOMPILER_EXPERIMENTであり、Claude、Console、Batch、judge、modelのlive試験へ再分類しない。

## Organization Overlay handoff
- adopted surface、organization/workspace、dataset owner/access、retention/export、holdout custodian

- candidate model/effort/prompt/tool/config、grader/judge allowlist、人手sampleとdisagreement threshold

- improver/training/eval corpusの分離、leak response、freshness/failure corpus更新周期

- Batch quota/retry/deadline、infra-error policy、evidence retention

- release threshold、critical slice、variance、zero-tolerance、exception/rollback authority

## Provenance
- Anthropic, Define success criteria and build evaluations: <[https://platform.claude.com/docs/en/test-and-evaluate/develop-tests](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)> (accessed 2026-09-09)

- Anthropic, Prompt engineering overview / prompting best practices: <[https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview)>; <[https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices)> (accessed 2026-09-09)

- Anthropic, Improve your prompts in the developer console: <[https://claude.com/blog/prompt-improver](https://claude.com/blog/prompt-improver)> (published 2024-10-14; accessed 2026-09-09)

- Anthropic, Batch processing / Structured outputs: <[https://platform.claude.com/docs/en/build-with-claude/batch-processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing)>; <[https://platform.claude.com/docs/en/build-with-claude/structured-outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)> (accessed 2026-09-09)

- Anthropic, claude-cookbooks: <[https://github.com/anthropics/claude-cookbooks](https://github.com/anthropics/claude-cookbooks)>; issue #761: <[https://github.com/anthropics/claude-cookbooks/issues/761](https://github.com/anthropics/claude-cookbooks/issues/761)> (opened 2026-07-07; accessed 2026-09-09)

- Shi et al., Judging the Judges: <[https://aclanthology.org/2025.ijcnlp-long.18/](https://aclanthology.org/2025.ijcnlp-long.18/)>

- Dekoninck et al., ConStat: <[https://arxiv.org/abs/2405.16281](https://arxiv.org/abs/2405.16281)>

## Exit condition
Anthropicの評価guidance、Console improvement/evaluation、Batch、structured outputs、agent trajectory、cookbookを別surfaceとしてCORE-07へ対応づけ、temporal compatibilityとevaluator-infrastructure failureを境界付きで追加した。公開情報から確認できないConsole artifactのversion/export/retention、live judge性能、organization閾値はUNKNOWNのまま残す。Core claimは変更しないためGemini/GPT遡及監査は不要である。

NEXT: /agent/start.json
STOP: This is a knowledge projection, not an execution order.