{
  "format": "agent-article-v0",
  "article_id": "EA-750",
  "canonical_url": "/tools/ea-750-spot-preemptible-gpu-checkpoint",
  "page_type": "reference",
  "claim_ids": [],
  "evidence_ids": [
    "EVD-EA-750-URL-FF704E292777",
    "EVD-EA-750-URL-CE9D11064147",
    "EVD-EA-750-URL-C84E54048392",
    "EVD-EA-750-URL-8F83DA98D931",
    "EVD-EA-750-URL-BBD568FE8874",
    "EVD-EA-750-URL-9EE8C8EDE198",
    "EVD-EA-750-URL-269403C307E6",
    "EA-750-CURRENT-APPROVAL",
    "EA-750-CURRENT-BODY-READBACK"
  ],
  "unknown_ids": [
    "EA-750-UNKNOWN-01"
  ],
  "operational_object_ids": [],
  "provider": [],
  "models": [],
  "plans": [],
  "surfaces": [],
  "regions": [],
  "current_status": "CURRENT",
  "observation_status": "SPEC_ONLY",
  "freshness": "CURRENT",
  "last_verified": "2026-10-20",
  "supersedes": [],
  "superseded_by": [],
  "provenance": [
    "EA750-CANDIDATE-20261020-1205JST-C1",
    "SRC-EA-750-URL-FF704E292777",
    "SRC-EA-750-URL-CE9D11064147",
    "SRC-EA-750-URL-C84E54048392",
    "SRC-EA-750-URL-8F83DA98D931",
    "SRC-EA-750-URL-BBD568FE8874",
    "SRC-EA-750-URL-9EE8C8EDE198",
    "SRC-EA-750-URL-269403C307E6",
    "BODY-SHA256:4e11f2af77fe48eb3cc54400dd0fd566c3001ac84166677a4cbe0f2b02133f02"
  ],
  "article_sources": [
    {
      "id": "EVD-EA-750-URL-FF704E292777",
      "evidence_class": "RESEARCH",
      "title": "AWS EC2 User Guide, Spot Instance interruption notices:",
      "href": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-FF704E292777"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- AWS EC2 User Guide, Spot Instance interruption notices: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EVD-EA-750-URL-CE9D11064147",
      "evidence_class": "RESEARCH",
      "title": "AWS EC2 User Guide, Best practices for EC2 Spot:",
      "href": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-CE9D11064147"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- AWS EC2 User Guide, Best practices for EC2 Spot: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EVD-EA-750-URL-C84E54048392",
      "evidence_class": "RESEARCH",
      "title": "AWS EC2 User Guide, Initiate a Spot Instance interruption:",
      "href": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/initiate-a-spot-instance-interruption.html",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-C84E54048392"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- AWS EC2 User Guide, Initiate a Spot Instance interruption: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/initiate-a-spot-instance-interruption.html"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EVD-EA-750-URL-8F83DA98D931",
      "evidence_class": "RESEARCH",
      "title": "Google Cloud Compute Engine, Spot VMs:",
      "href": "https://docs.cloud.google.com/compute/docs/instances/spot",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-8F83DA98D931"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- Google Cloud Compute Engine, Spot VMs: https://docs.cloud.google.com/compute/docs/instances/spot"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EVD-EA-750-URL-BBD568FE8874",
      "evidence_class": "RESEARCH",
      "title": "Google Cloud Compute Engine, Create and use Spot VMs:",
      "href": "https://docs.cloud.google.com/compute/docs/instances/create-use-spot",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-BBD568FE8874"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- Google Cloud Compute Engine, Create and use Spot VMs: https://docs.cloud.google.com/compute/docs/instances/create-use-spot"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EVD-EA-750-URL-9EE8C8EDE198",
      "evidence_class": "RESEARCH",
      "title": "Microsoft Azure Architecture Center, Build workloads with Azure Spot Virtual Machines:",
      "href": "https://learn.microsoft.com/azure/architecture/guide/spot/spot-eviction",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-9EE8C8EDE198"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- Microsoft Azure Architecture Center, Build workloads with Azure Spot Virtual Machines: https://learn.microsoft.com/azure/architecture/guide/spot/spot-eviction"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EVD-EA-750-URL-269403C307E6",
      "evidence_class": "RESEARCH",
      "title": "Microsoft Azure Virtual Machines, About Azure Spot Virtual Machines:",
      "href": "https://learn.microsoft.com/azure/virtual-machines/spot-vms",
      "supports": [],
      "limit": "元の監査済み資料に記載された参照先です。URLの復元は、資料の最新性や全主張の支持を新たに検証したことを意味しません。",
      "origin_class": "SOURCE_PACKET_REFERENCE",
      "source_or_experiment_ids": [
        "SRC-EA-750-URL-269403C307E6"
      ],
      "supporting_locators": [
        "Source / provenance locators",
        "- Microsoft Azure Virtual Machines, About Azure Spot Virtual Machines: https://learn.microsoft.com/azure/virtual-machines/spot-vms"
      ],
      "scope": "Source / provenance locators",
      "provenance_status": "SOURCE_PACKET_REFERENCE_PROJECTED"
    },
    {
      "id": "EA-750-CURRENT-APPROVAL",
      "evidence_class": "RESEARCH",
      "title": "現行本文の監査・revision対応",
      "supports": [
        "現行台帳のAUDIT_PASSと候補・revision・本文ハッシュの対応を確認しています。"
      ],
      "limit": "監査状態と来歴の記録です。外部根拠の最新性や本文の全主張を追加保証しません。",
      "origin_class": "AUDITED_CANONICAL_RESEARCH_PACKET",
      "source_or_experiment_ids": [
        "EA750-CANDIDATE-20261020-1205JST-C1"
      ],
      "supporting_locators": [
        "1KqhJtiSb2S84rWmeYVwKcT_76dBwZ_B0ZZuAWQqH0rQ/EA_REVISION_LEDGER#ea_id=EA-750&revision_id=EA-750-RESEARCH-20261020-R1&candidate_id=EA750-CANDIDATE-20261020-1205JST-C1",
        "EA-750-RESEARCH-20261020-R1"
      ],
      "provenance_status": "CURRENT_CANONICAL_APPROVAL_READBACK"
    },
    {
      "id": "EA-750-CURRENT-BODY-READBACK",
      "evidence_class": "IMPL",
      "title": "現行本文の収録照合",
      "supports": [
        "公開生成入力の本文SHA-256が、監査済み候補に結び付いたハッシュと一致しています。"
      ],
      "limit": "本文の同一性の検査です。外部システムの挙動や全主張の真偽を実証したものではありません。",
      "checked_at": "2026-10-20",
      "origin_class": "IMPLEMENTATION_READBACK",
      "source_or_experiment_ids": [
        "BODY-SHA256:4e11f2af77fe48eb3cc54400dd0fd566c3001ac84166677a4cbe0f2b02133f02"
      ],
      "supporting_locators": [
        "content/staging/EA-750.md",
        "4e11f2af77fe48eb3cc54400dd0fd566c3001ac84166677a4cbe0f2b02133f02",
        "EA-750-RESEARCH-20261020-R1"
      ],
      "provenance_status": "CURRENT_APPROVED_BODY_HASH_MATCH"
    }
  ],
  "research_sections": [
    {
      "heading": "主要claim",
      "text": "Spot/preemptible GPUは「安価な通常GPU」ではなく、実行中断・再取得失敗・再起動待ちを通常系として扱うcapacity classです。中断通知の猶予はproviderごとに異なり、通知自体がbest-effortである場合もあります。このため、checkpoint設計は通知を受けてから初めて保存する方式だけに依存せず、平常時の周期checkpointと、中断signal受信時のbest-effort final checkpointを組み合わせます。\n\nAWS EC2 Spotは、stop/terminateの前に通常2分のinterruption noticeを出しますが、通知はbest effortであり、hibernateでは2分の猶予なしに処理が始まります。AWSはrebalance recommendationをより早い予兆として利用できる場合がある一方、2分noticeより早く来る保証はないと説明しています。\n\nGoogle Compute Engine Spot VMは、2026年時点でpreemption notice durationを120秒または0秒として作成時に設定できます。0秒がdefaultで、その場合はdedicated notice delayがなく、その後のbest-effort shutdown periodは最大30秒です。120秒を選ぶ場合はworkload側でpreemption metadataを監視してprogressを保存できます。GPU付きSpot VMもcapacity availabilityに依存し、managed instance groupが再作成を試みてもGPU resourceが利用可能である保証はありません。\n\nAzure Spot VMはcapacityまたはprice条件によりevictされ得て、作成後のavailability SLAはありません。Scheduled EventsのPreempt signalはeviction前の短い猶予を提供しますが、orchestrationは即時evictionやreplacement capacity不足も想定する必要があります。\n\nしたがって、Spot GPUの採否は単純な「時間単価の割引率」ではなく、失われるcompute、checkpoint I/O、restart/reload、reacquisition wait、fallback capacity、deadline/SLO penaltyを含むexpected costで評価します。"
    },
    {
      "heading": "運用上の意味・failure",
      "text": "最大のfailureは、checkpoint intervalをproviderの通知時間と同一視することです。通知がbest-effort、短時間、または0秒設定の場合、checkpoint sizeが大きいtraining/inference stateを中断時だけで安全に永続化できません。通常運転中にdurable checkpointを確定し、通知は追加の最終保存・drain・新規work停止に使います。\n\n第二のfailureは、VM再起動とGPU再取得を同義に扱うことです。Spot VMが停止・削除された後、同じGPU SKU/zoneで直ちにcapacityを再取得できるとは限りません。復旧設計には同一pool再試行だけでなく、別zone/instance type、standard/on-demand fallback、job queue待機、deadline超過時の昇格条件が必要です。\n\n第三のfailureは、checkpointを「書けた」で完了扱いすることです。中断直前のpartial upload、object listの更新順序、distributed rank間の世代不一致により、復旧時に破損または不完全checkpointを掴む可能性があります。checkpoint generation ID、atomic/commit marker、manifest/hash、readbackを用い、resume可能な世代だけを確定します。"
    },
    {
      "heading": "具体的control / implementation implication",
      "text": "1. workloadごとに `checkpoint_bytes`, `checkpoint_write_seconds`, `checkpoint_load_seconds`, `restart_seconds`, `expected_reacquisition_seconds`, `checkpoint_interval_seconds`, `deadline`, `fallback_policy` を計測値として持ちます。\n2. 周期checkpointをprimary durability mechanismとし、Spot interruption/rebalance/preemption signalでは新規batch投入停止、最新state flush、final checkpoint attempt、worker drainを実行します。\n3. checkpointはgeneration単位でimmutableに保存し、全shard/rankの完了後にcommit markerを確定します。復旧時は最新時刻ではなく、完全性検証済みの最新generationを選びます。\n4. interruption handlerはproviderごとのnotice contractを分離します。AWS 2-minute notice、GCP 120秒/0秒設定とbest-effort shutdown、Azure Scheduled Eventsを同じ固定timeoutとして実装しません。\n5. replacement strategyを `same_pool -> alternate_zone/pool -> alternate GPU SKU -> standard/on-demand fallback -> queue/defer` のように明示し、各段階の最大待機時間とcost ceilingを定義します。\n6. interruption testを定期実施します。AWSはFault Injection Service/EC2 interruption initiation、GCPはpreemption handling test、Azureはevictionを想定したorchestration testを使い、checkpoint生成からrestore、data-loader/sampler/RNG/optimizer state、distributed membershipまでreadbackします。\n7. 経済性は少なくとも次で比較します。`effective_spot_cost = spot_compute_cost + checkpoint_storage_IO + restart/reload_cost + expected_lost_compute + expected_reacquisition_idle + fallback_cost + deadline/SLO penalty`。これがstandard/on-demandの総費用・期限リスクを下回るworkloadだけをSpot適格とします。\n8. checkpoint intervalは固定値を全jobへ配らず、checkpoint costとinterruption exposureを計測してworkload class別に設定します。長時間job、大規模state、高い再取得待ちでは短縮余地を評価し、checkpoint I/Oがtraining throughputを損なう場合は非同期/増分checkpoint等を検討します。\n9. GPU capacityのavailabilityはquotaと別に監視します。quotaがあってもSpot capacityがあるとは限らないため、launch failure/reacquisition latencyを独立metricとして保存します。"
    },
    {
      "heading": "適用境界",
      "text": "providerの通知時間は契約・機能・設定・termination behaviorに依存します。AWSの2分noticeはhibernateでは同じ猶予を意味しません。GCP Spot VMは120秒noticeを選べる現行機能がありますがdefaultは0秒であり、古い「常に30秒」の理解を固定しません。AzureもScheduled Eventsを受信できることをcapacity保証や自動再開保証と解釈しません。\n\ncheckpoint経済性はworkload依存です。短時間batchや再実行が安いstateless inferenceではcheckpoint自体が不要な場合があります。一方、大規模trainingでoptimizer stateが巨大な場合、頻繁なfull checkpointはI/Oとstorage costを増やします。Spot採用を全GPU workloadの既定値にしません。"
    },
    {
      "heading": "counterevidence / limitations",
      "text": "Spot discountが大きくても、中断率・GPU scarcity・deadlineが厳しい環境ではeffective costがstandard/on-demandを上回り得ます。逆に、短いjob、十分なpool diversification、低いrestore cost、elastic worker構成では中断コストが小さく、Spot比率を高められます。\n\n過去の中断率だけから将来のavailabilityを保証できません。capacityはregion/zone/SKU/timeで変動し、providerのallocation behaviorも変化します。経済性モデルは実測のinterruption/reacquisition telemetryで継続更新します。"
    },
    {
      "heading": "OPEN_UNKNOWN",
      "text": "- 対象GPU SKU/region/zoneごとの実測interruption rateとreacquisition latency分布\n- checkpoint I/Oがshared storageへ与えるpeak負荷とthrottling\n- distributed training frameworkごとのrank loss/resume semantics\n- GPU SKU変更fallback時のmodel compatibilityとperformance差\n- deadline penaltyを含むbusiness costの換算方法\n- provider pricing/discount変動を含むbreak-even threshold"
    },
    {
      "heading": "SAFE_WORDING",
      "text": "「Spot/preemptible GPUは中断と再取得待ちを通常系として設計し、周期checkpointを主手段、provider通知をbest-effortの追加退避signalとして扱う。採否は時間単価だけでなく、失われるcompute、checkpoint I/O、restart/reload、capacity再取得、fallback、期限リスクを含む実効費用で判断する。」"
    },
    {
      "heading": "Source / provenance locators",
      "text": "- AWS EC2 User Guide, Spot Instance interruption notices: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html\n- AWS EC2 User Guide, Best practices for EC2 Spot: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html\n- AWS EC2 User Guide, Initiate a Spot Instance interruption: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/initiate-a-spot-instance-interruption.html\n- Google Cloud Compute Engine, Spot VMs: https://docs.cloud.google.com/compute/docs/instances/spot\n- Google Cloud Compute Engine, Create and use Spot VMs: https://docs.cloud.google.com/compute/docs/instances/create-use-spot\n- Microsoft Azure Architecture Center, Build workloads with Azure Spot Virtual Machines: https://learn.microsoft.com/azure/architecture/guide/spot/spot-eviction\n- Microsoft Azure Virtual Machines, About Azure Spot Virtual Machines: https://learn.microsoft.com/azure/virtual-machines/spot-vms"
    }
  ],
  "human_url": "/tools/ea-750-spot-preemptible-gpu-checkpoint",
  "markdown_url": "/agent/article.md?path=%2Ftools%2Fea-750-spot-preemptible-gpu-checkpoint",
  "json_url": "/agent/article.json?path=%2Ftools%2Fea-750-spot-preemptible-gpu-checkpoint"
}