Batch AI

A large share of enterprise AI work—classification, enrichment, evaluation, summarization, embedding, media processing—does not require interactive latency. Batch paths can materially lower cost and raise throughput, but trade away immediate completion, ordering assumptions, mutability, and sometimes endpoint/tool coverage.

Enterprise AI Core / provider mapping / architecture / cost / operability / governance

When should a workload be routed from synchronous inference to provider batch execution, and what manifest, SLA, retry, reconciliation, and provenance controls preserve correctness?

• source class: official product documentation; exact title: Batch API; author/org: OpenAI; canonical URL: https://developers.openai.com/api/docs/guides/batch ; access date: 2026-09-16; API revision: current docs; locator: Batch API overview; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: support/availability. Current docs state 50% cost discount vs synchronous APIs, separate/higher rate-limit headroom, and completion within 24 hours; requests use unique custom_id for result correlation and an input file is single-model.
• source class: official product documentation; exact title: Batch processing; author/org: Anthropic; canonical URL: https://platform.claude.com/docs/en/build-with-claude/batch-processing ; access date: 2026-09-16; API revision: current docs; locator: pricing / batch lifecycle; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: support/limit. Current docs state 50% discount; submitted batch cannot be modified, requiring cancel/new submission for changes; batch rate limits are separate from Messages API limits.
• source class: official product documentation; exact title: Batch API; author/org: Google; canonical URL: https://ai.google.dev/gemini-api/docs/batch-api ; access date: 2026-09-16; API revision: current Gemini API; locator: overview / supported models / pricing / SLO; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: support/availability. Current docs state asynchronous processing at 50% standard cost, target 24-hour turnaround, generateContent-only note, model-specific support, and context caching support within batch requests.
• source class: official product documentation; exact title: Batch Predict with Gemini using GCS data / BigQuery data; author/org: Google Cloud; canonical URLs: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/samples/generativeaionvertexai-batch-predict-gemini-createjob-gcs and https://docs.cloud.google.com/vertex-ai/generative-ai/docs/samples/googlegenaisdk-batchpredict-with-bq ; access date: 2026-09-16; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: implementation/context. Demonstrates Vertex AI batch prediction paths using GCS and BigQuery inputs/outputs.

Batch is an execution class, not merely “many synchronous calls.” The Core should route work by deadline/SLO, dependency graph, mutability, ordering, data locality, output size, and cost sensitivity. Every input needs a stable request_id/custom_id and immutable request manifest/hash; outputs must be reconciled by ID, not assumed position. Batch creation, partial failure, cancellation, expiration, completion, output retrieval, and retry are separate states. Provider price discounts are current product claims, not permanent architecture guarantees.

No interactive latency guarantee; provider turnaround target can be too slow for operational triggers; submitted jobs may be immutable; partial request failures; duplicate resubmission after uncertain status; result ordering mismatch; model/endpoint/tool/modality coverage gaps; input/output file retention and residency concerns; batch completion notification loss; stale business data by execution time; large failed batches causing expensive replay.

Provider/model/endpoint/modality; region; input source/file limits; request count/token limits; completion window; batch-specific quota/rate limits; data residency; pricing; cancellation/expiration behavior; output retention.

OpenAI, Anthropic, Gemini API, and Vertex AI expose current batch/asynchronous bulk inference mechanisms. Exact model coverage and quotas are provider-volatile.

identical workload through synchronous and batch paths with fixed model/revision. Measure quality equivalence, completion-time distribution, throughput, cost/request and cost/token, partial-failure rate, reconciliation correctness, retry amplification, duplicate-output rate, and operational recovery time. Include cancellation and malformed-request fixtures.

Persist batch manifest hash, submitter/service identity, provider/project/region, model revision, request IDs, input/output object references and hashes, submit/start/complete/expire timestamps, per-item status, retry lineage, cost, and deletion/retention state. Batch discount must not justify moving sensitive data to a weaker residency/retention path.

“For non-urgent independent workloads, provider batch APIs can trade interactive latency for lower unit cost and higher bulk throughput. Treat batch as an asynchronous job system with explicit manifests, item IDs, reconciliation, partial-failure handling, and retention controls.”

Add BATCH_EXECUTION_CONTRACT: deadline/SLO, immutable input manifest/hash, provider/model/revision, request_id, job_id, lifecycle state, output mapping/hash, retry lineage, completion notification, retention/deletion, and cost. Re-pin provider discounts and turnaround commitments at publication.

exact per-model batch coverage, quotas, file/request limits, and enterprise residency/retention matrix as of publication. STOP_CONDITION: architecture and failure contract established; volatile provider matrix deferred to update Work.

---

OpenAI Batch API numeric/product claims stay as-of; batch acceptance, completion window and item outcomes remain distinct.