Semantic／Prompt Cache

Repeated long system prompts, tool definitions, reference context, and semantically equivalent user requests can dominate enterprise inference cost and latency. However, provider prompt/context caches and application semantic caches are different mechanisms with different correctness, privacy, invalidation, and observability properties. Treating them as one feature can create stale or unauthorized answer reuse.

Enterprise AI Core / provider mapping / cost / architecture / security / privacy / audit / implementation

Which cache layer can safely reduce repeated inference, under what cache-key/TTL/version/ACL conditions, and how should false semantic hits, stale results, tenant isolation, and cache invalidation be governed?

• source class: official product documentation; exact title: Prompt caching; author/org: OpenAI; canonical URL: https://developers.openai.com/api/docs/guides/prompt-caching ; access date: 2026-09-16; revision/model: current docs, GPT-5.6+ behavior documented; locator: How caching works / Cache location / Summary of model differences; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: support/context. Key evidence: GPT-5.6+ minimum cacheable visible prefix 1,024 tokens; cache writes 1.25x uncached input rate and reads 0.1x; cache reuse is prefix/breakpoint based; caches are not shared across organizations or regional processing boundaries; prompt_cache_key can separate accounting and reduce cache-hit probing across users.
• source class: official product documentation; exact title: Prompt caching; author/org: Anthropic; canonical URL: https://platform.claude.com/docs/en/build-with-claude/prompt-caching ; access date: 2026-09-16; revision: current docs; locator: TTL support / pricing / cache storage and sharing; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: support/limit/context. Key evidence: default 5-minute and optional 1-hour TTL; 5-minute write 1.25x, 1-hour write 2x, cache read generally 0.1x base input; cache entries isolated between organizations and, on listed platforms, workspaces; no manual clear, expiry-based lifecycle.
• source class: official product documentation; exact title: Context caching; author/org: Google; canonical URL: https://ai.google.dev/gemini-api/docs/caching ; updated: 2026-09-02 UTC; access date: 2026-09-16; model/API: Gemini 2.5+ / current Gemini API; locator: Implicit caching; evidence label: VERIFIED_SPEC / PRODUCT_VOLATILE; relation: support/availability. Key evidence: implicit caching enabled by default for Gemini 2.5+; minimum token thresholds are model-specific; explicit cache objects require generateContent rather than Interactions API.
• source class: author-maintained OSS; exact title: GPTCache: A Library for Creating Semantic Cache for LLM Queries; author/org: ZillizTech; canonical URL: https://github.com/zilliztech/gptcache ; access date: 2026-09-16; version: exact current release/tag not pinned in this run; locator: README / How does it work?; evidence label: OSS_PRACTICE; relation: implementation/context. Key evidence: semantic cache uses embeddings/vector similarity and explicitly acknowledges false-positive hits and false-negative misses.
• source class: author issue/failure report; exact title: Semantic Cache is not working #651; author/org: GPTCache issue reporter; canonical URL: https://github.com/zilliztech/GPTCache/issues/651 ; publication: 2024-09-16; access date: 2026-09-16; evidence label: SOURCE_REPORT; relation: counter/limit. Reported failure: same cached response returned for different questions in one semantic-cache setup. This is not generalized to all versions/deployments.
• source class: author issue/security report; exact title: Possible Leakage of Private Info in Semantic Cache Requests #652; author/org: GPTCache issue reporter/security research group; canonical URL: https://github.com/zilliztech/GPTCache/issues/652 ; publication: 2024-09-18; access date: 2026-09-16; evidence label: SOURCE_REPORT; relation: counter/security. Report alleges timing differences may expose private attributes; unverified beyond issue scope and not generalized.
• source class: primary research; exact title: SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models; author/org: Jiaxing Li et al.; canonical URL: https://arxiv.org/abs/2406.00025 ; publication: 2024-05-24; access date: 2026-09-16; evidence label: RESEARCH; relation: context/benchmark. Research reports semantic-aware caching can improve hit ratio/token savings over compared GPTCache configurations; benchmark does not establish enterprise correctness/privacy.

Provider prompt/context caching is primarily reuse of model-side computation for matching prompt prefixes/context; it does not by itself reuse an old generated answer. Application semantic caching may return a previously generated answer for a merely similar request and therefore creates an additional correctness and authorization boundary. Cache keys must include all answer-affecting dimensions that are not safely encoded in the matched content: tenant/workspace/user security scope as needed, model/revision, system/developer prompt version, tool schema/version, retrieval corpus snapshot or freshness epoch, policy/safety version, locale, output schema, and material generation parameters. Semantic cache should default to bypass for high-stakes, authorization-sensitive, freshness-sensitive, personalized, or action-producing requests unless a bounded equivalence contract is proven.

False semantic hit; stale answer after source/policy/model change; cross-tenant or cross-user leakage; cache-hit timing/probing; cached answer containing revoked data; low hit rate from variable prefixes; cache-write cost exceeding savings; cache stampede; inability to manually purge provider-managed ephemeral cache; semantic similarity that ignores negation, numbers, time, identity, or authorization context. Provider cache isolation claims do not prove application-layer semantic cache isolation.

Provider/model/API endpoint; minimum cacheable tokens; implicit vs explicit mode; TTL/retention; region/data-residency path; batch interaction; account/organization boundary; application cache store/vector index; eviction/invalidation support.

Provider prompt/context caching is production product capability in OpenAI, Anthropic, and Gemini APIs as of 2026-09-16, with materially different semantics. Semantic response caching is implementable via OSS/application infrastructure but requires application-owned equivalence, ACL, invalidation, and audit logic.

fixed workload with exact-repeat, prefix-repeat, paraphrase, negation flip, numeric/date change, user/tenant change, ACL change, corpus revision, policy revision, model revision, and no-cache controls. Measure token cache-hit rate, semantic-cache precision/recall, false-hit severity, stale-hit rate, unauthorized-hit rate, p50/p95 latency, provider input/write/read cost, cache-store cost, and purge/invalidation propagation time. Never optimize hit ratio without false-hit and stale-hit constraints.

Audit cache layer, cache key namespace, matched source entry, creation/expiry time, model/prompt/corpus/policy versions, tenant/ACL scope, hit/miss/bypass reason, and invalidation event. Do not place raw secrets/PII in application cache keys; use scoped opaque identifiers/hashes where appropriate. Cost model must include cache writes, misses, storage/vector search, invalidation, and false-hit remediation, not only read-token discount.

“Prompt/context caching can reduce repeated-prefix processing cost and latency without reusing an old answer, while semantic response caching may skip inference by reusing an answer for a similar request. The latter therefore requires explicit equivalence, freshness, authorization, invalidation, and audit controls.”

Build one Core cache contract, but expose two modes: PROVIDER_PREFIX_CACHE and APPLICATION_RESPONSE_CACHE. Require cache_namespace + model_ref + prompt/tool/policy/corpus version + security_scope + TTL/freshness + hit provenance. Re-pin provider model/TTL/pricing/retention behavior at publication time.

exact current GPTCache release/tag/commit and maintenance status; provider-specific data-retention interaction for every enterprise plan/region; measured false-hit envelope on enterprise corpora. STOP_CONDITION: packet sufficient for architecture inclusion; product-volatility matrix must be refreshed at update time.

---

GPT-5.6+ prompt-cache semantics, pricing, TTL and breakpoint behavior are model/version-pinned and not frozen across releases; cache hit does not imply deterministic output.