Speech AI大全

Enterprise AIの音声経路は、単なる音声入出力ではなく、認識・話者分離・turn detection・LLM/agent orchestration・音声合成・interrupt/barge-in・privacy/loggingが直列する実時間システムである。STT精度やTTS自然性だけを単独評価すると、会話成立性、tool実行、安全性、地域/モデルavailabilityを見落とす。公開後更新ではSpeech AIを非LLM部品として独立させつつ、音声AgentでLLM/Agent層と接続する必要がある。

non-LLM component / Enterprise AI Core / provider mapping / interoperability / benchmark / implementation / privacy / operability

• Speech AIはSTT/TTS/diarization/realtime agentを別能力として扱うべきか。
• realtime voice agentの品質はWERやTTS naturalnessだけで評価可能か。
• providerの音声機能はmodel/locale/region/API modeでどこまでavailabilityが変わるか。
• audio-native/realtime経路とSTT→text LLM→TTS cascadeは同一の運用特性を持つか。

[E1] official product documentation | Voice Live API Overview | Microsoft | https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live | accessed 2026-09-15 | API family: Voice Live; current docs expose gpt-realtime-1.5/gpt-realtime/gpt-realtime-mini and Data Zone variants | locator: Features / API design and compatibility / model table | VERIFIED_SPEC + PRODUCT_VOLATILE | support/context [E2] official API how-to | How to use the Voice Live API | Microsoft | https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-how-to | accessed 2026-09-15 | api-version=2026-04-10; transcription choices include azure-speech, mai-transcribe(preview), whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize depending on chat model | locator: WebSocket endpoint / Audio input transcription | VERIFIED_SPEC + PRODUCT_VOLATILE | support/limit [E3] official product documentation | What is speech to text? | Microsoft | https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-to-text | updated/published 2026-07 approx.; accessed 2026-09-15 | current Speech service | locator: real-time / fast / batch / diarization | VERIFIED_SPEC + PRODUCT_VOLATILE | support [E4] official product documentation | Detect different speakers in an audio recording | Google Cloud | https://docs.cloud.google.com/speech-to-text/docs/multiple-voices | accessed 2026-09-15 | Speech-to-Text; V2 request example; diarization_config | locator: Speaker diarization / RecognitionFeatures | VERIFIED_SPEC + PRODUCT_VOLATILE | support/limit [E5] official model documentation | Chirp 2: Enhanced multilingual accuracy | Google Cloud | https://docs.cloud.google.com/speech-to-text/docs/models/chirp-2 | accessed 2026-09-15 | model id chirp_2; STT V2; StreamingRecognize language support limited by locale | locator: model details / API methods | VERIFIED_SPEC + PRODUCT_VOLATILE | support/availability [E6] official product documentation | Supported voices and languages | Google Cloud Text-to-Speech | https://docs.cloud.google.com/text-to-speech/docs/list-voices-and-types | accessed 2026-09-15 | Chirp 3 HD current voice family | locator: voice type table / Chirp 3 HD notes | VERIFIED_SPEC + PRODUCT_VOLATILE | support/limit [E7] official quota documentation | Quotas & limits | Google Cloud Text-to-Speech | https://docs.cloud.google.com/text-to-speech/quotas | accessed 2026-09-15 | current quota table; subject to change | locator: content limits / requests limit | VERIFIED_SPEC + PRODUCT_VOLATILE | limit/operability [E8] official product documentation | Partitioning speakers (diarization) | Amazon Transcribe | https://docs.aws.amazon.com/transcribe/latest/dg/diarization.html | accessed 2026-09-15 | current Transcribe | locator: max 30 speakers; batch/streaming speaker labels | VERIFIED_SPEC + PRODUCT_VOLATILE | support/limit [E9] official product documentation | Transcribing multi-channel audio | Amazon Transcribe | https://docs.aws.amazon.com/transcribe/latest/dg/channel-id.html | accessed 2026-09-15 | current Transcribe | locator: two-channel identification | VERIFIED_SPEC + PRODUCT_VOLATILE | context/limit [E10] official product documentation | Generative voices | Amazon Polly | https://docs.aws.amazon.com/polly/latest/dg/generative-voices.html | accessed 2026-09-15 | Generative engine; regional availability current | locator: region list / real-time and asynchronous synthesis / formats | VERIFIED_SPEC + PRODUCT_VOLATILE | support/availability [E11] primary research | VoiceAgentBench: Are Voice Assistants ready for agentic tasks? | Jain et al. | https://arxiv.org/abs/2510.07978 | 2025-10-09; accessed 2026-09-15 | benchmark paper | locator: 5,500+ synthetic spoken queries; tool selection/workflows/multiturn/safety; reported gaps in orchestration, Indic generalization, adversarial robustness | RESEARCH | counter/context [E12] primary research | RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems | Ayllon et al. | https://arxiv.org/abs/2607.14846 | 2026-07-16; accessed 2026-09-15 | benchmark paper | locator: multidimensional TTS/STS/SU/ASR; dimension-specific performance; real-world accent/emotion/noise failures | RESEARCH | counter/context [E13] primary research | SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia | Liao et al. | https://arxiv.org/abs/2609.09672 | 2026-09-09; accessed 2026-09-15 | 97,194 samples / 99 eval sets / 597h / 11 SEA languages | locator: task and language coverage; temporal/emotion/translation gaps; low-resource prompt gap | RESEARCH | counter/context

F1 [VERIFIED_SPEC]: STT、speaker diarization、TTS、realtime voice agentは別能力であり、同じprovider内でも別API/model/modeとして実装される。Microsoft Voice Liveはaudio input/output、noise suppression、echo cancellation、interruption/end-of-turn、function callingを一つのrealtime layerに統合するが、入力transcription modelは別選択である。 F2 [VERIFIED_SPEC]: diarizationとchannel identificationは同義ではない。Amazon Transcribeはspeaker diarizationで最大30 speakerをlabel可能だが、channel identificationは最大2 channel。Google STTもdiarizationをspeaker change/number labelとして提供する。物理channel分離を話者識別能力へ外挿しない。 F3 [VERIFIED_SPEC + PRODUCT_VOLATILE]: availabilityはmodel × locale × region × API/modeで変動する。Google Chirp 2はV2専用でStreamingRecognizeのlanguage supportに制約があり、Chirp 3 HD TTSはglobal/eu/us endpointを持つ一方、現行docsではregionalization/data residencyの対象外と明記される。AWS Polly Generative voicesもregion限定。 F4 [INFERENCE grounded in E1-E10]: realtime voice agentには少なくとも input capture → VAD/turn detection → ASRまたはaudio-native understanding → agent/tool reasoning → response generation → TTS/audio generation → playback → interruption/barge-in のlatency chainがある。単一のend-to-end latency値だけではbottleneckを特定できない。 F5 [RESEARCH]: VoiceAgentBenchは音声agent評価をtranscriptionからtool selection、multi-tool、multi-turn、安全性へ広げ、実験上orchestration/generalization/adversarial robustnessの不足を報告する。製品保証・failure rateへは外挿しない。 F6 [RESEARCH]: RW-Voice-EQ BenchはTTS naturalness/expressiveness/identity stability/reliability、STS、speech understanding、ASR robustnessが独立に崩れ得ることを示す。SEA-SpeechBenchも多言語・paralinguistic・temporal理解で大きな差を報告する。したがってWERだけをSpeech AI全体の品質指標にしない。

• clean-speech WER改善はnoise、accent、code-switch、overlap、emotion、far-field、telephony、domain terminologyでの精度を保証しない。
• diarization speaker IDは通常session内generic labelであり、本人identity verificationではない。speaker recognition/biometricsと混同禁止。
• TTS naturalnessが高くてもpronunciation、number/date/entity rendering、voice identity stability、latency、interruptibility、安全なconsent運用は別軸。
• audio-native modelはcascadeより常に低遅延/高品質とは確定しない。network、turn detector、tool latency、synthesis/playbackが支配的になり得る。
• transcriptが正しくてもtool orchestrationやspoken instruction handlingが失敗し得る。逆にASR文字列差があってもtask successする場合がある。
• benchmark結果はfixture/language/model snapshot依存であり、特定商用tenantのSLAや現行failure rateを証明しない。

provider; model/voice ID; API revision; region/endpoint; locale/language; streaming vs batch; realtime vs async; diarization/channel mode; subscription/resource type; quota/concurrency; preview/GA status; data-residency mode; custom voice/voice cloning consent and eligibility.

Microsoft: Voice Liveはrealtime agent統合層として現行docsで提供。WebSocket api-version 2026-04-10。transcription backend/model選択とchat/realtime model互換性を別管理。 Google: STT V2/Chirp系、speaker diarization、TTS Chirp 3 HD等を別service/modelとして提供。locale/endpoint/feature差を固定して使う必要がある。 AWS: Transcribeでstreaming/batch、speaker diarization/channel identification、PollyでTTS/generative voice。realtime conversational agent全体はこれら部品を組み合わせる構成として評価し、単一Speech製品とみなさない。

Core評価は最低でも次を分離する: ASR WER/CER + entity/number/date accuracy; diarization DER/JERまたはspeaker-attributed word error; TTS intelligibility/pronunciation + naturalness/expressiveness + speaker/voice stability; turn detection false-cut/late-turn; interruption/barge-in success; first-audio latency/turn latency/p95; tool selection/argument/task success; multilingual/code-switch/noise/overlap robustness; safety/privacy behavior。単一総合scoreは補助値に留める。

Privacy: raw audio、transcript、speaker label、voiceprint/embedding、custom-voice reference、tool transcriptを別artifactとしてretention/consent/access control対象化。diarization labelを本人識別情報と誤認しない一方、voice biometricを扱う場合は別privacy gateを設ける。 Audit: session ID、audio input hash/locator、model/voice IDs、API revision、locale、timestamps、partial/final transcript、speaker/channel labels、turn boundary、tool calls、interrupt events、final text/audio provenanceを保持。 Cost/Operability: audio duration、characters/tokens、streaming session/concurrency、custom model/voice、network egress等のmeterを分離。quotaはcurrent確認必須。Google TTSのcurrent docsではrequest/content/concurrent streaming limitsが明示され、変更可能性も明記される。

「Speech AIはSTTやTTSの単体機能ではなく、認識、話者分離、turn-taking、Agent/tool実行、音声合成、interrupt処理までを含む実時間経路として評価する。各機能のavailabilityと品質はmodel、language、region、API modeで異なるため、provider名だけで能力を一般化しない。WER、自然性、会話latency、task success、安全性は別指標として測る。」

Core候補として SPEECH INTERACTION CONTRACT を追加検討。fields: session_id, provider, region, input_mode, audio_codec/sample_rate, STT/audio-model ID+version, locale, diarization/channel config, VAD/turn config, agent/model ID, tool policy, TTS voice/model ID, output codec, retention/consent, latency trace, transcript provenance, interruption events, task outcome。 TEST_PROFILE（未実施）: consent済みsynthetic/public audioのみで、clean/noise/far-field/telephony、accent、多言語/code-switch、2-speaker overlap、interrupt、numbers/dates/proper nouns、single/multi-tool taskを固定fixture化。pipeline別に cascade(STT→text agent→TTS) と audio-native/realtime を同一taskで比較し、WERだけでなく task success、DER、turn-cut、barge-in、p50/p95 first-audio/turn latency、cost/minute、audit completenessを記録する。

/ STOP_CONDITION

OpenAI direct Realtime APIの2026-09-15時点の正確なcurrent model/API revision・region/data controlsは今回一次資料を十分固定できず、provider mapping採用時に公式docsで再採集する。

provider横断で比較可能な標準化されたend-to-end voice-agent latency/safety benchmarkの成熟度。

STT/TTS/diarization/realtime agentの分離、current Microsoft/Google/AWS implementation、availability gates、multidimensional benchmark evidence、failure envelope、future Core contract/test profileまで後続Workが採否判断可能な粒度に到達したためEA-071 bounded packetを閉じる。

---