Inside Grok Voice 3.0: How SpaceXSI's Voice Agent Turns a Phone Call Into Events

01A voice call is a stream of JSON events over one socket

Every Grok Voice session is a single WebSocket to wss://api.x.ai/v1/realtime?model=grok-voice-latest. Your client sends microphone audio up, and the server sends synthesized speech back down, both as discrete events on the same connection (Speech to Speech guide, "Quick Start"[2]).

The first thing a client sends is session.update. It carries the system prompt (instructions), the voice, the tool list, the audio codecs, and the turn-detection mode. Audio then flows as input_audio_buffer.append events, base64-encoded by default. Speech comes back as response.output_audio.delta events, which you decode and play as they arrive rather than waiting for the full reply (guide, "Audio Transport" and "Additional High-Impact Recommendations"[2]).

Base64 inflates audio by about a third. The docs offer a way out: set transport to "binary" and raw codec bytes travel as WebSocket binary frames, while lifecycle events like response.done stay JSON. Output is strict about this. A change to output.transport only takes effect at the next response boundary, so one utterance never mixes the two (guide, "Audio Transport"[2]).

That design keeps the wire simple. It also means your client owns playback, buffering, and timing, which is where most voice bugs come from.

02The server decides when the caller has stopped talking

Turn-taking is the problem that makes voice agents feel broken, and the API puts it on the server by default. With turn_detection.type set to server_vad, the server runs voice activity detection and decides when the user has finished. In that mode the client only ever sends append events (guide, "Audio"[2]).

Turn it off and you take the job back. You then send a commit event when the user is done and a clear event to throw away uncommitted audio. The docs recommend server VAD "for automatic, natural barge-in," meaning the caller can interrupt the agent mid-sentence (guide, "Additional High-Impact Recommendations"[2]).

SpaceXSI's announcement claims a 97% score on Full Duplex Bench, which it describes as a measure of pauses, turn-taking, and interruptions (Introducing Grok Voice 3.0, "The Most Accurate Voice Model"[1]). I haven't opened Artificial Analysis's own results page, so treat that number as SpaceXSI's report of a third-party score. The docs don't say what VAD model runs server-side or expose its thresholds beyond an optional idle_timeout_ms.

That's the trade. You get barge-in for free, but you can't tune the endpointing if it cuts off a caller who pauses to read a serial number.

03Tool calls are a four-step handshake you have to sequence

A voice agent that can't touch your systems is a phone tree with a nicer voice. The API supports two classes of tools, and the split matters (guide, "Using Tools"[2]).

Server-side tools run inside SpaceXSI. Those are web_search, x_search, file_search over your uploaded collections, and remote mcp servers, where SpaceXSI holds the connection and executes the call. MCP servers must speak Streaming HTTP or SSE. Client-side function tools are yours: the model emits arguments, and your code does the work.

The function flow runs in four documented steps:

1. The server sends response.function_call_arguments.done with a call_id and JSON arguments.
2. Your code runs the function.
3. You send conversation.item.create with a function_call_output item carrying that call_id.
4. You send response.create so the model continues.

Two failure modes are spelled out. If the model wants several functions at once, it emits several done events before any audio, and you must return every output before sending a single response.create. Send it early and the model answers without the full context (guide, "Parallel Tool Calling"[2]).

The second is audio overlap. The server delivers all of a turn's audio deltas first, then the function call events alongside response.done. Your speaker is probably still playing that audio. If you fire response.create the moment your tool returns, the next reply starts generating on top of it. The docs tell you to wait for playback to finish and show a "thinking" indicator in the meantime (guide, "Avoid Audio Overlap During Tool Calls"[2]).

So the protocol doesn't hide latency from tools. It hands you the pause and expects you to design around it.

04Worked example: a Starlink-style replacement call

Take the job SpaceXSI says its own agent does at Starlink: verify a customer, troubleshoot hardware, issue credits, and ship replacement equipment (announcement, "Used by Millions of Customers at Tesla & SpaceX"[1]). Here's how that maps onto the documented API. The mapping is my construction, not SpaceXSI's published architecture.

The call arrives over the phone network. The docs support SIP routing and native G.711 μ-law and A-law (audio/pcmu, audio/pcma) at 8 kHz, so there's no transcoding step on a telephony leg (guide, "Configuring Audio Format" and "Built for Enterprise Voice"[2]).

Before the model says anything, the client sends a force_message item: "This call is being recorded." That item is synthesized verbatim without involving the model, and with interruptible: false the server drops caller audio until playback ends. You don't send response.create for it. The force message is the turn (guide, "Force Message"[2]).

The caller reads out a dish serial number. That's a transcription risk, so the session sets audio.input.transcription.keyterms with product names (up to 100 terms, 50 characters each). The model calls lookup_account and check_dish_status in parallel. Your client resolves both, posts two function_call_output items, waits for the agent's "one moment" line to finish playing, then sends one response.create.

The agent decides a replacement is warranted and calls ship_replacement. If the caller's line drops here, session resumption rescues the context: opt in with resumption.enabled: true, save the id from conversation.created, and reconnect with ?conversation_id=. Transcripts, tool calls, and your function outputs replay. History expires after 30 minutes of inactivity (guide, "Session Resumption"[2]).

At $0.08 a minute, a six-minute call costs $0.48 in model time (announcement, "Pricing and Availability"[1]). That figure excludes your telephony carrier and whatever your own tools cost to run.

The limitation: resumption replays text and tool state, not audio. A caller mid-sentence when the line dropped will have to repeat themselves.

05"Speech to speech" still has a text layer inside

SpaceXSI calls this a speech-to-speech model, and the docs contain hints that text sits in the middle of the pipeline. The replace parameter swaps phrases "in the model's output ... before text-to-speech," so only the audio changes and the transcript keeps the original spelling (guide, "Pronunciation Replacements"[2]). Session parameters also include ASR controls (language_hint, keyterms) that "bias transcription."

That doesn't prove the model cascades speech to text to speech for every token. It does show that output text exists before synthesis, at least where replace applies. SpaceXSI hasn't published a model card or architecture paper for Grok Voice 3.0 that I could find, so the internal design is a documented gap.

The engineering consequence is practical either way. Pronunciation fixes go through replace, and because matching needs whole-word boundaries, "Acme-Mobile" won't match a key of "Acme Mobile" (guide, "Pronunciation Replacements"[2]).

06Compatibility with OpenAI Realtime is close, with a few edges

The API is built to be a drop-in target for OpenAI Realtime clients. You change the base URL and key, and the OpenAI SDK connects (guide, "Migrating from OpenAI Realtime"[2]).

The edges are where ported code breaks. Input transcription arrives as conversation.item.input_audio_transcription.updated, a cumulative transcript that may revise earlier text, not OpenAI's incremental delta. Several events never fire, including conversation.item.done and rate_limits.updated. Code that waits on either will hang (guide, "OpenAI Realtime API Compatibility"[2]).

Reasoning is on by default too. reasoning.effort defaults to "high" on Grok Voice 3.0, and you can set it to "none" (guide, "Step 3"[2]). The docs don't quantify the latency cost of that default, and for a phone agent that's the number you'd want first.

07What the documentation shows versus what the launch post claims

The docs establish the protocol, codecs, tool flow, resumption behavior, and the model aliases: grok-voice-latest now points at grok-voice-3.0, and grok-voice-2.0 is deprecated with an unchanged API (guide, "Model Selection"[2]).

The launch post is where the performance claims live: the 86.9 Speech-to-Speech Index score, sub-second latency, "40% lower latency" than GPT-Live-1 on τ-Voice, and the Starlink figures of 20,000 calls a day with roughly 70% resolved without a human. Every one of those is a single-source claim from SpaceXSI, and the Starlink numbers come from SpaceXSI's own sister company. Test latency and resolution on your own call mix before you plan staffing around them.