xAI Grok audio models now available on Vercel AI Gateway

xAI's audio models are now live on AI Gateway. Realtime voice, text to speech, and speech to text are all available through the AI SDK with the same routing, observability, and spend controls as your other models.

These capabilities are available on the AI SDK 7 release.

1
npm install ai @ai-sdk/react @ai-sdk/gateway

Link to headingAvailable models

Link to headingRealtime

A voice agent has two pieces: a server route that mints a short-lived token, so your API key never reaches the client, and a browser component that connects with it.

Add the token route: this example sets model to xai/grok-voice-think-fast-1.0:

1
import { gateway } from '@ai-sdk/gateway';
2

3
export async function POST() {
4
  const { token, url } = await gateway.experimental_realtime.getToken({
5
    model: 'xai/grok-voice-think-fast-1.0',
6
  });
7
  return Response.json({ token, url, tools: [] });
8
}

Then connect from the browser. The useRealtimehook from @ai-sdk/react fetches that route and manages the WebSocket connection, microphone capture, and audio playback:

1
'use client';
2
import { experimental_useRealtime as useRealtime } from '@ai-sdk/react';
3
import { gateway } from '@ai-sdk/gateway';
4

5
// Inside a client component:
6
const { status, connect, startAudioCapture } = useRealtime({
7
  model: gateway.experimental_realtime('xai/grok-voice-think-fast-1.0'),
8
  api: { token: '/api/realtime/token' },
9
  sessionConfig: { turnDetection: { type: 'server-vad' } },
10
});
11
// Call connect(), then startAudioCapture(stream) to start talking.

Link to headingText to speech

Generate spoken audio from text with generateSpeech. Pass a voice and an output format, then write the result to a file with xai/grok-tts:

1
import { generateSpeech } from 'ai';
2
import { writeFile } from 'node:fs/promises';
3

4
const result = await generateSpeech({
5
  model: 'xai/grok-tts',
6
  text: 'Thanks for trying out AI Gateway.',
7
  voice: 'eve',
8
  outputFormat: 'mp3',
9
});
10
await writeFile('speech.mp3', result.audio.uint8Array);

Link to headingSpeech to text

Transcribe recordings into text with transcribe. This example uses xai/grok-stt:

1
import { transcribe } from 'ai';
2
import { readFile } from 'node:fs/promises';
3

4
const result = await transcribe({
5
  model: 'xai/grok-stt',
6
  audio: await readFile('audio.mp3'),
7
});
8
console.log(result.text);

Link to headingPlayground

You can also try the xAI audio models directly in the AI Gateway playground. Open the models list and click into any of the models to use them directly in the browser. The xai/grok-voice-think-fast-1.0 playground here allows you to talk to the agent and see responses instantly:

Link to headingMore information

• Realtime quickstart

• Speech quickstart

• See all xAI models