You want a video that plays while it generates. You also want to change the prompt mid-run. A finite video job stops with a file. generateLiveVideo() opens a session instead.
Call generateLiveVideo() on the server. It returns a connect payload: a model id, a prompt, and a token. The browser uses that payload to open the stream.
Experimental. The API can change. Live models bill per session-second while a GPU is held.
Pick one adapter. They do not return a download URL.
import { generateLiveVideo } from '@tanstack/ai'
import { reactorVideo } from '@tanstack/ai-reactor'
const live = await generateLiveVideo({
adapter: reactorVideo('helios'),
prompt: 'A chef tosses noodles in a steel wok, flames leaping',
})
// Hand live.token, live.model, and live.prompt to the browser.import { generateLiveVideo } from '@tanstack/ai'
import { falLiveVideo } from '@tanstack/ai-fal'
const live = await generateLiveVideo({
adapter: falLiveVideo('minimax/h3-max/director'),
prompt: 'Live shopping stream: a host holds up a gold watch to camera',
})
// live.model is 'fal-ai/minimax-h3-max-director'. Hand it to the browser.Set REACTOR_API_KEY or FAL_KEY, or pass apiKey in the adapter config. Do not put the key in the JSON body.
A full app lives in examples/ts-react-media.
The server half is the same for every live adapter. The browser client is not.
Install @reactor-team/js-sdk. Connect with the token. Helios uses set_sr_scale, then set_prompt, then start.
import { Reactor } from '@reactor-team/js-sdk'
const reactor = new Reactor({ modelName: live.model })
const video = document.querySelector('video')
reactor.on('trackReceived', (name, _track, stream) => {
if (name !== 'main_video') return
if (!video) return
video.muted = true
const attach = () => {
video.srcObject = null
video.srcObject = stream
void video.play().catch((error: unknown) => {
if (error instanceof DOMException && error.name === 'AbortError') return
})
}
attach()
for (const track of stream.getTracks()) {
track.addEventListener('unmute', attach)
}
})
await reactor.connect(live.token)
await reactor.sendCommand('set_sr_scale', { sr_scale: '2x' })
await reactor.sendCommand('set_prompt', { prompt: live.prompt })
await reactor.sendCommand('start', {})FastH3 uses enqueue only. LongLive uses set_shot then start.
Helios can also take a seed image. Pass a File from <input type="file">. Do not send base64.
const picker = document.querySelector('input[type="file"]')
if (!(picker instanceof HTMLInputElement)) {
throw new Error('Pick a seed image')
}
const file = picker.files?.[0]
if (file === undefined) {
throw new Error('Pick a seed image')
}
const image = await reactor.uploadFile(file)
await reactor.sendCommand('set_conditioning', { prompt: live.prompt, image })
await reactor.sendCommand('start', {})A later set_prompt morphs the shot at the next chunk.
See the Reactor adapter for model ids.
Install @fal-ai/client@alpha. Keep FAL_KEY on the server. live.model is the WMA app id. Open WMA through a proxy that attaches the key. Do not send live.token as Key credentials.
The example proxy is src/routes/api.fal.proxy.ts in examples/ts-react-media. It attaches the key and forwards only to wma.fal.run (/ice, /session, /session/heartbeat) and Director /ice.
import { createFalClient } from '@fal-ai/client'
import { wma } from '@fal-ai/client/realtime'
const fal = createFalClient({ proxyUrl: '/api/fal/proxy' })
const video = document.querySelector('video')
const session = fal.realtime.open(wma(live.model), {
receive: ['video', 'audio'],
onError: (error) => {
console.error(error)
},
onMedia: (stream) => {
if (!video) return
video.muted = true
video.srcObject = stream
void video.play().catch((error: unknown) => {
if (error instanceof DOMException && error.name === 'AbortError') return
})
},
})
session.send({
type: 'configure',
prompt: live.prompt,
prompt_version: 1,
protocol_version: 1,
})
await session.readyTo steer, send { type: 'prompt', prompt, prompt_version } and increase prompt_version each time. To stop, send { type: 'stop' } and close the session.
Director bills a 60 second minimum. Resolution is 480p or 768p. See the fal adapter.
| Adapter | Id | What it does |
|---|---|---|
| reactorVideo() | helios | Interactive realtime video |
| reactorVideo() | fast-h3 | Fast live clips on a live track |
| reactorVideo() | longlive-v2 | Shot-based live clips |
| falLiveVideo() | minimax/h3-max/director | Steerable live stream over WMA |
For Orbis and LingBot, use World Generation. For a file that finishes, use Video Generation.
A server call that returns a connect payload. The browser opens the stream and steers it until you stop.