Show HN: Realtime talking avatar running free locally (90+FPS on RTX4090)

OpenSpline released a self-hosted tool that turns a portrait image and generated audio into synchronized talking-avatar video and speech on a local NVIDIA GPU, reporting 90+ frames per second on an RTX 4090. The Python-first package (pip install openspline) streams PCM audio to an avatar and ships connectors for OpenAI Realtime, with Node support, a built-in viewer, React and web components, and one active session per GPU worker. The backend requires CUDA 12.8 for Blackwell and RTX 50-series cards and CUDA 12.6 for older NVIDIA GPUs, uses FP32 on pre-Ampere cards, and does not support AMD or Apple GPUs.

Add a live face to any voice AI agent in five lines. Turn a portrait and generated audio into synchronized video and speech on your own GPU. Try Online no GPU or installation needed https://dashboard.openspline.dev/explore - Bring your agent. Keep your prompts, tools, microphone, and voice provider. - Python first, Node supported. Stream audio or use a ready-made connector. - Ready to display. Built-in viewer, React component, and web component. - Self-hosted. One active session per GPU worker; add GPUs for concurrency. Quick start quick-start · Connectors connectors · Configuration configuration · Node SDK node-sdk · Frontend frontend · GPU workers gpu-workers Avatars created with OpenSpline Cloud https://github.com/openspline/openspline/blob/main/experiments/openspline cloud . Click a preview or Try online to open the live avatar. | Nikola Tesla | Meera | Lisa | |---|---|---| | Try online https://dashboard.openspline.dev/nikola-tesla1 | Try online https://dashboard.openspline.dev/meera | Try online https://dashboard.openspline.dev/hemant-virtual-testing2 | | Acharya | Elon Musk | Ye | | Try online https://dashboard.openspline.dev/acharya-ji | Try online https://dashboard.openspline.dev/elon | Try online https://dashboard.openspline.dev/ye-west | Create, share, and embed your own avatar at openspline.dev https://openspline.dev , or explore the Cloud app source and embed examples https://github.com/openspline/openspline/blob/main/experiments/openspline cloud/README.md use-in-code . Run the following command in a linux device/WSL2 with NVIDIA GPU: curl -fsSL https://raw.githubusercontent.com/openspline/openspline/main/install.sh | sh The installer creates a private Python environment and downloads the Low and High models. sh ~/.local/share/openspline/run.sh Open http://localhost:7860 http://localhost:7860 to access the demo. Install the client package where your agent runs: pip install openspline Stream generated audio to a portrait: python from openspline import Openspline client = Openspline async with client.avatar "portrait.jpg", quality="low" as avatar: print avatar.viewer url await avatar.stream agent.audio stream , sample rate=24000 stream accepts generated PCM audio and returns after playback finishes. WAV and MP3 files are also supported. Avatars keep generating fresh idle video while listening and after interruptions, using near-silent audio with very low-level noise for model conditioning. Speech and idle video share the same motion state; idle clips are never looped or replaced with an animated portrait. The noise is not sent to speakers. Generation keeps at most one idle batch ahead of playback, and speech takes priority. The GPU stays active throughout a connected session; smooth continuous motion requires inference to keep up with playback. GPU compatibility: PyTorch uses CUDA 12.8 for Blackwell including RTX 50-series and CUDA 12.6 for older NVIDIA GPUs. Pre-Ampere cards use FP32, which needs more VRAM and may be slower. AMD and Apple GPUs are not supported by this backend. Blackwell and pre-Ampere inference still need hardware validation. These snippets plug into an existing async agent. Openspline owns output playback; disable your previous speaker output to avoid playing audio twice. Provider credentials stay on your backend. Each connector links to a complete runnable example. Install pip install openai==2.54.0 . Wrap your existing OpenAI Realtime connection; tools and transcripts still reach your event handler. python from openspline import Openspline from openspline.integrations import OpenAIRealtime async with Openspline .avatar "portrait.jpg" as avatar: print avatar.viewer url await avatar.wait for viewer adapter = OpenAIRealtime avatar, connection try: async for event in adapter.wrap connection : await handle event event your existing handler finally: adapter.close Run the example https://github.com/openspline/openspline/blob/main/examples/python/openai realtime.py : OPENAI API KEY=... python examples/python/openai realtime.py portrait.jpg . Install pip install 'livekit-agents openai ==1.8.3' . Add the avatar before starting your existing AgentSession ; playback uses your normal LiveKit frontend. python from openspline import Openspline from openspline.integrations.livekit import AvatarSession avatar = AvatarSession "portrait.jpg", client=Openspline quality="low" await avatar.start agent session, room=ctx.room await agent session.start room=ctx.room, agent=agent Run the example https://github.com/openspline/openspline/blob/main/examples/python/livekit agent.py : PORTRAIT=portrait.jpg python examples/python/livekit agent.py dev . Set OPENAI API KEY , LIVEKIT URL , LIVEKIT API KEY , and LIVEKIT API SECRET . The native service includes LiveKit support. Install pip install google-genai==1.75.0 . Forward every audio part from your existing Gemini Live session, including interruption and turn-end events. python from openspline import Openspline from openspline.integrations import GeminiLive async with Openspline .avatar "portrait.jpg" as avatar: print avatar.viewer url await avatar.wait for viewer async for event in GeminiLive avatar .wrap live session.receive : await handle event event your existing handler await avatar.end turn Run the example https://github.com/openspline/openspline/blob/main/examples/python/gemini.py : GEMINI API KEY=... python examples/python/gemini.py portrait.jpg . Repeat the receive loop for subsequent turns; set GEMINI LIVE MODEL to override the model. Connect an existing ElevenLabs agent WebSocket https://elevenlabs.io/docs/eleven-agents/api-reference/eleven-agents/websocket . No additional provider SDK is needed. Pass decoded events, including the initial metadata, so the adapter uses the agent's negotiated audio format. python import json from openspline import Openspline from openspline.integrations import ElevenLabsAgents async with Openspline .avatar "portrait.jpg" as avatar: print avatar.viewer url await avatar.wait for viewer adapter = ElevenLabsAgents avatar async for message in connection: event = json.loads message await adapter.handle event await handle event event keep your input, pong, and tool handlers The adapter supports PCM and μ-law audio, flushes final audio, and handles interruptions. It preserves transcripts and tool events for your existing handler. If attaching after connection setup, pass audio format="pcm 24000" or the negotiated format to the constructor. Node exports ElevenLabsAgents from @openspline/node/integrations , with await adapter.handle event and an optional format as the second constructor argument. Python example https://github.com/openspline/openspline/blob/main/examples/python/elevenlabs agents.py : ELEVENLABS AGENT ID=... python examples/python/elevenlabs agents.py portrait.jpg . Node example https://github.com/openspline/openspline/blob/main/examples/node/elevenlabs agents.mjs : ELEVENLABS AGENT ID=... node examples/node/elevenlabs agents.mjs portrait.jpg . Set ELEVENLABS API KEY for private agents; the examples request a signed URL on the backend. Each plays one response; the demo supports microphone conversations. Configure the agent's client events to include audio , interruption , and agent response complete . Custom client tools and required dynamic variables need your application's handlers and initiation data; the demo uses the agent's saved configuration. Use any TTS provider or agent with an async PCM16 audio iterator. stream buffers audio, flushes the final block, and waits for playback to finish. python from openspline import Openspline client = Openspline quality="low" async with client.avatar "portrait.jpg" as avatar: print avatar.viewer url await avatar.stream agent.audio stream , sample rate=24000 For callbacks, await avatar.send audio chunk, sample rate=24000 , then await avatar.end turn when speech ends. Call await avatar.interrupt on barge-in. Float32 uses encoding="pcm f32le" ; encoded streams use encoding="mp3" . File example https://github.com/openspline/openspline/blob/main/examples/python/universal.py . More connectors: OpenAI Agents https://github.com/openspline/openspline/blob/main/examples/python/openai agents.py , Pipecat https://github.com/openspline/openspline/blob/main/examples/python/pipecat bot.py , Google ADK https://github.com/openspline/openspline/blob/main/examples/python/google adk.py , Vocode https://github.com/openspline/openspline/blob/main/examples/vocode/main.py , and TEN https://github.com/openspline/openspline/blob/main/examples/ten/graph.json . Python examples have locked dependencies https://github.com/openspline/openspline/blob/main/examples/python/requirements.lock ; Vocode uses its own environment https://github.com/openspline/openspline/blob/main/examples/vocode/requirements.lock . Pass settings directly when initializing, or reuse a typed config object: python from openspline import Openspline, OpensplineConfig client = Openspline url="http://localhost:7860", quality="low" config = OpensplineConfig quality="high", timeout=300, viewer timeout=90 client = Openspline config=config a dict works too | Python / Node option | Default | Purpose | |---|---|---| | url | OPENSPLINE URL or http://localhost:7860 | Service address | | quality | low | Default quality: low or high | | timeout | 180 s / 180,000 ms | HTTP and audio request timeout | | viewer timeout / viewerTimeout | 60 s / 60,000 ms | Wait for the playback destination | Explicit Python parameters override the config object; explicit settings override environment defaults. avatar ..., quality="high" overrides that session only. Invalid configuration fails before connecting. JSON Schema https://github.com/openspline/openspline/blob/main/config.schema.json describes Python settings, with /$defs/node and /$defs/server for Node and worker configuration. Only avatar.session should be passed to a browser. Use Node 20+ and install the SDK with one command: npm install @openspline/node js import { Openspline, type OpensplineConfig } from "@openspline/node"; const config: OpensplineConfig = { quality: "low", timeout: 180000 }; const client = new Openspline config ; await using avatar = await client.avatar "portrait.jpg" ; console.log avatar.viewerUrl ; await avatar.stream agent.audioStream , { sample rate: 24000 } ; Node examples https://github.com/openspline/openspline/blob/main/examples/node : run npm ci there, then node universal.mjs portrait.jpg speech.wav . Without await using , call avatar.close in finally . Pass avatar.session from your backend to your frontend. Keep the publisher token on the backend. js import { Avatar } from "@openspline/react"; <Avatar session={session} / For vanilla JS, Vue, or Svelte, call registerAvatarElement from @openspline/browser/element , then set document.querySelector "openspline-avatar" .session = session . Both players handle connection, autoplay permission, and cleanup. React also exports useAvatar . Install the component you need with npm install @openspline/react or npm install @openspline/browser . The built-in demo uses WebSocket playback through the server's HTTP port, including SSH port forwarding. For the same transport in your frontend, pass {...session, transport: "websocket"} ; components otherwise use WebRTC. Choose GPUs directly in the demo: one for Low, or one or more for High. The picker shows GPU names and memory; your selection is applied when you start and saved for restarts by the curl install. You can also set OPENSPLINE GPUS=0,1 when installing. Explicit CUDA VISIBLE DEVICES limits are respected. /readyz reports the active devices and inference ranks. Edit ~/.local/share/openspline/workers.yaml example https://github.com/openspline/openspline/blob/main/workers.yaml . Each worker loads weights once and serves one session at a time; device assignments cannot overlap. workers: - { id: low-0, quality: low, devices: 0 } - { id: low-1, quality: low, devices: 1 } The demo switches quality automatically for a single worker. For SDK sessions or multiple workers, configure a worker with quality: high, devices: 0, 1 to use a GPU pair, then run sh ~/.local/share/openspline/run.sh --prepare before starting. Busy workers return a typed CapacityError ; an unconfigured quality returns ConfigurationError . IDs are relative to CUDA VISIBLE DEVICES when set. For multiple workers, remove OPENSPLINE GPU , OPENSPLINE GPUS , and OPENSPLINE QUALITY overrides from the installed .env and configure devices in workers.yaml . Set OPENSPLINE HOST , OPENSPLINE PORT , OPENSPLINE PUBLIC URL , and provider credentials in the installed .env . Use HTTPS for remote microphone access. WebRTC playback additionally needs reachable UDP ports or TURN configured in ice servers . API reference: http://localhost:7860/docs http://localhost:7860/docs . Health: /readyz ; metrics: /metrics . Development and validation Run uv sync --all-packages , npm ci && npm run build , then uv run pytest && npm test . uv run openspline serve --backend test --dev starts a static-portrait development service without a GPU. Keep optional framework environments separate; from examples/python , use uv pip sync requirements.lock in an activated environment. Release workflows build and optionally publish Python and npm packages. Regenerate the native installer dependency locks with python scripts/lock-gpu.py requires uv . The installer combines requirements/server.txt with the selected cu126.txt or cu128.txt runtime lock. Real low/high inference, GPU streaming, browser/React playback, and native LiveKit have been exercised. Short A100 tests measured about 64 FPS for low quality and 8 FPS for high quality under different load conditions; these are engine rates, not playback latency. Long-duration, multi-GPU, cloud-provider, and TEN runtime validation remain incomplete. MIT https://github.com/openspline/openspline/blob/main/LICENSE . The vendored inference code retains its Apache-2.0 license https://github.com/openspline/openspline/blob/main/packages/server/src/openspline server/ vendor/LICENSE and third-party notices https://github.com/openspline/openspline/blob/main/packages/server/NOTICE ; model weights have separate terms. Openspline builds on open research and infrastructure, with thanks to the teams and contributors behind: - SoulX-FlashHead https://github.com/Soul-AILab/SoulX-FlashHead for the audio-driven talking-head models and inference code that power avatar generation. - LiveKit Agents https://github.com/livekit/agents for the real-time agent framework that Openspline integrates with to bring avatars into voice conversations. - Wan https://github.com/Wan-Video/Wan2.1 for video generation research and VAE components used in the inference backend through SoulX-FlashHead. - LTX-Video https://github.com/Lightricks/LTX-Video for video autoencoder components used in the inference backend through SoulX-FlashHead. Openspline carries this work forward with a self-hosted runtime for interactive avatars: synchronized audio and video streaming, continuous idle motion, interruption handling, GPU worker orchestration, and a shared API across voice providers. Its Python and Node SDKs, agent connectors, and embeddable players make these capabilities available to developers building their own conversational experiences. Pass an optional background file path or bytes when creating the avatar. Images stay still; videos loop silently during speech and idle playback. Upload a PNG portrait with its background removed : compositing uses the original portrait's alpha matte a fixed silhouette, not automatic per-frame segmentation . Opaque portraits retain their original background. The background fills the portrait's aspect ratio with a centered cover crop. async with Openspline .avatar "portrait.png", background="scene.mp4" as avatar: Use any existing audio connector with this avatar. await avatar.play file "speech.wav" js const avatar = await new Openspline .avatar "portrait.png", { background: "scene.jpg", // MP4/WebM or image; path or Uint8Array } ; LiveKit's Python AvatarSession and Node LiveKitAvatarSession , Pipecat's AvatarProcessor , and TEN's extension properties also accept background . OpenAI Realtime/Agents, Gemini Live, Google ADK, ElevenLabs, Vocode, and universal audio adapters use the background configured on their shared avatar session. Browser, React, web component, WebRTC, and native LiveKit viewers receive the already-composited video without extra configuration. The HTTP API accepts an optional background file alongside portrait and quality in POST /v1/sessions . Limits: 25 MiB per background, 16 megapixels, and video up to 60 seconds / 120 fps. Video decoding retains only the current and next frame. Background selection lasts for the session; omit the field to keep the existing behavior.