{"slug": "show-hn-realtime-talking-avatar-running-free-locally-90-fps-on-rtx4090", "title": "Show HN: Realtime talking avatar running free locally (90+FPS on RTX4090)", "summary": "OpenSpline released a self-hosted tool that turns a portrait image and generated audio into synchronized talking-avatar video and speech on a local NVIDIA GPU, reporting 90+ frames per second on an RTX 4090. The Python-first package (pip install openspline) streams PCM audio to an avatar and ships connectors for OpenAI Realtime, with Node support, a built-in viewer, React and web components, and one active session per GPU worker. The backend requires CUDA 12.8 for Blackwell and RTX 50-series cards and CUDA 12.6 for older NVIDIA GPUs, uses FP32 on pre-Ampere cards, and does not support AMD or Apple GPUs.", "body_md": "**Add a live face to any voice AI agent in five lines.** Turn a portrait and generated audio into synchronized video and speech on your own GPU.\n\n[Try Online (no GPU or installation needed)](https://dashboard.openspline.dev/explore)\n\n- **Bring your agent.** Keep your prompts, tools, microphone, and voice provider.\n- **Python first, Node supported.** Stream audio or use a ready-made connector.\n- **Ready to display.** Built-in viewer, React component, and web component.\n- **Self-hosted.** One active session per GPU worker; add GPUs for concurrency.\n\n[Quick start](#quick-start) · [Connectors](#connectors) · [Configuration](#configuration) · [Node SDK](#node-sdk) · [Frontend](#frontend) · [GPU workers](#gpu-workers)\n\nAvatars created with [OpenSpline Cloud](https://github.com/openspline/openspline/blob/main/experiments/openspline_cloud). Click a preview or **Try online** to open the live avatar.\n\n| Nikola Tesla | Meera | Lisa | \n|---|---|---|\n| [Try online](https://dashboard.openspline.dev/nikola-tesla1) | [Try online](https://dashboard.openspline.dev/meera) | [Try online](https://dashboard.openspline.dev/hemant-virtual-testing2) | \n| Acharya | Elon Musk | Ye | \n| [Try online](https://dashboard.openspline.dev/acharya-ji) | [Try online](https://dashboard.openspline.dev/elon) | [Try online](https://dashboard.openspline.dev/ye-west) | \n\nCreate, share, and embed your own avatar at [openspline.dev](https://openspline.dev), or explore the [Cloud app source and embed examples](https://github.com/openspline/openspline/blob/main/experiments/openspline_cloud/README.md#use-in-code).\n\nRun the following command in a linux device/WSL2 with NVIDIA GPU:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/openspline/openspline/main/install.sh | sh\n```\n\nThe installer creates a private Python environment and downloads the Low and High models.\n\n```\nsh ~/.local/share/openspline/run.sh\n```\n\nOpen [http://localhost:7860](http://localhost:7860) to access the demo.\n\nInstall the client package where your agent runs:\n\n```\npip install openspline\n```\n\nStream generated audio to a portrait:\n\n``` python\nfrom openspline import Openspline\nclient = Openspline()\nasync with client.avatar(\"portrait.jpg\", quality=\"low\") as avatar:\n    print(avatar.viewer_url)\n    await avatar.stream(agent.audio_stream(), sample_rate=24000)\n```\n\n`stream()` accepts generated PCM audio and returns after playback finishes. WAV and MP3 files are also supported.\n\nAvatars keep generating fresh idle video while listening and after interruptions, using near-silent audio with very low-level noise for model conditioning. Speech and idle video share the same motion state; idle clips are never looped or replaced with an animated portrait. The noise is not sent to speakers. Generation keeps at most one idle batch ahead of playback, and speech takes priority. The GPU stays active throughout a connected session; smooth continuous motion requires inference to keep up with playback.\n\n**GPU compatibility:** PyTorch uses CUDA 12.8 for Blackwell (including RTX 50-series) and CUDA 12.6 for older NVIDIA GPUs. Pre-Ampere cards use FP32, which needs more VRAM and may be slower. AMD and Apple GPUs are not supported by this backend. Blackwell and pre-Ampere inference still need hardware validation.\n\nThese snippets plug into an existing async agent. Openspline owns output playback; disable your previous speaker output to avoid playing audio twice. Provider credentials stay on your backend. Each connector links to a complete runnable example.\n\nInstall `pip install openai==2.54.0`. Wrap your existing OpenAI Realtime connection; tools and transcripts still reach your event handler.\n\n``` python\nfrom openspline import Openspline\nfrom openspline.integrations import OpenAIRealtime\n\nasync with Openspline().avatar(\"portrait.jpg\") as avatar:\n    print(avatar.viewer_url)\n    await avatar.wait_for_viewer()\n    adapter = OpenAIRealtime(avatar, connection)\n    try:\n        async for event in adapter.wrap(connection):\n            await handle_event(event)  # your existing handler\n    finally:\n        adapter.close()\n```\n\n[Run the example](https://github.com/openspline/openspline/blob/main/examples/python/openai_realtime.py): `OPENAI_API_KEY=... python examples/python/openai_realtime.py portrait.jpg`.\n\nInstall `pip install 'livekit-agents[openai]==1.8.3'`. Add the avatar before starting your existing `AgentSession`; playback uses your normal LiveKit frontend.\n\n``` python\nfrom openspline import Openspline\nfrom openspline.integrations.livekit import AvatarSession\n\navatar = AvatarSession(\"portrait.jpg\", client=Openspline(quality=\"low\"))\nawait avatar.start(agent_session, room=ctx.room)\nawait agent_session.start(room=ctx.room, agent=agent)\n```\n\n[Run the example](https://github.com/openspline/openspline/blob/main/examples/python/livekit_agent.py): `PORTRAIT=portrait.jpg python examples/python/livekit_agent.py dev`. Set `OPENAI_API_KEY`, `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET`. The native service includes LiveKit support.\n\nInstall `pip install google-genai==1.75.0`. Forward every audio part from your existing Gemini Live session, including interruption and turn-end events.\n\n``` python\nfrom openspline import Openspline\nfrom openspline.integrations import GeminiLive\n\nasync with Openspline().avatar(\"portrait.jpg\") as avatar:\n    print(avatar.viewer_url)\n    await avatar.wait_for_viewer()\n    async for event in GeminiLive(avatar).wrap(live_session.receive()):\n        await handle_event(event)  # your existing handler\n    await avatar.end_turn()\n```\n\n[Run the example](https://github.com/openspline/openspline/blob/main/examples/python/gemini.py): `GEMINI_API_KEY=... python examples/python/gemini.py portrait.jpg`. Repeat the receive loop for subsequent turns; set `GEMINI_LIVE_MODEL` to override the model.\n\nConnect an existing [ElevenLabs agent WebSocket](https://elevenlabs.io/docs/eleven-agents/api-reference/eleven-agents/websocket). No additional provider SDK is needed. Pass decoded events, including the initial metadata, so the adapter uses the agent's negotiated audio format.\n\n``` python\nimport json\nfrom openspline import Openspline\nfrom openspline.integrations import ElevenLabsAgents\n\nasync with Openspline().avatar(\"portrait.jpg\") as avatar:\n    print(avatar.viewer_url)\n    await avatar.wait_for_viewer()\n    adapter = ElevenLabsAgents(avatar)\n    async for message in connection:\n        event = json.loads(message)\n        await adapter.handle(event)\n        await handle_event(event)  # keep your input, pong, and tool handlers\n```\n\nThe adapter supports PCM and μ-law audio, flushes final audio, and handles interruptions. It preserves transcripts and tool events for your existing handler. If attaching after connection setup, pass `audio_format=\"pcm_24000\"` (or the negotiated format) to the constructor. Node exports `ElevenLabsAgents` from `@openspline/node/integrations`, with `await adapter.handle(event)` and an optional format as the second constructor argument.\n\n[Python example](https://github.com/openspline/openspline/blob/main/examples/python/elevenlabs_agents.py): `ELEVENLABS_AGENT_ID=... python examples/python/elevenlabs_agents.py portrait.jpg`. [Node example](https://github.com/openspline/openspline/blob/main/examples/node/elevenlabs_agents.mjs): `ELEVENLABS_AGENT_ID=... node examples/node/elevenlabs_agents.mjs portrait.jpg`. Set `ELEVENLABS_API_KEY` for private agents; the examples request a signed URL on the backend. Each plays one response; the demo supports microphone conversations. Configure the agent's client events to include `audio`, `interruption`, and `agent_response_complete`. Custom client tools and required dynamic variables need your application's handlers and initiation data; the demo uses the agent's saved configuration.\n\nUse any TTS provider or agent with an async PCM16 audio iterator. `stream()` buffers audio, flushes the final block, and waits for playback to finish.\n\n``` python\nfrom openspline import Openspline\nclient = Openspline(quality=\"low\")\nasync with client.avatar(\"portrait.jpg\") as avatar:\n    print(avatar.viewer_url)\n    await avatar.stream(agent.audio_stream(), sample_rate=24000)\n```\n\nFor callbacks, await `avatar.send_audio(chunk, sample_rate=24000)`, then `await avatar.end_turn()` when speech ends. Call `await avatar.interrupt()` on barge-in. Float32 uses `encoding=\"pcm_f32le\"`; encoded streams use `encoding=\"mp3\"`. [File example](https://github.com/openspline/openspline/blob/main/examples/python/universal.py).\n\nMore connectors: [OpenAI Agents](https://github.com/openspline/openspline/blob/main/examples/python/openai_agents.py), [Pipecat](https://github.com/openspline/openspline/blob/main/examples/python/pipecat_bot.py), [Google ADK](https://github.com/openspline/openspline/blob/main/examples/python/google_adk.py), [Vocode](https://github.com/openspline/openspline/blob/main/examples/vocode/main.py), and [TEN](https://github.com/openspline/openspline/blob/main/examples/ten/graph.json). Python examples have [locked dependencies](https://github.com/openspline/openspline/blob/main/examples/python/requirements.lock); Vocode uses its [own environment](https://github.com/openspline/openspline/blob/main/examples/vocode/requirements.lock).\n\nPass settings directly when initializing, or reuse a typed config object:\n\n``` python\nfrom openspline import Openspline, OpensplineConfig\n\nclient = Openspline(url=\"http://localhost:7860\", quality=\"low\")\nconfig = OpensplineConfig(quality=\"high\", timeout=300, viewer_timeout=90)\nclient = Openspline(config=config)  # a dict works too\n```\n\n| Python / Node option | Default | Purpose | \n|---|---|---|\n| `url` | `OPENSPLINE_URL` or`http://localhost:7860` | Service address | \n| `quality` | `low` | Default quality: `low` or`high` | \n| `timeout` | 180 s / 180,000 ms | HTTP and audio request timeout | \n| `viewer_timeout` /`viewerTimeout` | 60 s / 60,000 ms | Wait for the playback destination | \n\nExplicit Python parameters override the config object; explicit settings override environment defaults. `avatar(..., quality=\"high\")` overrides that session only. Invalid configuration fails before connecting. [JSON Schema](https://github.com/openspline/openspline/blob/main/config.schema.json) describes Python settings, with `#/$defs/node` and `#/$defs/server` for Node and worker configuration. Only `avatar.session` should be passed to a browser.\n\nUse Node 20+ and install the SDK with one command:\n\n```\nnpm install @openspline/node\njs\nimport { Openspline, type OpensplineConfig } from \"@openspline/node\";\nconst config: OpensplineConfig = { quality: \"low\", timeout: 180000 };\nconst client = new Openspline(config);\nawait using avatar = await client.avatar(\"portrait.jpg\");\nconsole.log(avatar.viewerUrl);\nawait avatar.stream(agent.audioStream(), { sample_rate: 24000 });\n```\n\n[Node examples](https://github.com/openspline/openspline/blob/main/examples/node): run `npm ci` there, then `node universal.mjs portrait.jpg speech.wav`. Without `await using`, call `avatar.close()` in `finally`.\n\nPass `avatar.session` from your backend to your frontend. Keep the publisher token on the backend.\n\n``` js\nimport { Avatar } from \"@openspline/react\";\n\n<Avatar session={session} />\n```\n\nFor vanilla JS, Vue, or Svelte, call `registerAvatarElement()` from `@openspline/browser/element`, then set `document.querySelector(\"openspline-avatar\").session = session`. Both players handle connection, autoplay permission, and cleanup. React also exports `useAvatar()`.\n\nInstall the component you need with `npm install @openspline/react` or `npm install @openspline/browser`.\n\nThe built-in demo uses WebSocket playback through the server's HTTP port, including SSH port forwarding. For the same transport in your frontend, pass `{...session, transport: \"websocket\"}`; components otherwise use WebRTC.\n\nChoose GPUs directly in the demo: one for Low, or one or more for High. The picker shows GPU names and memory; your selection is applied when you start and saved for restarts by the curl install. You can also set `OPENSPLINE_GPUS=0,1` when installing. Explicit `CUDA_VISIBLE_DEVICES` limits are respected. `/readyz` reports the active devices and inference ranks.\n\nEdit `~/.local/share/openspline/workers.yaml` ([example](https://github.com/openspline/openspline/blob/main/workers.yaml)). Each worker loads weights once and serves one session at a time; device assignments cannot overlap.\n\n```\nworkers:\n  - { id: low-0, quality: low, devices: [0] }\n  - { id: low-1, quality: low, devices: [1] }\n```\n\nThe demo switches quality automatically for a single worker. For SDK sessions or multiple workers, configure a worker with `quality: high, devices: [0, 1]` to use a GPU pair, then run `sh ~/.local/share/openspline/run.sh --prepare` before starting. Busy workers return a typed `CapacityError`; an unconfigured quality returns `ConfigurationError`. IDs are relative to `CUDA_VISIBLE_DEVICES` when set. For multiple workers, remove `OPENSPLINE_GPU`, `OPENSPLINE_GPUS`, and `OPENSPLINE_QUALITY` overrides from the installed `.env` and configure devices in `workers.yaml`.\n\nSet `OPENSPLINE_HOST`, `OPENSPLINE_PORT`, `OPENSPLINE_PUBLIC_URL`, and provider credentials in the installed `.env`. Use HTTPS for remote microphone access. WebRTC playback additionally needs reachable UDP ports or TURN configured in `ice_servers`. API reference: **[http://localhost:7860/docs](http://localhost:7860/docs)**. Health: `/readyz`; metrics: `/metrics`.\n\n## Development and validation\n\nRun `uv sync --all-packages`, `npm ci && npm run build`, then `uv run pytest && npm test`. `uv run openspline serve --backend test --dev` starts a static-portrait development service without a GPU. Keep optional framework environments separate; from `examples/python`, use `uv pip sync requirements.lock` in an activated environment. Release workflows build and optionally publish Python and npm packages.\n\nRegenerate the native installer dependency locks with `python scripts/lock-gpu.py` (requires `uv`). The installer combines `requirements/server.txt` with the selected `cu126.txt` or `cu128.txt` runtime lock.\n\nReal low/high inference, GPU streaming, browser/React playback, and native LiveKit have been exercised. Short A100 tests measured about 64 FPS for low quality and 8 FPS for high quality under different load conditions; these are engine rates, not playback latency. Long-duration, multi-GPU, cloud-provider, and TEN runtime validation remain incomplete.\n\n[MIT](https://github.com/openspline/openspline/blob/main/LICENSE). The vendored inference code retains its [Apache-2.0 license](https://github.com/openspline/openspline/blob/main/packages/server/src/openspline_server/_vendor/LICENSE) and [third-party notices](https://github.com/openspline/openspline/blob/main/packages/server/NOTICE); model weights have separate terms.\n\nOpenspline builds on open research and infrastructure, with thanks to the teams and contributors behind:\n\n- [SoulX-FlashHead](https://github.com/Soul-AILab/SoulX-FlashHead) for the audio-driven talking-head models and inference code that power avatar generation.\n- [LiveKit Agents](https://github.com/livekit/agents) for the real-time agent framework that Openspline integrates with to bring avatars into voice conversations.\n- [Wan](https://github.com/Wan-Video/Wan2.1) for video generation research and VAE components used in the inference backend through SoulX-FlashHead.\n- [LTX-Video](https://github.com/Lightricks/LTX-Video) for video autoencoder components used in the inference backend through SoulX-FlashHead.\n\nOpenspline carries this work forward with a self-hosted runtime for interactive avatars: synchronized audio and video streaming, continuous idle motion, interruption handling, GPU worker orchestration, and a shared API across voice providers. Its Python and Node SDKs, agent connectors, and embeddable players make these capabilities available to developers building their own conversational experiences.\n\nPass an optional `background` file path or bytes when creating the avatar. Images stay still; videos loop silently during speech and idle playback. Upload a **PNG portrait with its background removed**: compositing uses the original portrait's alpha matte (a fixed silhouette, not automatic per-frame segmentation). Opaque portraits retain their original background. The background fills the portrait's aspect ratio with a centered cover crop.\n\n```\nasync with Openspline().avatar(\"portrait.png\", background=\"scene.mp4\") as avatar:\n    # Use any existing audio connector with this avatar.\n    await avatar.play_file(\"speech.wav\")\njs\nconst avatar = await new Openspline().avatar(\"portrait.png\", {\n  background: \"scene.jpg\", // MP4/WebM or image; path or Uint8Array\n});\n```\n\nLiveKit's Python `AvatarSession` and Node `LiveKitAvatarSession`, Pipecat's `AvatarProcessor`, and TEN's extension properties also accept `background`. OpenAI Realtime/Agents, Gemini Live, Google ADK, ElevenLabs, Vocode, and universal audio adapters use the background configured on their shared avatar session. Browser, React, web component, WebRTC, and native LiveKit viewers receive the already-composited video without extra configuration.\n\nThe HTTP API accepts an optional `background` file alongside `portrait` and `quality` in `POST /v1/sessions`. Limits: 25 MiB per background, 16 megapixels, and video up to 60 seconds / 120 fps. Video decoding retains only the current and next frame. Background selection lasts for the session; omit the field to keep the existing behavior.", "url": "https://wpnews.pro/news/show-hn-realtime-talking-avatar-running-free-locally-90-fps-on-rtx4090", "canonical_source": "https://github.com/openspline/openspline", "published_at": "2026-10-19 20:32:31+00:00", "updated_at": "2026-10-19 20:46:42.318496+00:00", "lang": "en", "topics": ["ai-products", "ai-tools", "generative-ai", "ai-agents", "ai-infrastructure"], "entities": ["OpenSpline", "OpenAI Realtime", "PyTorch", "NVIDIA", "RTX 4090", "CUDA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-realtime-talking-avatar-running-free-locally-90-fps-on-rtx4090", "markdown": "https://wpnews.pro/news/show-hn-realtime-talking-avatar-running-free-locally-90-fps-on-rtx4090.md", "text": "https://wpnews.pro/news/show-hn-realtime-talking-avatar-running-free-locally-90-fps-on-rtx4090.txt", "jsonld": "https://wpnews.pro/news/show-hn-realtime-talking-avatar-running-free-locally-90-fps-on-rtx4090.jsonld"}}