{"slug": "embeddinggemma-2-transcribe-with-cuda", "title": "Embeddinggemma-2 Transcribe with CUDA", "summary": "An independent developer released an experimental Linux port of EG2 Transcribe 1.6, called Embeddinggemma-2 Transcribe with CUDA, that runs the upstream EmbeddingGemma-2 speech model weights in PyTorch on CPU or NVIDIA CUDA without macOS or Core ML. The port requires Linux, Python 3.13, uv, ffmpeg and curl, downloads roughly 715 MB of verified model files, and outputs text, JSON word timestamps on a 40 ms decoder frame grid, or SRT subtitles, with an optional persistent server on 127.0.0.1:8000 accepting uploads up to 1 GiB and audio up to four hours. The release matters because it moves the transcription model off Apple's runtime onto commodity Linux and NVIDIA hardware, though the server ships with no authentication or TLS and an unavailable CUDA device fails explicitly rather than falling back to CPU.", "body_md": "Speech transcription on Linux using EG2 Transcribe's model weights and PyTorch. Supports CPU and NVIDIA CUDA, with punctuation, capitalization, word timestamps, and text, JSON or SRT output. No macOS or Core ML runtime is required.\n\nThis is an independent, experimental port of EG2 Transcribe 1.6. It uses the upstream weights without retraining.\n\nRequires Linux, Python 3.13, [uv](https://docs.astral.sh/uv/), ffmpeg and curl.\nCUDA also requires a compatible NVIDIA GPU and driver.\n\nFrom this directory, create a virtual environment:\n\n```\nuv venv --python 3.13\n```\n\nInstall the CPU packages:\n\n```\nuv pip install --python .venv/bin/python numpy torch --torch-backend cpu\n```\n\nFor CUDA, use this instead. It also replaces an existing CPU-only PyTorch install:\n\n```\nuv pip install --python .venv/bin/python --upgrade numpy torch --torch-backend auto\n```\n\nDownload the models:\n\n```\n.venv/bin/python download.py\n```\n\nThe download is approximately 715 MB. The downloader verifies file sizes and\nSHA-256 hashes, skips matching files, and leaves failed downloads as `.part` files.\nSkip installation and downloading if the environment and models are already ready.\n\nTranscribe on CPU, the default device:\n\n```\n.venv/bin/python transcribe.py recording.wav\n```\n\nUse CUDA:\n\n```\n.venv/bin/python transcribe.py recording.wav --device cuda\n```\n\nSave word timestamps or subtitles:\n\n```\n.venv/bin/python transcribe.py recording.wav --format json > transcript.json\n.venv/bin/python transcribe.py recording.wav --format srt > subtitles.srt\n```\n\nThe CLI accepts audio formats ffmpeg can decode. Transcripts go to stdout;\nprocessing timings go to stderr. Add `--device cuda` to either output command\nfor GPU inference. An unavailable CUDA device fails explicitly, without a CPU fallback.\n\nJSON contains the transcript and individual word records:\n\n```\n{\n  \"text\": \"Hello.\",\n  \"words\": [\n    {\"text\": \"Hello.\", \"start\": 0.4, \"end\": 0.44}\n  ]\n}\n```\n\nTimes are in seconds on a 40 ms decoder frame grid. They describe token runs, not complete spoken-word boundaries or forced alignment. SRT groups words into cues of at most 12 words, splitting when adding a word would exceed six seconds or the gap exceeds 0.8 seconds.\n\nStart a persistent server:\n\n```\n.venv/bin/python server.py --device cpu\n```\n\nUse `--device cuda` for GPU inference. Startup loads one transcriber and warms\nall four graphs before printing `Ready`. The same instance handles subsequent\nrequests, one at a time. No additional Python dependencies are needed.\n\nSend the audio file as the raw request body, not multipart form data:\n\n```\ncurl --fail-with-body \\\n  -H 'Content-Type: application/octet-stream' \\\n  --data-binary @recording.m4a \\\n  'http://127.0.0.1:8000/transcribe?format=json'\n```\n\n`POST /transcribe` returns JSON by default. Use `?format=text` or `?format=srt`\nfor the other outputs. Errors return JSON with an appropriate HTTP status.\nM4A/MP4 uploads use a temporary seekable cache, so they do not require conversion.\nUploaded media cannot open external files or network URLs through ffmpeg.\n\n| Setting | Default or limit | \n|---|---|\n| Bind address | `127.0.0.1:8000` , configurable with`--host` and`--port` | \n| Upload size | 1 GiB | \n| Audio duration | Four hours | \n| Stalled socket read | 30 seconds | \n| Audio decoding | 60 seconds | \n| Request framing | `Content-Length` required; chunked uploads unsupported | \n\nThe server has no authentication or TLS. Keep it private or put it behind an authenticated reverse proxy before exposing it. Restart it after changing code.\n\n- CPU caches retain up to 64 MiB of constants per graph, 256 MiB total. Uncached weights are reloaded during inference; keeping the server running does not retain every expanded weight in RAM.\n- CUDA retains expanded weights in VRAM. Allow space for weights, activations and workspaces. Decoding, log-mel preprocessing, weight unpacking and embedding lookups remain on CPU; graph arithmetic and candidate reranking run on GPU.\n- Graph floating-point arithmetic uses FP32, including internal FP16 casts, to avoid overflow. Numerical parity between CPU, CUDA and Core ML is not guaranteed.\n- The decoded recording is held in RAM. HTTP uploads also use temporary disk space up to the uploaded file's size.\n- Recognition can omit or misidentify words. Punctuation and capitalization do not correct recognition errors. Speed and accuracy depend on hardware and audio.\n\nPass a finite, one-dimensional NumPy array of mono samples at 16 kHz:\n\n``` python\nfrom transcribe import Transcriber, decode_audio\n\ntranscriber = Transcriber(device=\"cuda\")  # Use \"cpu\" for CPU inference.\nsamples = decode_audio(\"recording.wav\")\ntext = transcriber.transcribe(samples)\n# Alternatively, get formatted word records with start/end times:\nwords = transcriber.transcribe_words(samples)\n```\n\nReuse the transcriber across recordings. Each transcription method runs inference; call only the method needed for your output.\n\n- `download.py` : verified model downloads.\n- `mil.py` : PyTorch execution of the shipped MIL graphs and packed weights.\n- `transcribe.py` : audio decoding, inference, word decoding and output formatting.\n- `server.py` : persistent, sequential HTTP API.\n\nRun the built-in MIL format self-check:\n\n```\n.venv/bin/python mil.py\n```\n\nOriginal project: [https://eg2.haku.sh/](https://eg2.haku.sh/)\n\nModel manifest: [https://haku-models.tslfiles.org/eg2demo/v5/manifest.json](https://haku-models.tslfiles.org/eg2demo/v5/manifest.json)\n\nModel files are governed by their upstream terms. See the downloaded\n`models/NOTICE` and preserve upstream notices when sharing model files.\nThis port does not change their licensing.", "url": "https://wpnews.pro/news/embeddinggemma-2-transcribe-with-cuda", "canonical_source": "https://github.com/skorotkiewicz/eg2-linux", "published_at": "2026-10-17 12:13:12+00:00", "updated_at": "2026-10-17 12:32:57.519402+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-tools", "natural-language-processing", "developer-tools"], "entities": ["Embeddinggemma-2 Transcribe with CUDA", "EG2 Transcribe 1.6", "PyTorch", "NVIDIA CUDA", "Linux", "Python 3.13", "ffmpeg", "Core ML"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/embeddinggemma-2-transcribe-with-cuda", "markdown": "https://wpnews.pro/news/embeddinggemma-2-transcribe-with-cuda.md", "text": "https://wpnews.pro/news/embeddinggemma-2-transcribe-with-cuda.txt", "jsonld": "https://wpnews.pro/news/embeddinggemma-2-transcribe-with-cuda.jsonld"}}