{"slug": "speech-to-markdown-with-local-models", "title": "Speech to Markdown with Local Models", "summary": "Voice-to-Md.dev released Speech-to-Markdown, a 100% local app for macOS and iOS that converts voice into structured markdown documents using whisper.cpp and a local LLM, with no cloud dependency. The app offers global dictation and an agent mode with format, edit, and append capabilities.", "body_md": "**Talk. Get clean markdown. 100% local. 🔗 voice-to-md.dev**\n\nSpeech-to-Markdown turns your voice into structured documents — and it ships as **two apps**:\n\n🖥️ Desktop — macOS |\n📱 Mobile — iOS |\n|\n|---|---|---|\n| Platform | macOS 13+ (menu-bar app) | iOS 26+, iPhone 15 Pro or newer (Apple Intelligence required; Apple-Silicon iPads too) |\n| Speech-to-text |\n|\n\n`whisper-cpp`\n\n, `ffmpeg`\n\n, a local LLM server**none**— zero setup** 100% offline**— nothing leaves your phoneJump to: [🖥️ Desktop app](#%EF%B8%8F-desktop-app-macos) · [📱 Mobile app](#-mobile-app-ios)\n\nA menu-bar app: voice → whisper.cpp → your local LLM → clean markdown. No cloud. No API keys. Nothing leaves your Mac. Everything in this part — including the dependencies and LLM-server setup below — applies to the **Mac app only**; the mobile app needs none of it.\n\n*↑ This entire product spec was dictated by voice — a local LLM structured it in real time.*\n\n-\n**⌨️ Global Dictation —** Speak, and the transcript is typed straight into whatever field has focus. Terminal, browser, Slack — anything. A Spotlight-style pill shows what's happening:`⌘⌥]`\n\nanywhere. -\n**📝 Agent Mode — live markdown editor.** Speak freely; a local LLM streams your words into a clean, structured markdown document in real time. Raw transcript stays one click away. Start it from the menu bar:\n\nEverything in Agent Mode is driven from one floating, draggable capsule that hovers over the editor:\n\nLeft to right:\n\n— don't wait for the auto-flush: transcribe whatever you just said and send it to the LLM✈️ Send (`⌘↩`\n\n)*right now*. Enabled while recording and idle (no LLM call in flight).**🎙️ Mic**— start, pause, and resume the session. Red pulse = listening.** Status**—`Recording`\n\n/`Processing`\n\n/`Paused`\n\nat a glance, with errors surfaced inline.**Mode switcher** and**format dropdown**— how your voice is applied (see below) and what comes out:** MD**(default),** TXT**, or** HTML**. Each format ships its own expectations + example to the LLM, works in every mode, and the session file extension follows (`.md`\n\n/`.txt`\n\n/`.html`\n\n— switching mid-session renames the file). Your choice is remembered across launches.**👁️ Preview**— opens the session document in the default app for its type (browser for HTML, your editor for markdown/plain text).**🗑️ Trash**— clears the session (audio, transcript, and document) after a confirmation.\n\nThe modes:\n\n| Mode | Icon | What it does |\n|---|---|---|\nFormat (default) |\n📄 | The whole document + your new words go to the LLM, which returns the complete restructured document. Best for free-form dictation where the model keeps improving the overall structure. |\nEdit |\n✏️ | Your voice is an instruction, not content: \"change the subtitle to Weekly Notes\", \"turn that list into a table\". Select text in the editor first and the model treats it as the focus of the edit. |\nAppend |\n➕ | Speed mode for long documents: only the last 3 sentences + your new words are sent, and the model returns just the new content, which is appended. The LLM never re-reads the whole file, so latency stays flat as the document grows. |\n\nThe flow underneath: audio is transcribed by whisper.cpp in ~4 s chunks and buffered; once ~30 words accumulate (or you pause for 5 s, or hit **Send**), the buffer is flushed to the LLM using the prompt for the current mode and output format — and tokens stream straight into the editor.\n\nOne line — installs dependencies, builds from source, and drops the app in /Applications:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/xajik/voice-to-md/main/install.sh | bash\n```\n\nOr from a checkout:\n\n```\ngit clone https://github.com/xajik/voice-to-md.git && cd voice-to-md\nmake install       # deps + build + copy to /Applications\n# …or for development: make setup && make run\n```\n\nFirst launch: grant **Microphone** + **Accessibility** access, then download a Whisper model from **Settings…** in the menu bar (Base is a great start).\n\n**Signing:** builds are automatically signed with your \"Apple Development\" identity when one is in the keychain, so re-installs keep their Microphone/Accessibility grants. No identity → ad-hoc fallback (macOS will re-ask for permissions after updates). Override with `make install SIGN_IDENTITY=\"…\"`\n\n.\n\nAgent Mode talks to any **OpenAI-compatible** server. Point Speech-to-Markdown at it in **Settings…** (default: `http://127.0.0.1:8000/v1`\n\n, model auto-picked). The Whisper STT model is picked there too:\n\nPick your server:\n\nServe any MLX model on port 8000 — Speech-to-Markdown's default, zero config needed:\n\n```\nbrew install omlx\nomlx serve Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-4bit\nbrew install ollama\nollama pull qwen3.5        # or: ollama pull gemma3\nollama serve\n```\n\nBase URL: `http://127.0.0.1:11434/v1`\n\nDownload from [lmstudio.ai](https://lmstudio.ai), grab a model, start the local server.\nBase URL: `http://127.0.0.1:1234/v1`\n\n```\nbrew install llama.cpp\nllama-server -m your-model.gguf --port 8080\n```\n\nBase URL: `http://127.0.0.1:8080/v1`\n\n| Model | Why |\n|---|---|\nQwen3.5 27B (4-bit) |\nBest formatting quality; the default pick |\nGemma 26B (8-bit) |\nFast, great instruction following |\n\nAnything that follows instructions well works — Speech-to-Markdown streams tokens as they arrive, so even bigger models *feel* instant.\n\nAgent Mode on your phone — **everything runs offline, on the device**. No LLM server, no whisper install, no settings, no network. None of the desktop dependencies above apply here.\n\nRequirements:iOS 26 or lateron anApple Intelligence device — iPhone 15 Pro or newer(or an Apple-Silicon iPad). Apple Intelligence must be enabled in Settings; the app shows an actionable banner if it isn't.\n\n**STT**: Apple's SpeechAnalyzer streams your speech to text live, fully on-device (volatile text shows in gray as you talk; finalized words feed the document).**LLM**: Apple** Foundation Models**— the on-device Apple Intelligence model — formats the transcript. No server, no API keys, works in airplane mode.- Same three modes (\n**Format / Edit / Append**), same** MD / TXT / HTML**output formats, sessions restorable from history, documents visible in the Files app.\n\nBuilding from source: `make build-ios`\n\n/ `make test-ios`\n\n(see [Development](#%EF%B8%8F-development)). Running on a device needs a development team: `xcodebuild -scheme SpeechToMarkdownIOS -allowProvisioningUpdates CODE_SIGN_STYLE=Automatic DEVELOPMENT_TEAM=<TEAMID> …`\n\n(and Developer Mode enabled on the phone).\n\n```\n# macOS app\nmake check      # verify dependencies\nmake build      # release build + sign\nmake test       # unit tests\nmake generate   # regen .xcodeproj after editing project.yml\nmake install    # build + install to /Applications\nmake uninstall  # remove from /Applications\n\n# iOS app\nmake build-ios  # build for the iOS simulator\nmake test-ios   # run the iOS test suite\n```\n\nBoth apps share one core (`Shared/`\n\n): transcript buffering, prompts, formats, session files. Desktop pipeline: AVAudioEngine → whisper.cpp → local LLM server. Mobile pipeline: AVAudioEngine → SpeechAnalyzer → Foundation Models. Either way, everything stays on your hardware.", "url": "https://wpnews.pro/news/speech-to-markdown-with-local-models", "canonical_source": "https://github.com/xajik/voice-to-md", "published_at": "2026-07-10 10:25:36+00:00", "updated_at": "2026-07-10 10:35:15.149500+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools", "natural-language-processing"], "entities": ["voice-to-md.dev", "whisper.cpp", "ffmpeg", "Apple", "macOS", "iOS"], "alternates": {"html": "https://wpnews.pro/news/speech-to-markdown-with-local-models", "markdown": "https://wpnews.pro/news/speech-to-markdown-with-local-models.md", "text": "https://wpnews.pro/news/speech-to-markdown-with-local-models.txt", "jsonld": "https://wpnews.pro/news/speech-to-markdown-with-local-models.jsonld"}}