I Built a Local AI Chrome Extension. The Llama API Was the Easy Part.

A developer built Local Prompt Enhancer, a Chrome Manifest V3 extension that sends draft prompts from ChatGPT's ProseMirror-backed contenteditable composer to a local llama.cpp server on 127.0.0.1:8080 using the OpenAI-compatible /v1/chat/completions endpoint, tested against the google_gemma-4-E2B-it-Q4_K_M.gguf Gemma model. The developer reports that model integration was the easy part, while identifying the real editor, preserving structured text such as code and Windows paths, preventing late responses from overwriting newer drafts, and defining what "tested" means against a changing website were the hard problems. The extension keeps the user in control with Apply, Copy and Cancel actions and never submits the message to ChatGPT on the user's behalf.

I started with an idea that sounded almost trivial: add an “Improve Prompt” button to ChatGPT, send the draft to a local language model, and let the user replace the original with a clearer version. No cloud API subscription. No complicated application framework. Just a Chrome extension and a local llama.cpp server. The model integration worked surprisingly quickly. The difficult part was everything around it: identifying the real editor, preserving structured text, keeping the interface from breaking, preventing late responses from overwriting newer drafts, and deciding what “tested” actually means when software interacts with a changing website. This is the engineering story of Local Prompt Enhancer: what I built, what broke during review, what the tests caught, and what remains to be validated before a public release. The intended workflow was deliberately small: write a prompt, click Improve Prompt, send that text to a locally running model, review the suggestion, and choose whether to apply it. The extension never needs to send the message to ChatGPT on the user’s behalf. That last detail matters. A prompt editor should not become an invisible intermediary with permission to submit conversations. A suggestion panel with Apply, Copy, and Cancel keeps a person in control of the final request. I used a llama.cpp server listening on 127.0.0.1:8080 and tested the API adapter against a local Gemma GGUF model, google gemma-4-E2B-it-Q4 K M.gguf. The extension calls the OpenAI-compatible /v1/chat/completions endpoint; it is separate from the ChatGPT API. My first instinct was to look for a CSS class associated with ChatGPT’s composer. In the inspected page, one wrapper was named ComposerLayoutBody-uBBf1A. That looked promising, but it was only the surrounding layout. The editable element was a ProseMirror-backed contenteditable node, identified more reliably through attributes such as role=”textbox” and data-composer-markdown. Generated class names can change after a frontend update, so treating them as stable extension APIs is a maintenance trap. js const editor = document.querySelector ' data-composer-input contenteditable="true" role="textbox" ' ; The practical lesson: use a selector strategy with fallbacks, check that the editor is connected and visible, and be prepared for it to be replaced during navigation. A MutationObserver can help reattach controls, but it must not create duplicate buttons or expensive mutation loops. I also learned that reading rich text is not the same as reading a textarea’s value. Paragraph breaks, nested lists, fenced code, URLs, Windows paths, and intentional blank lines all require care. A prompt enhancer that silently corrupts code or paths has failed before the language model even runs. The first working architecture used plain JavaScript and Chrome Manifest V3. I intentionally avoided React, Vite, and npm packages for the MVP. That kept installation simple: Chrome could load the project directly as an unpacked extension. The content script handles ChatGPT’s editor and the Shadow DOM interface. The background service worker validates messages and makes the local HTTP request. A separate API module handles model discovery, timeout, response parsing, and error reporting. Settings live in chrome.storage.local. Content script → MV3 service worker → 127.0.0.1:8080 ← improved prompt← llama.cpp The boundary is useful for both security and maintenance. The content script does not need to fetch the model endpoint directly, and the worker does not need to manipulate ChatGPT’s DOM. Host permissions can remain limited to the local server instead of requesting access to arbitrary websites. That approach also made the code easier to test. I could mock the HTTP adapter independently of the editor, and I could test messaging without running an actual language model every time. For the first version, every Llama call included a complete editing instruction and the user’s current draft. No earlier conversation was supplied. The model was told to correct grammar, improve clarity and structure, preserve original meaning and technical details, and return only the rewritten prompt — not an answer to the prompt itself. messages: { role: "system", content: COMPLETE EDITING INSTRUCTION }, { role: "user", content: capturedDraft } This is more than a prompt-engineering preference. It makes each request reproducible and reduces accidental context leakage. The model cannot rely on instructions that happened to be sent in a previous call. The API client also discovers the loaded model through /v1/models rather than assuming that a local GGUF filename is universally valid. It rejects malformed responses and truncated generations and surfaces failures in the interface rather than replacing the draft with unreliable output. One subtle bug appeared in review: trimming the returned text removed leading indentation and trailing blank lines. For prose, that may seem harmless. For a prompt that contains code, it is a correctness defect. The hardened implementation preserves the meaningful whitespace and normalizes line endings. Generating a suggestion is easy to reverse: if the user dislikes it, they can simply ignore the preview. Replacing text inside a rich editor is different. The website maintains its own editor state and undo history; writing directly to innerHTML or textContent can make the visible DOM disagree with that state. The MVP uses a native editing operation, document.execCommand “insertText” , rather than replacing the editor’s HTML. That command is deprecated, so it is a compatibility choice, not a future-proof integration API. The hardened Apply path checks that the same editor is still present, that the draft has not changed, and that the editor can actually receive focus. It then attempts the edit and verifies the resulting text. If replacement cannot be trusted, the extension leaves a Copy fallback and an original-text recovery field. The remaining limitation is important: two checks of visible text do not prove that ProseMirror’s private state and undo stack are healthy. Only live-browser tests — Apply, keep typing, change focus, return to the draft, and use Ctrl+Z — can establish acceptable real-world behavior. This became my clearest design principle: a writing assistant must be conservative about editing someone else’s draft. Local models can take time to respond. In that interval, users may type, navigate, cancel, open another chat, or try again. Without explicit request identities, a late result can appear in the wrong context or a cancelled request can leave the interface permanently busy. The first version already used an AbortController, loading state, and changed-draft protection. Review exposed a cancellation race: Cancel aborted the fetch, but the worker’s active-request slot remained occupied until asynchronous cleanup finished. A new request could receive BUSY even though the user had cancelled the old one. The fix was conceptually small: remove the active slot immediately on cancellation, abort the matching controller, and make completion cleanup conditional on request identity. That way, an old request cannot clear the slot belonging to a newer request. js if active.get key ?.id === requestId { const old = active.get key ; active.delete key ; old.controller.abort ;} An equally important rule is never to apply a suggestion if the draft changed during inference — even if the user later undid those edits and the visible text matches again. The interface records that the draft was touched, not merely whether its final string happens to match a snapshot. Service worker lifecycle rules add another constraint. Long model inference can outlive a browser extension worker’s reliable fetch window. The project therefore uses explicit timeout and channel-error handling, without silently retrying an inference request. That behavior still needs real Chrome validation with slow responses. Using a loopback endpoint is attractive because the extension does not need a third-party inference service. But “local” is a deployment property, not a magic privacy guarantee. A server accidentally bound to a public network interface, a permissive proxy, or verbose server logging can change the risk profile. The extension limits its configured API origin to http://127.0.0.1:8080, http://127.0.0.1:8080, omits credentials from fetch calls, rejects redirects, validates message sources, and renders model output as text rather than executable HTML. It reads the draft when the user requests improvement rather than sending each keystroke for processing. Those decisions are valuable, but they do not eliminate the need for a truthful privacy policy before publication. Users should know that the product requires their own running local server and that its logging and network exposure remain under their control. The first implementation was accompanied by 20 dependency-free tests. An API smoke test also returned a successful rewrite using the local Gemma model. After reviewing the code, I expanded the suite to 45 tests, adding 25 regression cases covering nested lists, whitespace, focus changes, cancellation races, stale results, composer recreation, shared limits, and timeout recovery. The tests mattered because they captured the defects that ordinary happy-path demos conceal. They also provided a clean way to verify small fixes without accidentally breaking the model adapter or request lifecycle. But those were mostly Node-based tests using controlled DOM and Chrome API doubles. A fake DOM cannot tell me whether the actual ChatGPT Send button is visually overlapped, whether native undo behaves correctly, or whether the real ProseMirror editor reconciles a replacement after several seconds. At the time of this write-up, live Chrome acceptance testing was still outstanding. Claiming the extension was production-ready because the automated suite passed would have confused test coverage with evidence of user-facing reliability. The proper acceptance sequence is straightforward: load the unpacked extension, verify model connectivity, generate a preview, Apply a single paragraph, test multiline code, continue typing, undo, cancel and restart, navigate between chats, and inspect both the ChatGPT console and the extension service worker. A basic Retry starts a new independent rewrite of the current draft. It may produce different wording because text generation is not necessarily deterministic, but it does not learn from its previous answer. Calling that refinement would be misleading. The next design iteration introduces a more deliberate Refine Further flow. The planned request includes only two text artifacts: the original user draft and the immediately preceding successful improvement. The editing instruction asks the model to improve the previous attempt while using the original as the source of truth. First pass: original → version 1Refine further: original + version 1 → version 2Refine again: original + version 2 → version 3 A small temperature increase, such as from 0.2 to 0.3, can permit different wording, but it cannot guarantee a better response. A good refinement instruction must allow the model to keep an already strong version mostly unchanged. Otherwise repeated refinement can drift away from what the user actually asked. This is a design direction described during the project, not a claim that every part of the new refinement flow has already passed end-to-end browser testing. Once the core extension was in place, the conversation shifted toward a Chrome Web Store release: recognizable PNG icons, a toolbar popup, a model connectivity indicator, onboarding, a privacy policy, store listing assets, and a clean ZIP package. These are not decorative extras. A product that depends on a separately running llama.cpp server needs to explain that dependency before users install it. A toolbar status display can make a disconnected server understandable rather than mysterious. A narrowly scoped permissions list and accurate store description build trust. The release preparation work was planned around a lightweight v0.2.0 candidate, while retaining the dependency-free extension architecture. Preparing a submission package is not the same thing as a confirmed Chrome Web Store publication or approval. · Design the “never automatically submit” rule before writing any UI code. It simplifies the entire trust model. · Write down the editor’s real DOM structure early, but treat selectors as hypotheses that need ongoing browser checks. · Make text preservation a first-class test requirement, especially for code, paths, lists, and blank lines. · Use unique IDs for every asynchronous request from the beginning; cancellation and navigation become much easier to reason about. · Separate API success, mocked browser behavior, and actual browser acceptance in every test report. · Keep a Copy fallback whenever integration with a third-party rich-text editor is uncertain. · Do not add frameworks before the interface actually needs them. Plain JavaScript was sufficient for this stage. This project began as a tiny local-AI feature and became a useful example of a broader engineering truth: the model call is often the least complicated part of an AI product. The real work is defining boundaries. Which element owns the user’s text? When is it safe to replace that text? What happens if the page changes while inference is running? How do you preserve intent and formatting? What counts as evidence that the software works? For Local Prompt Enhancer, the answer is not a larger model or a more elaborate frontend. It is a restrained workflow, explicit state management, testable modules, and a clear distinction between what has been implemented and what has been validated. That is the lesson I will carry into the next iteration — and into other applications that connect local AI models to existing software. Chrome Extensions — Manifest V3 documentation: https://developer.chrome.com/docs/extensions/ https://developer.chrome.com/docs/extensions/ llama.cpp — Server documentation: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md Medium — Writing and publishing your first story: https://help.medium.com/hc/en-us/articles/225168768-Writing-and-publishing-your-first-story https://help.medium.com/hc/en-us/articles/225168768-Writing-and-publishing-your-first-story Medium — Distribution guidelines: https://help.medium.com/hc/en-us/articles/360006362473-Medium-s-Distribution-Guidelines-How-curators-review-stories-for-Boost-General-and-Network-Distribution https://help.medium.com/hc/en-us/articles/360006362473-Medium-s-Distribution-Guidelines-How-curators-review-stories-for-Boost-General-and-Network-Distribution I Built a Local AI Chrome Extension. The Llama API Was the Easy Part. https://pub.towardsai.net/i-built-a-local-ai-chrome-extension-the-llama-api-was-the-easy-part-59f7b2dbf4bc was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.