Language integrated LLMs as an OCaml function

Developer Anil Madhavapeddy released ocaml-deepseek, an OCaml library that integrates DeepSeek's open-weight LLM directly into applications via a native inference engine, enabling local, dependency-free AI agents. The package, submitted to the opam package manager, allows OCaml functions to be used as tools for the model, demonstrated with a web traffic analysis agent.

Fable cut out on me https://www.theverge.com/ai-artificial-intelligence/949553/anthropic-fable-5-mythos-5-government-national-security at 1am on Saturday while I was sweeping over the OCaml runtime looking for concurrency bugs. There have been excellent takes https://x.com/rosstaylor90/status/2066067747738431504 on the sovereignity implications of this, and I figured I'd roll my sleeves up and get serious about using the open weights models. DeepSeek's models have been getting more capable since their first release /notes/deepseek-r1-advances , and v4 Flash https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash is small enough to run on my Mac admittedly, very high-end Macs with 128GB/512GB of RAM respectively for my laptop and desktop . The question is whether the agentic CLIs I've been using /notes/aoah-2025 can be easily replaced. The best way to learn how a system works is to build it unikernel style /projects/unikernels , and so I aimed to expose the LLM as a normal OCaml library. This avoids routing via bloated CLIs https://github.com/anthropics/claude-code/issues/8382 , and lets the linking application drive the agentic loop according to its specific needs. What makes this practical is Antirez https://github.com/antirez ' Dwarfstar https://github.com/antirez/ds4 , a self-contained native inference engine that supports Apple Metal https://developer.apple.com/metal/ and portable ish C. I bound this directly to OCaml 5 and Eio https://github.com/ocaml-multicore/eio as ocaml-deepseek , and now a plain function call on my laptop gets me an LLM in my application. For example, I can now embed Deepseek inference directly into the OCaml webserver that drives this very site in order to look for suspicious bot activity, and because it's open weights and running locally, there's no dependency on external services A traffic-triage agent in-process in OCaml. The agent is handed two OCaml function tools and works out for itself how to combine them. let agent = Agent.create engine ~system:"You are a web-traffic analyst." ~tools: Toolbox.read ~dir:logs; read-only sandboxed handle to the logs dir query db ~conn; a SELECT-only tool over the local database in Agent.send agent ~on event "Cross-reference today's 404 spikes in the access log against the \ client IPs in the requests table. Anything coordinated indicating a bad bot?" The log reader and the database query are just two OCaml functions the model is allowed to call, each scoped and sandboxed using Eio to exactly what it needs. The model decides when and how to combine them. 1 trying-out-humpty-the-ocaml-agent Trying out Humpty the OCaml agent I've submitted https://github.com/ocaml/opam-repository/pull/30053 the package to opam, so opam install deepseek or opam pin add deepseek https://tangled.org/anil.recoil.org/ocaml-deepseek.git should work. The package also ships a binary called humpty 1 with two variants: humpty-metal for Apple Silicon and a portable humpty-cpu that should run anywhere slowly .There are four subcommands that we'll use to explain how to build an agent up in OCaml: first list choose-the-right-deepseek-model the available models and one, then grab-the-deepseek-model-weights download with it statelessly, and then wrap that into an an-llm-is-a-stateless-request-reply-function chat . adding-state-to-make-an-agentic-ocaml-library agent 1.1 choose-the-right-deepseek-model Choose the right Deepseek model Before we can get started you'll first need the open model weights https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash downloaded. humpty list https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/bin/humpty.ml L101-134 prints a the catalogue of available weights: bash $ humpty-metal list Models download dir: /Users/avsm/.local/share/ds4 TARGET ALIASES DESCRIPTION q2-imatrix q2 2-bit Flash routed experts ~81 GB ; for 96-128 GB RAM. q2-q4-imatrix q2q4 Mixed Flash quant ~98 GB ; higher quality for 128 GB. q4-imatrix q4 4-bit Flash routed experts ~153 GB ; for 256 GB+ RAM. pro-q2-imatrix pro-q2 PRO q2 single file ~430 GB ; for 512 GB RAM. = present, = not downloaded Pick one based on how much RAM you have; I use q2q4 on my laptop with 128GB RAM , and the extremely beefy pro-q2 on my Mac Studio with 512GB RAM . There are also split files for running the model distributed across several machines, which I'll skip here for now. 1.2 grab-the-deepseek-model-weights Grab the Deepseek model weights Once you've chosen, humpty download q4 https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/bin/humpty.ml L66-97 or pro-q2 , or whichever shells out to the Hugging Face CLI to fetch the GGUF. You'll either need the Huggingface CLI installed https://huggingface.co/docs/huggingface hub/en/guides/cli or have uvx https://github.com/astral-sh/uv in your path. Once this gets doing go have a cup of tea while the gigabytes of LLM weights download, and then we'll start to build an agent from the camel up 2 building-an-agent-from-the-ground-up Building an agent from the ground up I first want to pin down what an "agent" actually means, as the term seems to have accreted much mystique this year. The whole OCaml Deepseek stack is a small library you can read through quickly, so let me build an agent up from scratch. The code below links to the Tangled source https://tangled.org/anil.recoil.org/ocaml-deepseek/commit/1edcdad29a19924f988c7adef4343cb189dcb4a2 . 2.1 an-llm-is-a-stateless-request-reply-function An LLM is a stateless request-reply function A basic LLM takes in a text prompt, performs inference on some weights, and generates a text reply back. To illustrate this in our OCaml code, we need to load the model weights and spin up an engine with a cache directory for the compiled Metal kernels if using the Apple GPU version : js let engine = Deepseek.V4.create ~cache ~model ~domain mgr ~sw in V4.generate engine "Explain monads in one sentence." ~on token:print string; - : unit = Monads are a design pattern that allows you to chain operations together while automatically handling extra behavior like error handling, state, or side effects, by wrapping values in a context and providing a way to transform and combine them. Deepseek.V4.create https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/lib/v4.mli L41-47 opens the GGUF https://huggingface.co/docs/hub/gguf model file and, the first time it runs, materialises the embedded Metal shaders in the cache . Generating a reply is then a single call to V4.generate that encodes the supplied prompt, runs a prefill, and samples one token at a time into the on token callback until the end-of-sequence marker. All the inference is done in the Metal library in a separate OCaml domain, so we can continue to use other Eio fibres in our main application.You can try this single-shot request/response using humpty chat https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/bin/humpty.ml L44-62 , which keeps no memory between runs and can't take any action beyond showing the reply. bash $ humpty-metal chat 'Explain algebraic effects in OCaml 5 in one sentence' Algebraic effects in OCaml 5 allow functions to suspend execution and invoke user-defined handlers for operations like state, exceptions, or generators via a lightweight, type-safe mechanism that integrates with the language's existing type system and is used primarily for effectful programming, such as with the new Effect library for handling delimited continuations. A "conversation" is therefore just a list of role-tagged messages e.g. system, user, assistant, tool that we concatenate in our library into a prompt string for the LLM. 3 how-the-stateless-llm-asks-for-effects-to-the-external-world How the stateless LLM asks for effects to the external world The single-step text-to-text function from earlier emits text in an agreed "shape" so we can figure out what to do next based on its output. DeepSeek has trained their model to understand a little markup language called DSML https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/encoding dsv4.py , which looks something like this: <｜DSML｜tool calls <｜DSML｜invoke name="edit" <｜DSML｜parameter name="path" string="true" /tmp/x.c</｜DSML｜parameter <｜DSML｜parameter name="line" string="false" 42</｜DSML｜parameter </｜DSML｜invoke </｜DSML｜tool calls DSML is a pseudo-XML language that's interspersed in the text responses from the agent. Those bars are actually the full-width vertical line ｜ U+FF5C and not an ASCII | . DeepSeek reserves the rarer codepoint for DSML's control tokens, so they can't be produced by ordinary text or code the model emits. We've got a DSML implementation in OCaml https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/main/dsml/lib/dsml.mli , which parses the XML into an OCaml record type https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/dsml/lib/dsml.mli L83-98 : type thinking mode = Chat | Thinking type reasoning effort = High | Max type task = Action | Query | Authority | Domain | Title | Read url type tool call = { id : string option; name : string; arguments : string } type parsed message = { content : string; reasoning content : string; tool calls : tool call list; } Parsing a raw text reply splits it into the visible content the user will see, the model's hidden reasoning content https://en.wikipedia.org/wiki/Reasoning model , and any tool calls it emitted along the way. A tool call is just a string name plus its arguments as a JSON string, with an optional identifier to pair the result back up. The other three types are knobs on how the model replies rather than what it returns: - a thinking mode of Chat answers directly while Thinking reasons in a <think block first reasoning effort turns that reasoning up to maximum at the cost of slower inference time task is a quick-instruction hint into DeepSeek's internal pipeline as to whether this turn is an Action , a Query , etc. 3.1 making-custom-tool-functions-in-ocaml Making custom tool functions in OCaml We now need to define specific tools that the model knows about. We do this by defining a bidirectional codec that decodes the model's JSON/DSML arguments into a typed OCaml value, and renders that value back. Here's a simple touch https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/lib/tool.mli L11-20 tool: php let touch = let open Dsml.Codec in Invoke.map "touch" fun path - path | Invoke.param ~enc:Fun.id "path" string ~description:"file to create" | Invoke.seal in Tool.v ~description:"Create an empty file." touch fun path - Out channel.with open text path ignore; "created " ^ path We just wrap the effect we are doing in this case, just writing an empty file with the JSON metadata to let the model know when and how to invoke the tool as it works its way through the prompt. Unlike most policy languages, we describe these in plain text since we're dealing with an LLM, and it decides when the description of the tool should be applied in the conversation. There's still no state here, but we can now use the DSML library to build up a full prompt string by keeping track of all the messages: php val encode messages : ?context:message list - ?drop thinking:bool - ?add default bos token:bool - ?reasoning effort:reasoning effort - thinking mode - message list - string encode messages mode messages renders the conversation to the prompt string. context prepends an already-encoded prefix and suppresses the leading token; drop thinking default true drops reasoning from turns before the last user message; reasoning effort Max maximises reasoning. 4 adding-state-to-make-an-agentic-ocaml-library Adding state to make an agentic OCaml library We're now ready to add state to this An "agent" is just a wrapper around the LLM that does three things: - remember the conversation so far for encode messages - keep the engine's KV-cache warm via a persistent session https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/lib/v4.mli L95-108 - run the tool callbacks as they arrive from the LLM The loop is expressed as a simple OCaml event datatype https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/lib/agent.mli L27-33 : type event = | Reasoning of string the model's <think text | Content of string a chunk of the reply | Tool call of Dsml.tool call | Tool result of string string | Done val send : t - on event: event - unit - string - unit When the LLM responds each turn, plain text means the turn is Done . Tool calls get looked up by their name, then run, and the results folded back into the conversation as tool messages. All the agent function does is just run this to a fixed point until the model answers with text alone. 4.1 writing-custom-tools-using-os-sandboxing-and-eio-capabilities Writing custom tools using OS sandboxing and Eio capabilities Here's where the unikernel-style magic shows up though. Given that a tool is just something we define ourselves, then we can start to take advantage of OCaml itself And in particular, I want better security and more fine-grained tool calls that are tailored to my applications and not generic shell scripts that are difficult to sandbox. Eio is a library /papers/2023-ocaml-eio built over OCaml 5's effects /papers/2021-pldi-retroeff that follows an object-capability https://en.wikipedia.org/wiki/Object-capability model discipline to eliminate ambient authority where possible. In our Toolbox https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/lib/toolbox.mli L4-22 module, we define some example Eio tools: php val read : dir: Eio.Path.t - Tool.t val write : dir: Eio.Path.t - Tool.t val dns : net: Eio.Net.t - Tool.t val bash : proc: Eio.Process.mgr - Tool.t Notice that each OCaml signature here takes in a capability for that particular tool: read and write can only access the directory you pass as ~dir and nothing above it, since Eio usesto sandbox the call. openat 2 https://linux.die.net/man/2/openat dns reaches the network only because it holds a net capability bash spawns processes only because it holds a process manager https://github.com/ocaml-multicore/eio/blob/main/README.md running-processes . When using these from applications, we can select the exact sandboxing we need, or just write our own tool functions with application-specific logic. This is exactly what humpty agent https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/bin/humpty.ml L154-201 does, confining the file and shell tools to a workspace directory you pass with --dir : Sandbox the filesystem tools to workspace so that tools only have access there Eio.Path.with open dir Eio.Path. fs / workspace @@ fun ws - let agent = Agent.create engine ~system ~thinking:think ~tools: Toolbox.read ~dir:ws; Toolbox.write ~dir:ws; Toolbox.dns ~net; Toolbox.bash ~proc; in ... The fs , net and proc values all come from Eio's stanard environment, and its now up to the programmer to decide how to dole them out. If you want a read-only agent, then just drop write and bash . Since a tool is just a Tool.v https://tangled.org/anil.recoil.org/ocaml-deepseek/blob/1edcdad29a19924f988c7adef4343cb189dcb4a2/lib/tool.mli L26 function over a codec and a handler, callers can add their own depending on the business logic of the app. For example, I've now got some tools in this webserver that query the size of the connection pool, another that can search the in-memory logs, and so on. 4.2 managing-deterministic-and-reproducibility Managing deterministic and reproducibility One of the advantages of running a local model is that we have better control over the determinism of the inference. Aside from obvious factors like the weights and the inference code being the same, we normally still get different results from GPU based inference due to the parallelism. However, if we don't mind taking a slowdown, the CPU backend here supports saving and passing in the same seed: bash $ humpty-cpu chat 'tell me a joke about camels in one sentence' -v humpty-cpu: INFO model: DeepSeek V4 Flash vocab 129280 humpty-cpu: INFO seed: 1690400691090126 random; pass --seed N to reproduce Why did the camel cross the desert? To get to the other hump-side $ humpty-cpu chat 'tell me a joke about camels in one sentence' -v humpty-cpu: INFO model: DeepSeek V4 Flash vocab 129280 humpty-cpu: INFO seed: 1690392111533869 random; pass --seed N to reproduce Why did the camel cross the desert? Because it was sick of the hump-drum of the same old sand dunes. $ humpty-cpu chat 'tell me a joke about camels in one sentence' --seed 1690400691090126 Why did the camel cross the desert? To get to the other hump-side I was surprised, however, to find that the deterministic seed also worked in the Metal backend bash $ humpty-metal chat 'tell me a joke about camels in one sentence' -v humpty-metal: INFO model: DeepSeek V4 Flash vocab 129280 humpty-metal: INFO seed: 2042575750328474 random; pass --seed N to reproduce Why did the camel cross the desert? Because it was tired of the hump-drum of its daily routine. $ humpty-metal chat 'tell me a joke about camels in one sentence' -v humpty-metal: INFO model: DeepSeek V4 Flash vocab 129280 humpty-metal: INFO seed: 2043271538873965 random; pass --seed N to reproduce Why do camels never get stuck in traffic? Because they have built-in "hump" day passes. $ humpty-metal chat 'tell me a joke about camels in one sentence' -v --seed 2043271538873965 humpty-metal: INFO model: DeepSeek V4 Flash vocab 129280 humpty-metal: INFO seed: 2043271538873965 Why do camels never get stuck in traffic? Because they have built-in "hump" day passes. 5 implications-of-using-llms-as-a-library Implications of using LLMs as a library We've seen so far that the LLM model can be exposed as a mostly pure function; and that a tool is a function whose type says what it may touch; and that an agent is just a loop over them. Crucially, there's no need for serialisation protocols, REST APIs, MCP authentication and many of the other layers built over them unless the application wants it. One of the big advantages of unikernel-style libraries is that necessary functionality can be specialised at compile-time much more easily. 5.1 building-a-safe-bastion-using-dikjstra-monads-or-refinements Building a safe bastion using Dikjstra monads or refinements This library approach is orthogonal to the idea of "language integrated" LLMs, which involve discharging verification conditions by having LLMs attempt to synthesise proofs of statements embedded within the source code. Ranjit Jhala observed https://x.com/RanjitJhala/status/2065280989367357725 that: "Language integrated" is a drum I've been beating on for a while e.g. with refinement types , but in the age of LLMs I wonder if itreallymatters, if the AIs are going to also be generating the proofs? -- Ranjit Jhala, June 2026 Yaron Minsky https://github.com/yminsky noted that https://x.com/yminsky/status/2065403677309915570 "language-integrated assertions and modular specification look like a form of intelligence amplification for whatever is doing the proofs" . I also found Shriram Krishnamurthi https://cs.brown.edu/~sk/ and Flatt's "Type-Error Ablation and AI Coding Agents" https://arxiv.org/abs/2606.01522 paper that found richer type errors help an agent fix code, and that a type system helps it more than test failures do. All this means is that we needn't stop at just using types to encode our safety properties. Since each tool is an ordinary OCaml function, nothing stops them being synthesised or checked by a proof assistant like Fstar https://www.fstar-lang.org/ , so that a tool can ship with a machine-checked guarantee that it stays inside the policy its signature advertises. We explored this last year in a HOPE abstract /papers/2024-hope-bastion via Dijkstra monads https://arxiv.org/abs/1903.01237 , since they let you reason precisely about the effects a computation is allowed to have. Patrick Ferris https://patrick.sirref.org Cyrus Omar https://web.eecs.umich.edu/~comar/ and I sketched how to modularise some more dependently typed policy reasoning in Bastion /papers/2024-hope-bastion . Eio's capabilities are really just a lightweight version of the sorts of things you can express in a full proof framework, and as Ranjit notes above, LLMs make it easier than ever to just pick the right specification language for the problem at hand. Another obvious direction to take my library-agentic-harness is to link it in with OCaml's compiler-libs https://ocaml.org/manual/5.4/api/compilerlibref/Compiler libs.html to build a much more tightly integrated agent debugger that doesn't need to go via CLI tooling 5.2 letting-a-service-watchand-mutateitself Letting a service watch...and mutate...itself What else could we do with an LLM we can call as cheaply as any function, on hardware you own? I'm assuming we have spare CPU/GPU here The experiment I'm mid-way through is wiring this into my zero-allocation OxCaml webserver /notes/oxcaml-httpz , which already emits custom runtime events https://ocaml.org/manual/runtime-events.html alongside OCaml's native GC and scheduler ones. Normally they pile up in a ring buffer nobody reads until something's already broken. The idea is to spend idle CPU on them, so when the server isn't busy, I hand a window of recent events to my agent function in a separate domain and check to see if the latency is drifting, or if that event is firing far more than yesterday based on summary stats , and so on. Whether a local model is any good at this is something I don't know yet, but I'm enjoying experimenting. As we discussed in the rewilding the web /notes/rewilding-the-web-report workshop and our Internet ecology /papers/2025-internet-ecology paper, self-hosted services unfortunately don't just run themselves /notes/recoil-self-hosting-2026 and need tending to. I could therefore imagine building enough local tooling so that the agent harness would recompile and re-exec the binary autonomously in response to external stimuli /notes/internet-immune-system ... The code's here https://tangled.org/anil.recoil.org/ocaml-deepseek if you'd like to pull it apart. I'd love to hear more about any improbable stunts or agentic harnesses you build yourself In the future, I'll investigate expanding this out to CUDA and also beyond just Deepseek.