Early experiments with AI programming in Livelymerge

Researcher Dan connected Grok 4.5, running inside the Cursor agent environment, directly to Livelymerge's live Morphic development environment in under an hour by treating the LLM agent as another collaborating user rather than building new API plumbing. The session, logged as the "Play with Me" (PWM) project, established the agent as a live code author whose changes could be transpiled into the running Morphic and QBF systems with Dan's permission. Livelymerge's Morphic environment, written by Dan in VS Code and later Cursor, already supported multiple simultaneous users, which made the agent-as-user approach possible.

“I had the red shoes all along” loosely from The Wizard of Oz Introduction It is not that unusual today to find LLMs connected directly to an execution environment. In most cases, a considerable amount of research into APIs of the LLM service and the host execution environment is required to make this work. The following brief note and transcript of my session with Grok recount a surprisingly simple path to this union in the case of Livelymerge. It was achieved in less than an hour by a researcher who knew nothing about the APIs involved nor the “plumbing” required to connect them. Livelymerge includes a Morphic development environment ../lm-06 similar to Squeak or Lively Kernel, but notable for allowing multiple users to collaborate in the construction and the programming of the development environment itself, as well as applications built in it. I Dan began writing Livelymerge’s Morphic implementation in VS Code, and after a while started using Cursor, a VS Code-like environment with built-in support for LLM agents. This has been great to help with coding, especially with subtle issues of coordinate transformations and tedious refactoring chores. Once the basic coding of Livelymerge was done, I had a chance to work on a real application see QBF ../lm-05 . Working more as a user, I found myself wanting a more immediate collaboration with the LLM. It would be so much easier to say “Make this change and run init” than it is to say “Make this change,” and then copy the new code file, paste it into Livelymerge, and type, select, and run init . I was aware of Gilad Bracha’s live LLM connection to his Newspeak document https://www.youtube.com/watch?v=D1mEQ6ixJUk , and also Craig Latta’s use of live LLM connection in https://www.youtube.com/watch?v=5HbbIa5NT0A , but for several weeks I shied away from the hard work of figuring out the APIs of Patchwork, reading about the plumbing of an LLM, and wiring the whole thing up so that it would do what I want. Recently I realized that Cursor’s agents had actually been running parts of my application live in a duplicate environment to perform various tests. It occurred to me that if our system works already for multiple users, all I should have to do is get the agent to be another user. That I did, and here is a log of what happened for the next hour, beginning with my prompt that got it started. Beyond that prompt, my only input was to grant a couple of permissions and confirm results seen on my shared display… The project log How to read this section. What follows is the transcript of my session with the agent Grok 4.5, running in Cursor , laid out like a script. My prompts are quoted exactly as I typed them. The agent’s replies are also included, trimmed of tool output and a few asides. Between the lines of dialogue, stage directions describe what the agent did. Exchanges that wandered off topic were left out. Founding I think I’d like to branch off here to a new project, but retaining access to the two main bodies of context we have shared, let’s call them simply ‘Morphic’ meaning a rudimentary self-supporting programming environment in this case in the Patchwork platform , and ‘QBF’ meaning a simple multiplayer game programmed in the same Morphic environment . I’ll refer to this project as “Play with Me”, or briefly PWM. Please establish that project and record its goal, which is: to establish ‘you’, ie the LLM agent with whom I am communicating, as a collaborator in various live programming sessions in at least these Morphic and QBF live systems. In addition to recording that goal, I’d like you to describe how we might achieve it; ie, by various steps that allow you to act as an active author of code which at least for now with my permission can not only become part of the “alldefs” and “QBF” codebases, but which will, even more importantly, be transpiled and entered into the running system that will become the focus of our collaboration. My hope is that you will be able to take a Patchwork URL from me, and, for instance, put on our shared screen a TextMorph with the string “Hello from Grok 4.5 ”. Please consider all of this, record it as the beginning of our “project log”, and tell me if you are ready to load up your own world from a URL that I have running PWM is founded. Docs live at livelymerge/pwm/PROJECT.md goal + approach and livelymerge/pwm/PROJECT LOG.md first log entry . Also created a Cursor long-running goal so this stays a tracked collaboration, not a one-off chat. Goal short : make me a live collaborator in Morphic and QBF—take your Patchwork URL, author with your OK into alldefs / QBF , and more importantly transpile/install into the running world. First proof: a shared TextMorph saying Hello from Grok 4.5 . How sketch : URL → agent-side Livelymerge runtime on that DocHandle → permission-gated eval → transpile/install → optional repo sync → Hello morph. Later: inspect, QBF, ephemeral vs persistent authorship. Ready for your URL? Yes—send the Patchwork link for the world you have running. I’ll treat joining and the Hello morph as the next step. Fair caveat: I can already edit sources and reason about the runtime; the live join may need a thin bridge handle + runtime.eval from this side . We’ll discover that together as soon as the URL is in hand. Grok gets to work on the bridge. It writes a headless script, pwm/join.ts , that opens an Automerge repo, finds a document by its URL, and binds the Livelymerge runtime to it with no canvas. It also writes a permission gate to be pasted into the world, and a test that places a TextMorph in an offline world. Please tell me what’s happening: are you trying it? Do you need me to add some other defs to, eg, alldefs.js? Short version: I’m not live in your world yet — I’ve been building the on-ramp so we can join when you give a URL. What’s done - PWM docs + project log pwm/worldSide.js — Morphic-side gate arm / hello / inbox ; you paste this once or we can fold it into alldefs later if you want - Hello rehearsal tests offline runtime.eval works pwm/join.ts — headless CLI: Automerge URL + sync WebSocket → runtime.eval / Hello What’s blocked I still need your running world’s URL automerge:… and/or Patchwork link . Without that I can’t attach to the same document. Bottom line: Ready for your URL; not attempting live Hello until that arrives. Getting Hello to work Awesome I’m glad you’re willing to give this a shot. As a start here is a Patchwork URL which I have been able to use as a second participant. The only thing is that it seems to require my evalling “initUI ” to show the screen. I don’t know if that will matter to you. Here’s the URL: https://patchwork.inkandswitch.com/ doc=automerge:Hopzo…6ro&type=livelymerge Joining that doc now and trying the Hello TextMorph. initUI is browser-local chrome — we shouldn’t need it for a shared morph write. The first attempt fails: the Automerge repo won't initialize in Node, and the default sync relay has never heard of the document. Grok switches to Ink & Switch's Subduction relay, joins the document as a peer, and evals a small script that creates a TextMorph and adds it to the world. It has no way to see whether the morph is visible. Would it be helpful for you to ‘see’ what’s on my screen? Yes — a lot. Agent-side eval succeeded, but that only proves the morph is in the shared heap. A look at your screen would tell us whether it’s actually visible position, size, buried under other morphs, or whether your tab still needs initUI . A screenshot of the Morphic canvas is ideal; a short description works too “I see the Hello text near the top-left” / “blank canvas” / “Morphic UI but no Hello” . Yes It works Congratulations Great — first PWM loop is closed: shared URL → agent peer → live TextMorph → both of us see it. Still open for the fuller goal: permission-gated evals, and authoring into alldefs / QBF then installing into the running world. Whenever you want to try one of those, say the word. Here’s a screen shot Yep — that matches: Hello sits across the ellipse and covers Recent changes / Morphic help in the world menu. Say the word and I’ll move it to a clearer spot on the shared world and leave your menu alone . Yes, please move it. That would be a thrill to see. Grok evals a script that moves the morph. Moved. The Hello morph should now be at about 420, 360 — lower-right of the demos/menu area. Take a look and tell me if that lands in the clear, or if you want it nudged again. Bingo Bingo indeed — live shared-world control is real now. Installing a method Grok moves on to the second half of the goal: not just evaluating a script in the shared world, but installing a method on a class. It writes a fragment of a class definition, a method called pwmPing on WorldMorph , and checks that it installs in an offline world. Next step for the fuller PWM loop: transpile a method into the live world same path as the Morphic browser’s replaceMethod , not just eval a script. Offline that already works — fragment in pwm/fragments/WorldMorph pwmPing.js , installs on WorldMorph , adds a small “pong from Grok live install ” note. If you’re game, I’ll: - Install pwmPing into your running world- Call it so you see the note appear - Optionally splice the same method into alldefs.js so it survives a cold start OK to do that live? yes, go for it Grok installs the method in Dan's running world through replaceMethod , the same path the system browser uses when you save a method. Then it calls the method, and writes the same source into alldefs.js on disk. It’s done now: Live install: replaceMethod 'WorldMorph', pwmPing → true Live call: Lively.pwmPing → pong Repo: same method written into alldefs.js You should see a note near the Hello morph: pong from Grok live install . That’s the full Morphic loop: fragment → transpile/install → shared screen → durable source. Ooh I see ‘pong’ That’s the whole Morphic PWM loop closed — shared screen proof that a method we authored was transpiled into your live world and mirrored into alldefs.js . QBF Grok turns to QBF. Dan's QBF board is an ephemeral morph, so it exists only in Dan's browser and Grok cannot see it. Grok installs a method, pwmTag , on the shared QBFMorph class, opens a new board, makes it persistent so that both of them will see it, and writes the method into QBF.js . I see hello – QBF collab PWM goal is done. You seeing Hello from Grok — QBF collab closes the last gap. Also just mirrored pwmTag into QBF.js , so Morphic and QBF both have the live install and the durable source. One wrinkle we hit: normal openQBF is ephemeral your private board wasn’t visible to me ; the shared demo used bePersistent . Worth remembering for future collab play. I closed your QBF - it was gobbling ops. Not sure why, but more for tomorrow Good call — that tracks. Stock QBF is meant to stay ephemeral; promoting a ticking board with bePersistent almost certainly pushed per-tick work into the shared doc. How did Hello work? Can you please give me a summary of how you did the “hello” - something like got the morphic URL running figured out what would put text on the screen wrote a method to do this installed that method with Dan's permission but how; did you eval a complete method def? ran the method but again, how; did you eval a call on that method ... and what do you do to eval in the system that you are running from the URL? Please write me a little essay with your answer. The short version of Hello is simpler than “write a method, install it, call it.” For the first shared-screen greeting we did not install a class method. We joined your live Automerge document as a headless peer and eval’d a small script that constructed a TextMorph and added it to the world. The later “pong” demo was the method-install path; Hello was the direct-eval path. Both use the same join/eval plumbing. 1. Get the Morphic URL running on the agent side . You gave a Patchwork link whose hash held doc=automerge:Hopzo… . That id is the shared world’s identity. From Cursor/Node we ran pwm/join.ts , which: - opens an Automerge Repo against Ink&Switch’s Subduction relay wss://subduction.sync.inkandswitch.com ; repo.find docUrl and waits until the handle is ready;- binds createLivelymergeRuntime handle — the same Livelymerge runtime Patchwork uses, but without a canvas browser stubs only . We do not load a second Patchwork UI. Your tab already has Morphic in the heap Lively , TextMorph , … . We attach to the same document so our writes sync to you. A second browser tab may need initUI to draw ; the agent never needs that. 2. Figure out what puts text on the screen. In Morphic, a greeting is a shared TextMorph owned by the world: js = { let t = new TextMorph rect 60, 60, 320, 48 , 'Hello from Grok 4.5 ' ; Lively.addMorph t ; return t; } addMorph not addEphemeralMorph matters: ephemeral attach stays local; shared attach becomes Automerge state every replica sees. 3. “Wrote a method / installed it” — for Hello, no. Hello was that IIFE above, hardcoded as HELLO SOURCE in join.ts and invoked with --hello . There was no replaceMethod , no prototype change, no splice into alldefs.js for the greeting itself. What looks like your outline happened next, for pwmPing : - author a class fragment pwmPing { … } in pwm/fragments/… ;- with your OK, eval replaceMethod 'WorldMorph', fragment — LM’s browser path: transpile one member and install it on the live class;- then eval Lively.pwmPing to run it;- optionally mirror the fragment into alldefs.js so cold starts match. So: Hello = do-it script . Pong = install method, then call it . 4. How do we “eval” in the system from the URL? After join, every action is: runtime.eval sourceString That is Livelymerge’s transactional eval against the DocHandle. It mutates the shared Morphic heap; Automerge sync pushes the change to your Patchwork tab; your renderer paints the new morph. Permission for Hello was conversational “go for it” / URL shared as invitation . We built a stricter gate Lively.$pwm.arm / inbox but Hello used the direct peer-eval path, not that gate. 5. Afterward. Moving Hello was the same mechanism with different source moveBy to pt 420,360 . Seeing it on your screen closed the loop: URL → peer join → runtime.eval → shared morph → your eyes. Epilogue What, if anything, is new here? Not the idea of an LLM wired to a live programming environment: agents that run code in a REPL are everywhere now, and Gilad and Craig have both connected LLMs to Smalltalk-style systems. What I Alex think is worth calling out is how little it took , and what about Livelymerge made that possible. Livelymerge is multi-user, so an agent can act as just another user. Grok did not get an interface of its own: it started a Node.js process and joined Dan’s world as a peer, using the same runtime that the Livelymerge tool in Patchwork uses. And Livelymerge is live just like Smalltalk: a user can evaluate any expression against the running objects, and anything the tools can do can be done that way, too, so an agent with no screen can still do anything a person can do. Grok used eval for both inspecting and changing, first to find out how to put text on the screen, then to put it there, and each change appeared on Dan’s screen the moment it was made. It never read from or wrote to the Automerge document directly, and it never needed to know how Livelymerge represents objects, classes, and methods there. We treat that representation as a hidden implementation https://dl.acm.org/doi/10.1145/361598.361623 , and have changed it more than once ../lm-05 without breaking any programs.