{"slug": "training-a-coding-model-to-paint-watercolours-with-trl-and-openenv", "title": "Training a coding model to paint watercolours with TRL and OpenEnv", "summary": "Hugging Face engineers reproduced a viral AI art project in which a language model trained with reinforcement learning writes JavaScript to paint watercolours, publishing all artifacts openly. The project, led by Surya Narreddi's original idea, uses TRL and OpenEnv to train a Qwen model to generate p5.brush code, with the full pipeline running on Hugging Face and the training recipe available as a single command.", "body_md": "Reinforcement Learning • Updated\n\n# Training a coding model to paint watercolours with TRL and OpenEnv\n\n[Update on GitHub](https://github.com/huggingface/blog/blob/main/train-to-paint-with-code.md)\n\nOn 23 August, [Surya\nNarreddi](https://x.com/kickingkeys/status/2091570990048276897) posted a beautiful video\nof watercolours painted by a language model. The model writes JavaScript through\n[p5.brush](https://github.com/acamposuribe/p5.brush), a library that \"adds natural\ndrawing tools to p5.js\". The video went viral fast, over 1.5M views at the time of\nwriting.\n\nThe video came with [a blog\npost](https://surya.website/rling-qwen-to-paint-with-code) explaining the training\nbehind an earlier and narrower stage of the project, close-up flowers rather than the\nfull compositions in the video, sadly without open artifacts yet. His site says a full\ntechnical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where [his skills are way beyond mine](https://x.com/kickingkeys/status/2094901433149612118). My attempt is on the engineering side, reproducing the recipe in the open with every piece published.\n\nNote:for the context behind the project, told by Surya himself, watch[this video of his thesis].\n\nIn this article I try and reproduce his idea with [TRL](https://huggingface.co/docs/trl) and\n[OpenEnv](https://github.com/huggingface/OpenEnv). The reference pool dataset, the RL\nenvironment, the training scripts and the trained models, all open.\n\nThe whole pipeline runs on Hugging Face, end to end:\n\n- training on\n[Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs) - the RL environment and the scorer model as\n[Spaces](https://huggingface.co/docs/hub/spaces) - the pairwise judge through\n[Inference Providers](https://huggingface.co/docs/inference-providers) - and every artifact on the Hub, gathered in\n[one collection](https://huggingface.co/collections/HuggingEnvs/paint-with-code-6a955b79d63f67f1631d9be6)\n\nOnce the two Spaces are up, the recipe is one command. Duplicate the\n[environment](https://huggingface.co/spaces/HuggingEnvs/watercolour-env) and the\n[scorer model](https://huggingface.co/spaces/HuggingEnvs/watercolour-hpsv3), set two\nenvironment variables for the reward mix, and launch:\n\n```\nhf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \\\n  --env-url https://<you>-watercolour-env.hf.space \\\n  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \\\n  --subject 'a peach hibiscus' --references 4 \\\n  --top-p 0.95 --top-k 20 \\\n  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \\\n  --scale-rewards none \\\n  --steps 110 --n-episodes 240 --num-generations 8 \\\n  --per-device-batch-size 1 --gradient-accumulation-steps 8 \\\n  --max-completion-length 8192 \\\n  --run-tag my-run --out <you>/watercolour-grpo --push-to-hub\n```\n\nThe rest of this article is the story of getting there, and [every piece is in the\nrepo](https://github.com/adithya-s-k/HuggingEnvs/tree/main/02-watercolour).\n\nI have followed the original blog step by step, and only changed something when strictly needed. Every idea of my own went into a list instead of into the experiment, and that list became \"What I would try next\" at the end, next to the full list of published artifacts. If you have already read his post, the framing and\nthe reward design will be familiar. The new material is the open implementation, the\nhand-rated pool, and three reward mixes trained and compared, and it starts at [The RL\nenvironment you need to build](#the-rl-environment-you-need-to-build).\n\n## Why people loved it\n\nThe paintings look loose, imperfect, handmade, at a moment when image models produce perfect (statistically average) pictures. My guess is that this contrast is a big part of why the video went\nviral. It reminded me of the early days of generative AI art, when the point was\nto explore the medium. [DeepDream](https://research.google/blog/inceptionism-going-deeper-into-neural-networks/) (2015) was a debugging tool that people turned into art, works like [Edmond de\nBelamy](https://en.wikipedia.org/wiki/Edmond_de_Belamy) (2018) came from artists probing\nwhat a GAN could do, and artists like [Mario Klingemann](https://quasimondo.com) spent those years making [dreamy portraits with neural networks](https://artsandculture.google.com/asset/memories-of-passerby-i-mario-klingemann/aAHG7iV3aXme8g).\n\nThis project feels closer to those early days. In his thesis, Surya describes the path\nthat led here. He started by prompting text-to-image models, where the prompt is the\nonly lever you can pull, and more detail buys more control only up to a point. Training\nthe model itself goes further. The\nother half of the idea is the medium. The model writes a program of about 150\nlines of JavaScript that paints the image. That model output is code. You can read it, edit it and run it\nagain, and the decision behind each brushstroke is visible. And the style comes from a\nrestriction where the model is only allowed *ten of the library's methods*. More on that below.\n\nIn that same period, [Anna Ridler](https://annaridler.com/works/myriad-tulips)\nphotographed thousands of tulips, hand-labelled every one, exhibited the dataset itself\nas the artwork, and later trained a model on it. I found her work through the references\nAI agents brought back while building this project and loved it because this\nproject does something very similar by curating a set of images by hand, and then training against them.\n\n## RL over taste\n\nMost of the recent RL work on language models uses rewards you can verify. For example, math problems with a known answer, code that passes tests, or graders that are right or wrong and cheap to run. This project is closer to the older exception, RLHF, where the model learns a reward model from human preferences.\n\nHere the reward is aesthetic preference. There is no *correct* answer. The real question\nof the project is whether you can do RL over taste.\n\nThe reward, as his blog defines it and as the RL environment I built implements it:\n\n| term | weight | what it measures |\n|---|---|---|\n`gate` |\n0.05 | the sketch compiles, paints something, does not cheat |\n`length` |\n0.05 | a soft push towards longer code snippets |\n| pairwise judge | 0.60 | style, compared against references drawn from a pool |\n|\n\n[HPSv3](https://huggingface.co/MizzenAI/HPSv3) is an open 7B preference model. Give it an image and a text description, and it returns\na score for how much a person would prefer that image. It was trained on a large set of\nhuman choices between pairs of images, so its score is an average of many people's taste. The pairwise judge is\n[Qwen3-VL-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct), a\ngeneral vision model called through HF Inference Providers. The pairwise judge sees the candidate\npainting\nnext to four references randomly selected from the pool, guided by a written description of what to weigh (bleeds, translucent washes, soft edges),\neach comparison in both presentation orders, and its score is the share of comparisons\nthe candidate wins. Its only standard is the pool, so its score is my taste, as encoded\nin those ratings.\n\nThose are the weights Narreddi converged on. The pool defines taste here. That moves the work from tuning hyperparameters to building the set that decides what is beautiful.\n\nI trained three runs with this reward. They differ only in how the weight splits between the two model judges:\n\n| run | pairwise judge | HPSv3 | role |\n|---|---|---|---|\n`judge-led` |\n0.60 | 0.30 | the original mix, stopped at step 110 |\n`hps-led` |\n0.30 | 0.60 | the middle point, stopped at step 110 |\n`hps-only` |\n0.00 | 0.90 | the validation run, stopped at step 60 |\n\nI started with `hps-only`\n\nto validate that the pipeline could learn at all. Once the reward was going up and the metrics were healthy, there was no reason to run it longer, so I launched the two longer runs instead. The question that the longer runs ask is how much of HPSv3's power can you hand to the pairwise judge? The more weight the judge carries, the more\nthe reward means *my* taste instead of everyone's, and the harder it should be to climb. Incidentally, if you push it far enough or your style is too far from the average, the model could stop entirely.\n\nFortunately, it did not stop and both runs with the pairwise judge on learned too. The hand-rated pool can steer the policy, at least as far as the metrics and the final paintings show. The numbers are below.\n\nDisclaimer.If we use a frontier model, it can already generate the JavaScript code that paints watercolour from a prompt. That's the starting point. The work here is about teaching a smaller model to do it combined with a person's own artistic preferences.\n\n## The RL environment you need to build\n\nThe environment wraps everything that sits between the model and the reward, including the JavaScript library that the model uses for painting, the system prompt that restricts it, the headless Chromium that renders each sketch, and the gate that rejects cheats.\n\nThe library does more of the work than it seems.\n[p5.brush](https://github.com/acamposuribe/p5.brush), by\n[@acamposuribe](https://x.com/acamposuribe), simulates a medium rather than drawing\nshapes: pigment bleeds past the edges of a fill, paper has texture, strokes have mass,\nflow fields drag brushwork around. When the model calls `brush.fillBleed(0.25)`\n\nit is deciding how far the ink\nruns.\n\nNote.The author of p5.brush had been trying to teach a machine to paint long before any of this. In 2022 he made a generative art series that hides a diary about teaching p5.js to draw like a child:\"It is barely able to use the crayons [...] It cannot follow simple commands. I'm done for today, very infuriating.\"The series was meant to have three pieces, and he made two. When Surya's video went viral,[he quoted it], sharing that diary and saying this work is the third piece arriving on its own.\n\np5.brush exposes 47 methods. The prompt allows 10: `scaleBrushes`\n\n, `noStroke`\n\n, `fill`\n\n, `noFill`\n\n, `fillBleed`\n\n, `fillTexture`\n\n, `beginShape`\n\n, `vertex`\n\n, `endShape`\n\nand `circle`\n\n. What the other thirty-seven add, lines, hatching, custom brushes, would break the watercolour look. With these ten the model can only paint filled shapes, and the library adds the bleed to every one of them.\n\nHis blog post saved me a lot of time I could have wasted iterating on the prompt. A long API reference makes the model invent methods that do not exist, and his 200 GEPA iterations converged on a strict allowlist with no documentation. I saw the same failures and wrote the allowlist by hand. My only addition to that recipe is one sentence: paint each petal two or three times, a big pass first and a smaller, more opaque one inside it. This small change made my outputs a lot more colorful.\n\nNote.If it's the first time you hear about[GEPA], it is an automatic prompt optimizer. A language model reflects in plain words on where the current prompt failed and proposes a better one, and the loop repeats.\n\nThe gate is the final piece. The sketch has to compile, use the library instead of direct p5 calls, put real pigment on the canvas, and not try to trick the scorer, for example by writing text on the canvas.\n\n## The pool is the reward function\n\nThe pool consists of [178 paintings](https://huggingface.co/datasets/HuggingEnvs/watercolour-reference-pool) divided into\ntwo tiers based on my personal preferences, `love`\n\nand `okay`\n\n. All of them are actually generated by a model. Four open-weight models,\ncalled through Inference Providers, wrote p5.brush sketches, each one working from a\nreal, openly licensed photo of a hibiscus from iNaturalist. A vision model gave written\nfeedback on every sketch, over three refinement iterations. Every final render was then rated one at a time, by me, and 178 made the cut.\n\n| generator | number of paintings |\n|---|---|\n|\n\n[Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3)[Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next)[Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)Here I chose four different families of models to test their different styles. These four were the open models that produced a valid sketch every time in a quick reliability check, and two other candidates were dropped for failing it. If you want to produce your own pool, you might choose others.\n\nThe tiers do real work in the reward. When the pairwise judge draws four references,\nhalf come from `love`\n\nand half from `okay`\n\n, so the policy always faces some rivals it\ncan sometimes beat, and a win pays the same against either tier. This is one of my few\ndeliberate changes: the original compares against its top tier only, and I kept the\neasier tier in the draw so a weak early policy still gets signal.\n\nNo human-made painting is in there, which is a real limitation. p5.brush is a niche library, and the human work that exists in it with accessible code is a handful of pieces, nowhere near what a training corpus would need, as his blog also notes.\n\nThe interesting idea here, as I've already discussed previously, is that the model will learn to imitate what the pool contains. If we point the environment at a different dataset, the reward would automatically change without touching a single line of code. For the dataset I generated and openly share, I also include the source sketch.\n\nIf we look closer at the judges, the two of them answer different questions. HPSv3 decides whether it is a flower, and the pairwise judge decides whether it is well painted in the style I chose.\n\n## Just one more yolo run\n\nBefore anything worked, there was a long stretch of flat reward curves. If you've tried to reproduce a research paper/blog without open artifacts, you probably can relate. Every run tested what I thought was a reasonable theory about what was wrong. A run takes a lot of time, so I queued the next one while still going through the last one's results. As always, starting from something easier that works and then building on top of that was the answer. A simple control task, with no browser and no judges, was the first attempt that learned, and the reason was that my learning rate was just too low.\n\nAnother change that cost me time to find was adjusting the LoRA parameters correctly. The usual `target_modules`\n\nlist assumes a dense model, and [ Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is a mixture of experts that names most of its projections differently, so the adapter was training ten layers out of forty. I solved this by changing it to\n\n`all-linear`\n\n, which reaches every linear layer. The routed experts in this architecture are fused tensors that even `all-linear`\n\nleaves frozen, but everything else gets an adapter, and that was enough to learn.The fix was four changes in TRL's [ GRPOTrainer](https://huggingface.co/docs/trl/grpo_trainer):\n\n| setting | from | to | why |\n|---|---|---|---|\n| learning rate | 2e-5 | 5e-5 |\nthe ceiling LoRA Without Regret uses for GRPO |\n| scheduler | `linear` |\n`constant_with_warmup` |\nlinear decay had spent most of the learning rate by mid-run, so the reward never took off |\n`scale_rewards` |\n`group` |\n`none` |\none gate rejection was shrinking every other advantage in the group |\n`target_modules` |\nhand list | `all-linear` |\nreach every linear layer |\n\nThese four changes unlocked the first successful run (`hps-only`\n\n), with the reward clearly improving.\n\nWith that configuration, all three runs learn. Both judge runs were launched for 200 steps and stopped at 110, with the reward still climbing slowly. A step takes fifteen to eighteen minutes, and the comparison between mixes was already stable, so I stopped both to save compute. Mean group reward over the first and final third of each run:\n\n| run | steps | first third | final third | Δ |\n|---|---|---|---|---|\n`hps-only` |\n60 | 0.58 | 0.71 | +0.13 |\n`judge-led` |\n110 | 0.45 | 0.72 | +0.27 |\n`hps-led` |\n110 | 0.57 | 0.82 | +0.24 |\n\nThe three curves line up with how much weight my taste carries. The more the judge\nweighs, the lower the start and the noisier the climb. `judge-led`\n\nspent its first thirty steps nearly flat\nbefore it moved. It is the same move that solved the debugging. Shrink the problem until\nsomething learns, then add the hard parts back one at a time.\n\nThe pairwise judge term itself climbed in the two runs that used it. The model wins more\ncomparisons against the pool as training advances, which is the claim `hps-only`\n\ncould\nnot make. No group in any run collapsed to identical rewards, the GRPO failure mode that\nkills the gradient. For the curious, the per-metric curves (HPSv3, paint coverage,\nentropy) are [in the\nrepository as CSV](https://github.com/adithya-s-k/HuggingEnvs/tree/main/02-watercolour/results).\n\nThe full launch command, the hardware and the two environment variables that turn this\ninto the other two runs are in [the\nrecipe](https://github.com/adithya-s-k/HuggingEnvs/tree/main/02-watercolour).\n\n## What it actually learned\n\n**In every run, the first thing the model learned was to stop producing bad\npaintings**, the near-blank canvases and shapeless washes that score under 0.3 in total\nreward. In\n`hps-only`\n\n, three quarters of the rise in the group mean comes from bad paintings\nbecoming rare. In the judge runs the collapse is even steeper: rollouts under 0.3 fall\nfrom 99 to 16 across `judge-led`\n\n's thirds, and from 37 to 4 in `hps-led`\n\n.\n\nThis is why the obvious visual, the best painting of each step, shows almost no difference in\n`hps-only`\n\n. It moves **+0.034** across the run while the median moves **+0.155**. The\nlearning is visible in the middle of the distribution.\n\n**What the pairwise judge changes is the top.** In `hps-only`\n\n, paintings got more\nreliable without getting better. The quality of the good ones added just +0.03 to the\ngroup mean, and once HPSv3 saw petals around a centre and a stem, it stopped asking for\nmore pigment. With the judge on, the other half of the story appears. *Better* here means closer to\nthe pool, so closer to what I rated as \"good\" or something I liked more. That added +0.12 in `judge-led`\n\nand +0.16\nin `hps-led`\n\n, the best of each step rose too, and\npaint coverage doubled in both runs (0.11 to 0.23, and 0.13 to 0.30) where `hps-only`\n\nbarely moved it. With a reference left to beat, a good painting can still get better,\nand the model starts being rewarded for using more pigment.\n\nOne more finding. The model ignores an explicit instruction, and it is right to. The\nsystem prompt asks for fifteen to thirty filled shapes. If we look at the real mean, it is between 7 and 9, and\n`n_shapes`\n\nbarely correlates with reward in any run (+0.000, −0.14, +0.07). The policy\nis not rewarded for obeying that sentence, so it does not obey it.\n\nThere is also a ceiling on the `hps-only`\n\nroute. If every rollout matched its good\nones, that run's mean would sit at 0.771. Whether more steps would break it is\nan open question.\n\nThe paintings also show something that the tables miss. Within each run, they all look\nsimilar. As training advances, the rewards inside each group get closer together, and the\nmedian paintings in the opening video look like takes of the same flower. That is GRPO\ndoing what it is designed to do with a pool built from one subject. The pool decides\nwhat counts as variety, the same way it decides what counts as quality. If the reward\nonly pays for matching one flower, the model learns to paint that one flower. More diverse output would need a\nmore diverse pool, and building one is more curation work. Surya's newer compositions are an example of this. [Alex Yango's animal\npaintings](https://x.com/alexyango/status/2091696296931574217) are the same recipe with\ndifferent choices in the pool. This is the biggest difference between an aesthetic\nreward and a maths grader. Behind the number there is a very human job, deciding what\nbelongs in the reward set. [Jason Liu's essay on\ntaste](https://x.com/jxnlco/status/2073819508729684462) says the general version in one\nline. AI shifted the bottleneck from making to noticing.\n\nSurya closes his blog with some of his favourites. Instead of picking mine, below is a wall with the 178 paintings the reward scored highest across the two judge runs, the same number the reference pool holds, in no particular order. Open it and pick your own.\n\nTo choose, you looked at many and kept a few, and that is exactly the job that built the\nreward of this project. Every painting of every run, with its sketch and its reward, is\nin the rollouts datasets, and browsable in [this gallery](https://huggingface.co/spaces/HuggingEnvs/watercolour-gallery).\n\nSince the reward was partly based on my taste, it is fair to close with my verdict as a viewer. To my\neye, `judge-led`\n\nis the run that ends up the most diverse and the most artistically\ninteresting. `hps-led`\n\npaints convincing watercolours, but its best ones share a soft,\nwet-on-wet look that is almost a style of its own. `hps-only`\n\nconverges the hardest, and\nmost of its paintings settle on the same colours. You can judge for yourself in [the\ngallery](https://huggingface.co/spaces/HuggingEnvs/watercolour-gallery), which has every painting of\nevery run, sortable by step and by reward.\n\n## Infra is hard\n\nThis project is mostly infra. A run needs a trainer, two Spaces, an inference router and a websocket to stay healthy for hours straight, and every piece that fails quietly turns into a wrong number somewhere else. Half the work is checking that the number you read matches what actually happened.\n\n**Failures of the infrastructure were entering the reward as zeros.** A render that\ntimed out or a scorer that did not answer scored the same as a bad painting, 0.0 inside\nthe group. Across all my runs that was about 1.5% of rollouts, and in the worst run it\nreached 5.2%. That trains the model on noise, so those paths now\nreturn `None`\n\nand the rollout is excluded from the group.\n\n**I also found a bug in OpenEnv, and sent the fix upstream.** The client keeps one\npersistent websocket, and a socket closed by the far end stayed cached, so every later\ncall failed even though the environment was healthy. It cost me two half-finished runs\nto find it. The fix is [submitted\nupstream](https://github.com/huggingface/OpenEnv/pull/1103), and the runs launched with\nit have been running clean since.\n\n**The reward of a step depends on which references it drew.** The pairwise judge\nsamples four references per step, so every step faces a different set of rivals, and\nsome draws are simply harder. GRPO itself is mostly safe, because advantages are\ncomputed inside the group and a hard draw moves the whole group together. The curve I\nwas reading was not safe, and some of what looked like a bad step was just a hard draw.\nThe image above is one example. Step 12 scored half a point below step 11 mostly because\nit drew the hardest references of the run, while the paintings themselves look close.\n\n## What it costs\n\nRounded numbers, and only for the runs that finished.\n\n| piece | what it needs |\n|---|---|\n| trainer | 1 H200. 18 hours for 60 steps, about 34 for 110 |\n| HPSv3 | an `a100-large` Space, up for the whole run |\n| the environment | a `cpu-upgrade` Space, which renders comfortably in time |\n| the pairwise judge | Inference Providers quota for `Qwen/Qwen3-VL-30B-A3B-Instruct` |\n| the pool, one-off | openly licensed photos from iNaturalist, Inference Providers quota for the four generators, and rating is your own hours |\n\nA step is eight rollouts and takes fifteen to eighteen minutes, of which **70 to 80% is\nrendering**. A single render takes 69 to 96 seconds against a 90 second deadline. Part of that is\nexpected, the Space has no GPU, so Chromium renders the WEBGL canvas in software and\np5.brush's bleeds and textures are heavy pixel work. Even so, I expected it to be\nfaster, and I have not found the full cause.\n\nA scorer can cost more than the training that uses it: HPSv3 has to be up for the whole run, so pause the Space, or set its sleep timer, when the run ends.\n\nEverything runs on [HF\nJobs](https://huggingface.co/docs/huggingface_hub/guides/jobs), with the environment as\na [Docker Space](https://huggingface.co/docs/hub/spaces-sdks-docker) and metrics in\n[trackio](https://huggingface.co/docs/trackio).\n\n## What I would try next\n\nThe rule of this project was to reproduce the recipe with every resource open, not to improve it, so a list of untried ideas piled up along the way. These are the ones I would actually try, in order of how much evidence there is.\n\n**Multi-step, and letting the model see what it paints.** This is the first thing I\nwould try. The original blog trains single turn, so I trained single turn, and in this\nsetup the model paints with its eyes closed. No image ever goes in, and the only\nfeedback it gets is one number. The evidence that a feedback loop works is the pool\nitself. The reference paintings came from models iterating three rounds under a vision\ncritic, and the later rounds are better. The material that defines the reward was made\nwith a loop the policy never gets.\n\n**Smaller models.** There is evidence that 35B is more than needed. In my side\nexperiments a 4B already wrote valid sketches that passed the gate. If a 4B can learn\nthis, the cost of the experiment drops by an order of magnitude.\n\nOther ideas on the list are SFT on the pool sources before starting RL, rewarding pigment\nexplicitly, moving the judge's reference mix from easy to hard as the run advances until\nonly `love`\n\nremains, widening the ten-method allowlist for more visual range (my\nattempts on this crashed more sketches and broke the watercolour look), and checking how consistent the pairwise judge really is by\nscoring the same image twice.\n\nAnd the method is not specific to flowers. [Alex Yango painted animals with the same\nmechanism](https://x.com/alexyango/status/2091696296931574217), and [Brendan Hogan\ntrained canvas animations](https://x.com/brendanh0gan/status/2092650655789855222)\nagainst a pool of hand-rated clips. I had also played with something similar before using [Simon\nWillison's pelican\nbenchmark](https://huggingface.co/blog/sergiopaniego/pelican-env-openenv), where code is\nrendered to an image and scored.\n\nAnd underneath all of it sits the question this project cannot close. **178 paintings\nmade by models define what this trained model considers beautiful.** The pool is the bottleneck,\nand it is the part of the pipeline with no principled answer.\n\n## What I changed from the original\n\nFor anyone reproducing this, there are two deliberate divergences from Narreddi's recipe, both argued earlier in the article:\n\n- The pairwise judge draws its references half from\n`love`\n\nand half from`okay`\n\n, instead of comparing against the top tier only, so a weak early policy still gets signal. - One small sentence of craft in the prompt, paint each petal two or three times, a big pass first and a smaller, more opaque one inside it.\n\nThe rest, `all-linear`\n\nfor the LoRA, infrastructure failures returning `None`\n\ninstead\nof 0.0, stopping the judge runs at step 110, are decisions I had to make along the way\nbecause his blog does not specify them. His implementation is not published, so\nI cannot tell whether they match his choices or diverge from them.\n\n## Everything is published\n\n| artifact | where |\n|---|---|\n| the recipe, and how to reproduce it |\n`02-watercolour/` |\n\n`watercolour-reference-pool`\n\n`watercolour-env`\n\n`watercolour-hpsv3`\n\n[·](https://huggingface.co/HuggingEnvs/watercolour-grpo-hps-only)`watercolour-grpo-hps-only`\n\n`watercolour-rollouts-hps-only`\n\n[·](https://huggingface.co/HuggingEnvs/watercolour-grpo-judge-led)`watercolour-grpo-judge-led`\n\n`watercolour-rollouts-judge-led`\n\n[·](https://huggingface.co/HuggingEnvs/watercolour-grpo-hps-led)`watercolour-grpo-hps-led`\n\n`watercolour-rollouts-hps-led`\n\n`watercolour-gallery`\n\n[·](https://huggingface.co/spaces/HuggingEnvs/watercolour-trackio-judge-led)`judge-led`\n\n[·](https://huggingface.co/spaces/HuggingEnvs/watercolour-trackio-hps-led)`hps-led`\n\n[, and the CSV files in](https://huggingface.co/spaces/HuggingEnvs/watercolour-trackio-hps-only)`hps-only`\n\n`results/`\n\n[Paint with Code](https://huggingface.co/collections/HuggingEnvs/paint-with-code-6a955b79d63f67f1631d9be6)The per-rollout numbers in this article can be recomputed from the published datasets. Nothing here depends on any Space staying switched on.\n\nThe method and original idea are Surya Narreddi's. The library is Alejandro Campos Uribe's.", "url": "https://wpnews.pro/news/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv", "canonical_source": "https://huggingface.co/blog/train-to-paint-with-code", "published_at": "2026-09-03 00:00:00+00:00", "updated_at": "2026-09-03 08:26:02.274233+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-research", "ai-tools"], "entities": ["Hugging Face", "Surya Narreddi", "TRL", "OpenEnv", "Qwen", "p5.brush", "p5.js"], "alternates": {"html": "https://wpnews.pro/news/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv", "markdown": "https://wpnews.pro/news/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv.md", "text": "https://wpnews.pro/news/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv.txt", "jsonld": "https://wpnews.pro/news/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv.jsonld"}}