# Fieldmix: a painting coach that mixes from the paints you actually own

> Source: <https://dev.to/preciousjac0b/fieldmix-a-painting-coach-that-mixes-from-the-paints-you-actually-own-25me>
> Published: 2026-10-11 21:49:14+00:00

*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass](https://dev.to/challenges/hacktoberfest-week1-2026-10-05)*

I sometimes have sip-and-paint evenings with friends and family. The canvas comes with the drawing already on it, and deciding which colors to put on it is always the bottleneck. **Fieldmix** is my solution to a problem I run into whenever I decide to touch grass.

It's a painting companion that settles the colors, then gets the phone out of your hand as fast as possible. You:

*Left: Gemma's suggestion for a pre-drawn canvas. Right: the same canvas after I corrected it with 18 taps. More on both below.*

The same problem follows you outdoors. You look at a hillside, squeeze out a yellow and a blue, and chase that green while the light moves. So Fieldmix is for sip-and-paint tables with friends, for plein air painters, and for anyone with a box of student acrylics who would rather paint the thing in front of them than follow a tutorial on a screen. That's the Touch Grass part: the screen gets a couple of minutes of setup, and then it's audio and your brush.

A language model will happily give you a mixing ratio. "Mix blue and yellow for green" is fine. "Use 3 parts ultramarine to 1 part cadmium" is a guess the model has no way to check, and a beginner would trust it.

So Fieldmix splits the work:

Gemma is shown *which* paints each recipe uses, but never the amounts, and it is not allowed to state any. The recipe you hear is spliced in by a template after the model's instruction. Any model output that contains a ratio fails validation:

```
RATIO_PATTERN = re.compile(r"""
    \b\d+\s*(?:parts?|portions?|drops?|dabs?|scoops?)\b      # "2 parts", "3 dabs"
    | \b\d+\s*:\s*\d+\b                                         # "2:1"
    | \b(?:one|two|three|four|five|six|seven|eight|half|equal)\s+(?:parts?|portions?|amounts?)\b
    | \b\d+\s*%                                                 # "30%"
    | \b(?:half\s+and\s+half|50\s*/\s*50)\b
    ...
""", re.IGNORECASE | re.VERBOSE)
```

The same check also rejects any paint name that isn't on your palette, so Gemma can't send you looking for a Viridian you don't own.

**A scene outdoors**, using my lagoon waterfront photo:

**A pre-drawn canvas**, using a sunflower I drew for this post:

*Both are real runs, captured frame by frame in the browser at phone size, with Gemma running on my laptop. The rings show where each tap lands. There's no hosted link yet: the public demo on Render is the next step, not done for Week 1.*

*That's the park photo from my eval set. The plan, labels and steps are Gemma's real output for it. The grass came back as 4 Burnt Sienna + 3 Phthalo Blue + 1 White, a muted, earthy green rather than a bright tube green.*

You can run it yourself in about five minutes (Ollama, `ollama pull gemma4:e4b`, then `scripts/dev.sh`). The README has the steps, including how to open it on your phone over Wi-Fi.

**A painting coach that mixes from the paints you actually own.** Photograph the view
in front of you (or the host's sample painting at a paint-and-sip), tap the paints in
your bag, and Fieldmix splits the picture into a few color shapes, works out how to mix
each one from *only* your paints, and talks you through the painting step by step, so
the phone goes face-down next to the palette.

Built for the DEV [Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass](https://dev.to/challenges/hacktoberfest-week1-2026-10-05).

`gemma4:e4b`, running locally through MIT licensed. It's a Vite + TypeScript PWA with no UI framework, plus a small FastAPI backend. There are 132 web tests and 101 server tests, including the CIEDE2000 reference pairs from Sharma et al. (2005).

```
 Phone (PWA, all color math)                 Laptop (optional)
 ───────────────────────────                 ─────────────────
 photo ─► k-means in CIELAB ─► color shapes
 your paints ─► spectral.js mixes ─► recipes
                     │
                     └── shapes + recipes + 768 px photo ──► FastAPI ─► Ollama ─► Gemma 4 E4B
                                                                │
 paint mode ◄── labels + ordered steps (validated JSON) ◄───────┘
 (speech, test dab, wake lock)
```

The photo is shrunk to 512 px and clustered with k-means in **CIELAB**, a color space where distance roughly matches what your eye sees. Clusters closer than ΔE 6 are merged, and specks under 1% of the picture are absorbed, leaving 4 to 10 shapes a person can actually paint. (ΔE is CIEDE2000, the standard "how different do these two colors look" number. Under 2 is hard to tell apart side by side.)

Average blue and yellow pixels and you get grey. Mix blue and yellow paint and you get green, because pigments absorb light rather than emit it. [spectral.js](https://www.npmjs.com/package/spectral.js) (MIT) models this with Kubelka-Munk theory over spectral reflectance curves, and Fieldmix uses it for every mix.

The recipe search is brute force: 1 to 3 colored paints, plus white and black if you have them, in whole parts from 1 to 6, with equivalent ratios removed (2:2 is 1:1). Two things made that fast enough for a phone:

`spectral.mix` to within 1e-9.
The most accurate recipe was often absurd: the sea in one test photo came out as 6 Ultramarine + 5 White + 3 Phthalo + 2 Cadmium Red. Nobody mixes that at an easel for a difference they can't see. So Fieldmix now picks the **simplest** recipe within ΔE 2 of the best one, but never one that drops the match a quality grade. That sea became 2 Phthalo + 2 White + 1 Alizarin.

**How close does it get?** I took three photos on my phone for the eval: a park, a street and the lagoon waterfront. The light was soft and low, so they're mostly greys, dull greens and near-black shadows under the trees, which is exactly where mixing by eye is hardest. From `scripts/eval.ts`:

| Palette | Regions | Recipe under ΔE 10 | Median ΔE | 
|---|---|---|---|
| Student set (6 colors + white) | 17 | 100% | 2.68 | 
| 3 primaries + white + black | 17 | 100% | 3.21 | 
| 4 paints (yellow, crimson, ultramarine, white) | 17 | 82% | 5.15 | 

The 4-paint misses are the same in every photo: the deepest foliage shadow, nearly black. Without a black or an earth color, that palette can't get there. The student set gets within ΔE 5 to 8 of it with Burnt Sienna, Phthalo Blue and a touch of Alizarin. Solving every region of a photo took 1 to 15 ms on the laptop.

The planner is **Gemma 4 E4B** (`gemma4:e4b` in Ollama, a 9.5 GB download, Apache 2.0), running locally on an Apple Silicon MacBook with 24 GB of RAM. It gets the 768 px photo and, for each shape: its color, lightness, share of the picture, position, and the names of the paints in its recipe. It returns JSON: a label per shape and 5 to 10 ordered steps.

Every plan is validated:

If a plan fails, Gemma gets one retry with the problems listed. If that fails too, a deterministic planner takes over: top to bottom and darks first for acrylic, oil and gouache, light to dark for watercolor. It exists twice, in Python and in TypeScript (so it runs on the phone), and both are tested against the same fixture file.

From `scripts/eval.py`, on the same three photos: 6 requests (each photo × acrylic and watercolor), 3 runs each:

| Valid plans | **18/18** , all on the first try, no fallbacks | 
| Latency, median | 7.0 s | 
| Latency, p95 | 13.1 s (the first request, which includes loading the model) | 

Three things I learned the hard way:

`format` came back `501 structured output is unavailable`. Homebrew's Ollama build runs this model on its MLX engine without the grammar library. So the backend now falls back to describing the JSON shape in the prompt, and leans on validation.
The phone parts matter as much as the model:

`speechSynthesis`. `pause()` is unreliable on Android Chrome, so speech is queued sentence by sentence and Pause restarts the current sentence.
A pre-drawn canvas is black lines on white, so there's nothing in the photo to mix. Fieldmix never invents colors from the picture, so on a canvas like that it lets you choose, and Gemma can suggest.

The drawing's lines become walls, and you tap a shape to fill it with one of your paints, as is or lightened with your white. Loose strokes like curly hair don't enclose anything, so color leaked into the background. I tried closing them automatically with a morphological filter. It broke the background into pieces and *still* didn't separate the hair, so I reverted it and added a **Draw outline** tool instead: drag your finger where the drawing is open, and fills stop there.

**Suggest colors** numbers the biggest shapes on the drawing and asks Gemma to pick one of your paints for each, plus how much white. Code still does every mix, and every answer passes the same checks as a plan: every number answered once, only your paints, no amounts.

It's a good start, not a finished answer, and the screenshots above show both sides. On the sunflower canvas (which I drew for this post), Gemma got the sun, the table and the cloud right. But it read shape 5, the flower's center, as "flower petals", and shape 6, the pot rim, as "stem and leaves". On a portrait in 4 earlier runs, every answer was valid and the picks were sensible (yellow hair, a red cap), but Gemma called the hand an "animal" every single time.

So it's presented as a starting point: tap any shape to change it, and Undo removes the whole suggestion. Fixing the sunflower took 18 taps, 12 of them petals, starting from a colored canvas instead of a blank one. In a later run on that same portrait, Gemma got it right: a warm brown face, dark brown curls, a brown hand, a blue cap, a pink heart, and a background split into blue and red. Runs vary, and the recording below caught a worse one: a yellow face and a red hand. Two taps with Raw Sienna fixed both before planning.

✏️ **In DEV's editor:** drag `docs/upload-to-dev/coloring-page-demo.gif` onto this line (DEV uploads it and inserts the image), then delete this line.

*The portrait run, recorded at phone size: Suggest colors, two corrective taps, then Gemma's plan and the skin-tone step (1 part Raw Sienna). Coloring page found on Pinterest (artist unknown).*

Recording the demo caught one more bug. After Color it in, the plan was still written from the *blank* drawing, so Gemma couldn't see which color went where, and it titled a red step "Flower Petals" when red was the pot. Now Gemma plans from the colored-in drawing. In the next run it named the sky, stem, sun and cloud correctly, but it still called the red pot "the flower". Labels are the one part that can be wrong; the recipe under each label is always computed.

**It runs where painters are.** All the color work is MIT-licensed code running in the phone's browser. Once the page is loaded, the shapes, recipes, test dabs and a built-in plan need no signal at all. Gemma runs on a laptop on the same Wi-Fi or phone hotspot, not in a data center. Here's exactly what works where:

| Setup | Shapes and recipes | Plan | Spoken steps | Questions | 
|---|---|---|---|---|
| Phone alone (page already loaded) | ✅ | Built-in planner | ✅ browser voice | ❌ | 
| Phone + laptop running Gemma | ✅ | ✅ Gemma | ✅ | ✅ | 

A hosted demo would add an HTTPS link anyone can open, but it isn't built yet.

**It costs nothing per brushstroke.** There's no API key, no account and no per-request bill. The model is a one-time 9.5 GB download.

**Your photos stay yours.** Photos of your painting, your street or your living room go from your phone to your own laptop and nowhere else.

**I could open every box.** This is where open beat closed for me in practice. I read spectral.js's mixing model, re-implemented it for speed, and proved my version matches the original to 1e-9. You can't do that with a closed color API. With open weights running locally, I could also turn Gemma's thinking off and measure the difference myself (about 3× faster, just as valid). And the provider is a small interface, so the model can be swapped.

To be clear, I didn't run a closed model against the same prompts, so I'm not claiming Gemma beats one. The win was control and locality: the color math is inspectable, and the model runs on hardware I own.

I took the eval photos outside on my phone: a park, a tree-lined street and the lagoon waterfront. To be honest, I didn't open Fieldmix out there. I ran the photos through it afterwards, and that was still worth it, because real photos found things my test images never did.

The next test is the real one: at the easel.

I built Fieldmix with Claude Code as a pair programmer. I wrote the spec, made the decisions, and tested on my phone; Claude wrote most of the code and kept a dated dev log of what broke and how it was fixed ([DEVLOG.md](https://github.com/PreciousJac0b/fieldmix/blob/main/DEVLOG.md)).

**Best Use of Gemma.** Gemma 4 E4B runs locally through Ollama and does three jobs:

It's deliberately kept away from the one thing it would get wrong: the color math. Every response is checked against a schema and a set of rules, with one retry and a deterministic fallback.

*This post was drafted with AI assistance (Claude), then checked and edited by me. All the numbers come from the repo's eval scripts.*