# Teaching a Web Page to Read a Chess Board

> Source: <https://dennisforbes.ca/blog/2026/11/reading_a_chess_board_from_a_picture/>
> Published: 2026-11-22 00:00:00+00:00

**Recommendation:** For small, occasional in-browser inference, skip ONNX Runtime Web and TensorFlow.js and hand-write the forward pass in plain loops; a 131KB model ran in about a quarter second.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

This entry has three interactive demos that need JavaScript, and they run the real<sup>[1](#user-content-fn-dated)</sup> solution running on the service: the same script and the
same network weights that the site serves. Everything else should be fully cogent viewed as a purely static page.

The demos do work better on a wider screen.

*Spoiler: A 131KB file of signed bytes and about fifty lines of JavaScript loops reads a chess position off a screenshot,
in the browser, in about a quarter of a second. Most of the work was not the neural network.*

# Twenty-Six Pieces, One At A Time

I’ve [written](https://dennisforbes.ca/blog/microblog/2026/10/chess_analysis_service) <sup>↗</sup>
a [couple](https://dennisforbes.ca/blog/microblog/2026/11/new_chess_features) <sup>↗</sup>
of times about the [chess thing](https://chessreport.ca) I’ve been building part-time on the side over the past few weeks. Unlike client work,
this is a project that I can talk about in detail without constraint. It has yielded a lot of content where I’ve
been doing much more interesting work on more significant projects, but they can’t fuel random blog content.

[One of its pages](https://chessreport.ca/board) is a board editor:
You set up a position, and then you can ask Stockfish about it, play it out against the engine, share it,
turn it into a puzzle, and so on.

Sharing a position can be a graphic, an embed (as seen right below in an iframe), and SVG, etc.

Setting up a position by hand is miserable. A middlegame has twenty-odd pieces on the board,
and you’re dragging each one into place while flipping back and forth to wherever you saw the position in the first
place. A screenshot from a game on your phone. A diagram in a book or a video.
A position someone dropped into a group chat with “what would you play here?”<sup>[2](#user-content-fn-chat)</sup>.

The position is right there in the picture. I wanted to paste the picture and have the board set itself up.

This entry is how that got built, including the parts that didn’t work. It took about three days<sup>[3](#user-content-fn-llm)</sup>,
and it went through four neural networks, none of which failed for the reason I expected.

# The Constraints

A few constraints shaped everything that follows. Different projects and goals have very different constraints, so obviously these should not be generalized.

**It runs in the browser, and the picture goes nowhere.** People’s screenshots have their usernames, their opponents,
their notifications and whatever else was on the screen. I don’t want any of it on the server, nor do I want the
server-load of running inference<sup>[4](#user-content-fn-cheap)</sup>. Client devices have gotten enormously powerful, and any recent smartphone runs
this network and correlating logic in milliseconds.

**No machine learning runtime.** The obvious way to run a model in a page is ONNX Runtime Web or TensorFlow.js.
Those are impressive projects that bring megabytes of WebAssembly along for the ride.
For a feature that most visitors will use occasionally if at all, I wanted the cost to be close to nothing,
and fetched only when a picture is actually given.
So the network had to be small enough that I could write its forward pass as a few plain loops.

**Screenshots, not photographs of a real board.** A board as a site, an app, a book or a video draws it: flat,
seen straight on, every knight identical to every other knight. Reading a wooden set on a kitchen table from an angle,
with pieces overlapping each other in perspective, is a very different and much harder problem<sup>[5](#user-content-fn-real)</sup>.
I did end up handling a photograph *of a screen*, which gets its own section below, but I don’t want to overstate how
revolutionary this is. It is dealing with the simplest, albeit by far the most common, scenario where people want to
turn an image into positions, which is usually screenshots. Sometimes photos of a screenshot.

With those in place the problem becomes fairly linear and simple-

The key simplification is the yellow box. I’m not asking a network to look at a picture and emit a position.
I’m asking it the easiest possible question, sixty-four times: *here is one square, what’s on it?*
Thirteen possible answers: empty, or one of six pieces in one of two colours.
Everything hard about the layout of the picture is dealt with by ordinary code before the network ever gets involved,
and everything about chess is dealt with by ordinary code afterwards.

Here it is, running in this page. Pick a sample, or paste a screenshot of your own.

The samples are synthetic, with garish made-up page furniture around the boards, for reasons I’ll get to.
They’re drawn in the site’s own piece set so that I’m allowed to publish them<sup>[6](#user-content-fn-licences)</sup>.

# Finding the Board

A screenshot is mostly not chess board. There’s browser chrome around it, a move list, clocks, avatars, an ad for a mobile game. Step one is to find the square region that’s the board, to within a pixel or two, because if the grid I cut is off by a tenth of a square then every one of the 64 crops has a slice of its neighbour in it.

My first instinct, like everyone’s in 2026, was to throw a detection network at it. I’m glad I didn’t.
A chess board is about the most regular object a picture can contain:
nine evenly spaced vertical lines and nine evenly spaced horizontal lines, with two colours alternating between them.
That is a structure you can just go looking for<sup>[7](#user-content-fn-opencv)</sup>.

So the picture is shrunk until its longest side is 480 pixels, turned grey, and then reduced to two one-dimensional signals. For each column of pixels, add up how much each pixel differs from its left neighbour. That’s how much “vertical edge” the column contains. Do the same for each row and its upper neighbour.

A column that runs along the boundary between two files of the board has a strong edge at almost every row, since for the entire height of the board it has a light square on one side and a dark one on the other. It sums to something enormous. A column through the middle of a file only picks up the outlines of whatever pieces are standing there. The board shows up in each signal as nine tall, evenly spaced spikes, like a comb.

Then it’s brute force, which is affordable because the signals are one-dimensional and tiny. For every size a board could be, from the whole picture down to a fifth of it, slide a nine-toothed comb of that spacing along each signal and score it by what’s under the teeth. A handful of the best positions are kept for each size and each axis.

Two details matter more than they look-

- The score is the sum under the nine teeth *plus four times the weakest of the seven interior teeth* . Without that,
a comb can score well by landing two teeth on some very strong edge elsewhere on the page (the border of a panel, say)
and the rest on nothing at all. A real board has all of its lines.
- The picture’s own edges count as lines. Plenty of pictures are cropped to the board exactly, and a board that fills the picture has no edge at its first and last line to find.

The best 160 candidates then get a second, different test: is this actually a checkerboard?
For each of the 64 squares the candidate implies, sample the grey level near its four corners,
where a piece almost never reaches<sup>[8](#user-content-fn-corners)</sup>. Split the squares into the two groups a chessboard would have.
If it’s a real board, each group is tightly clustered around its own colour and the two colours are far apart.
And any square that’s nowhere near its group’s colour is counted heavily against the candidate,
because that is precisely what happens when the comb has landed one square over: one whole row or file of “the board”
is actually the page beside it.

Try it. The slider chooses a board size; the comb jumps to wherever that size fits best.

The whole thing takes about 35 milliseconds,
and across 400 synthetic test screenshots it found the board to within a quarter of a square in every one.
No training data, no weights, and when it gets something wrong I can read the code and see why<sup>[9](#user-content-fn-debug)</sup>.

# The Board That Wasn’t Square

Then the first photograph of a laptop screen went through it, and came back as nonsense.

Of course it did. Someone had held their phone up to a screen, a little off to one side, and in the resulting picture the board is a slightly irregular quadrilateral. The comb search assumes upright, evenly spaced lines. It locked onto a square a full row below the real board, and the 64 crops were cut from that.

The fix has two parts: work out where the board’s four corners really are, and then undo the perspective.

## Four Corners Are Enough

For any four corner points there is exactly one way a camera could see a flat square between them.
That mapping is a homography, and the closed-form solution for “unit square to arbitrary quadrilateral”
has been sitting in Paul
Heckbert’s [1989 thesis on texture mapping](https://www2.eecs.berkeley.edu/Pubs/TechRpts/1989/Archive/CSD-89-516.pdf)
ever since. It’s about six lines of arithmetic-

```
// how a camera lays the board's own measure (0 to 8 each way) between four corners
const lay = (q) => {
    const [[x0, y0], [x1, y1], [x2, y2], [x3, y3]] = q;
    const dx1 = x1 - x2, dx2 = x3 - x2, sx = x0 - x1 + x2 - x3;
    const dy1 = y1 - y2, dy2 = y3 - y2, sy = y0 - y1 + y2 - y3;
    const den = dx1 * dy2 - dx2 * dy1;
    const g = (sx * dy2 - dx2 * sy) / den, h = (dx1 * sy - sx * dy1) / den;
    const a = x1 - x0 + g * x1, b = x3 - x0 + h * x3;
    const d = y1 - y0 + g * y1, e = y3 - y0 + h * y3;
    return (u, v) => {
        u /= 8;
        v /= 8;
        const z = g * u + h * v + 1;
        return [(a * u + b * v + x0) / z, (d * u + e * v + y0) / z];
    };
};
```

Hand it four corners and you get back a function that tells you where any point of the board, in board coordinates,
lands in the picture. The division by `z` is the perspective:
squares on the far side of the board come out narrower than the near ones, just as they are in the photo.

That function gets used twice. Once it’s the last step, to draw the board out flat: for every pixel of a 512 by 512 output, ask where in the photo it came from and sample there. But first it’s used to answer the more interesting question of how you find the four corners in the first place.

## Scoring a Guess

Given a guess at the corners, `lay` says where the fourteen interior lines of the board should be.
Step a little to either side of each of those lines and compare the brightness.
On a real board that difference is large everywhere along the line, and it flips sign at every square,
because light-then-dark becomes dark-then-light.
So multiply each difference by the sign a chessboard would have there and add them all up.
A guess that sits on the real grid gets a huge total. One that’s off by a bit loses contrast everywhere at once.
And random page clutter contributes about nothing, because it doesn’t alternate in step with the pattern.

The samples are taken near the ends of each square’s edge and never across the middle, since that’s where the pieces stand.

With a score in hand, the search is a patient hill-climb. Start from the square the comb search found. Try it shifted a whole square in each direction. Then nudge each corner by itself, then each side, then the whole thing, by half a square, a quarter, an eighth, down to about a sixty-fourth, keeping any change that improves the score.

Two things went wrong on the way, and I think both are instructive.

**The same board one square over is nearly as good.**
Shift the whole grid by one square and seven of the eight lines in that direction still land on real lines.
The more skewed photos kept coming out exactly one square off. My first idea,
penalizing the guess whenever the alternating pattern appeared to continue past its edge, made everything *worse*,
because the board’s true edge looks like half of such a line:
there are alternating squares on one side of it and whatever surrounds the board on the other.
What separates the two cases is that a true edge is worth about half a line,
and one square beyond it there is nothing at all. So the edges are credited for up to half a line’s worth,
and anything alternating one square beyond them is counted against.
Boards read correctly from the skewed test set went from 85 of 120 to 115 with that one change.

**It was “improving” boards that were already fine.**
Sampling a sixth of a square to either side of a line can’t place that line any more precisely than about a sixth of a
square. Perfectly straight screenshots were having their corners moved around by a fifth of a square,
and a few were read worse because of it. So the sampling distance now tightens as the steps get smaller,
and if the corners end up within a tenth of a square of being a true upright square,
the whole exercise is thrown away and the original pixels are used untouched. Of a thousand straight test screenshots,
none is resampled and none reads differently than before. I like that property a lot:
the fix for the weird case can’t hurt the normal one.

You can do the corner-finding by hand here. Drag the amber corners and watch what happens to the flattened board, the fit score, and what the network makes of it. It doesn’t take much error before a rook becomes a pawn.

|  | Before | With straightening | 
|---|---|---|
| Synthetic screenshots photographed “from the side”: position wholly right, of 120 | 33 | 116 | 
| Kaggle boards that fill the picture: board located, of 300 | 289 | 299 | 

It costs about 20 milliseconds. The second row was a freebie. When the comb search finds nothing at all, the hill-climb can start cold, trying squares of every size all over the picture, and that rescued most of the boards the comb had been missing.

# The Network

Now, finally, the neural network part. Each of the 64 squares is resized to 32 by 32 pixels and handed to this-

``` python
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.c1 = nn.Conv2d(3, 20, 3, padding=1)
        self.c2 = nn.Conv2d(20, 40, 3, padding=1)
        self.c3 = nn.Conv2d(40, 64, 3, padding=1)
        self.f1, self.f2 = nn.Linear(64 * 4 * 4, 96), nn.Linear(96, 13)
        self.pool, self.drop = nn.MaxPool2d(2), nn.Dropout(0.25)

    def forward(self, x):
        for c in (self.c1, self.c2, self.c3):
            x = self.pool(torch.relu(c(x)))
        return self.f2(self.drop(torch.relu(self.f1(x.flatten(1)))))
```

If you took a deep learning course at any point in the last thirty years, you’ve built this network. Three rounds of “ look at every 3 by 3 neighbourhood, keep what’s positive, halve the resolution by keeping the largest of every four”, then two fully connected layers that make the decision. There’s no batch normalization, there are no residual connections, and there is certainly no attention.

That’s deliberate. Every one of those would be something more to implement by hand in JavaScript and something more to get subtly wrong. A convolution, a ReLU, a max-pool and a matrix multiply are each a short loop. And the task is easy by the standards of the field: the inputs are computer-rendered, more or less centred, and there are thirteen classes. This is MNIST with horses.

Thirty-two pixels is small, but it’s enough. A king and a queen are distinguishable at that size,
which is the hardest call to make<sup>[10](#user-content-fn-queen)</sup>, and at 32 pixels three halvings land on a tidy 4 by 4.

A few notes on the training recipe, none exotic: AdamW with a one-cycle learning rate schedule peaking at 0.004,
a batch size of 256, twenty epochs, dropout of 0.25 before the last layer, and label smoothing of 0.05<sup>[11](#user-content-fn-smoothing)</sup>.
On the MPS backend of a Mac it trains in around fifteen minutes, which made it cheap to try things.
I tried a lot of things.

In the demo at the top, click any of the 64 squares in the middle panel. You’ll see the 32 by 32 input, the network’s top five guesses, and the feature maps after each round. The first round’s maps are still recognizably a picture of a piece, with different maps responding to different edges and colours. By the third round there are 64 maps of 4 by 4 and nothing a person would recognize.

# There Is No Dataset

Here’s the real problem. To train that network I need hundreds of thousands of pictures of single chessboard squares, each labelled with what’s on it, covering every piece set and board style that anyone’s screenshot might contain.

Nobody has that lying around, and I wasn’t going to label it by hand.

But think about what a screenshot of a chess board *is*. It’s a rendering.
Some program took a set of piece images and drew them over coloured squares. I can write that program too.
And if I write it, I know the label of every square for free, because I’m the one who put the piece there.

So the training data is entirely manufactured. The generator picks a piece from one of 41 piece sets ( every set that lichess publishes as SVG), picks a board, draws the square, and then spends the rest of its considerable length trying to ruin the picture in every way a real screenshot might be ruined.

Some of those are genuinely hard to make out, and that’s the point. The term of art is *domain randomization*:
if you can’t make your synthetic data look exactly like the real world, make it vary so wildly, along so many axes,
that the real world is just one more variation. Here’s what it varies, one effect at a time-

A few of these deserve a comment.

**The neighbours’ edges.**
Each square is drawn with a sliver of its neighbours around it and then cropped slightly wrong: off-centre by up to 5%,
a little too big or too small. The board finder is good but it isn’t perfect,
and the network needs to be indifferent to the difference.

**The clutter goes on empty squares more often.** Dots,
rings and arrows are drawn on empty squares at nearly twice the rate they’re drawn on occupied ones.
An occupied square with a dot on it is still obviously a piece.
An empty square with something painted on it is where the network gets tempted to see a piece,
so that’s where the examples need to be.

**The pieces themselves get restyled.** Half the time the piece is made lighter or darker, tinted,
rotated a few degrees, given a heavier outline or a drop shadow. The hope is that the network learns “knight” and not “
this specific knight”.

**Squares come in every size.** A square might be 20 pixels across on the screen or 128,
and it’s resized to 32 by one of four different resampling filters.
Small boards and large boards produce quite different 32 by 32 pictures, and so do different browsers’ scaling.

Thirty percent of the squares are empty. A real board is more like 60% empty or more, but empty squares are the easy ones and I’d sooner spend the network’s attention on pieces.

# Four Networks

This is the part I find most interesting in hindsight, so I’m going to go through it in order.

## The First: Great On Paper

The first network was a third smaller (69,021 parameters) and was trained on fifteen piece sets, on flat-coloured boards. To measure it honestly I held three sets back entirely, so it was tested on piece designs it had never seen. A whole position counts as right only if all 64 squares are right, which is an unforgiving measure: 99% per-square accuracy gets you about half of your boards.

It read 142 of 150 screenshots correctly in the sets it had never seen, and 148 of 150 in the ones it had.

I shipped it. Then I built a harder test: 400 screenshots, half of them on boards with a wood grain or a texture.

| Whole position right, of 200 | The first network | 
|---|---|
| Piece sets held out from training | 89 | 
| Piece sets it trained on | 107 | 

Oof. Nearly every error was on a textured board, where it saw pieces on empty squares. It had learned something closer to “anything that isn’t perfectly flat is a piece” than I would have liked. Which is a completely fair conclusion to draw from the data I’d shown it.

## The Second: More of Everything

The second network is the architecture above, about twice the size, trained on all the variation shown earlier: wood grain, noise, shading, grid lines, the check glow, dragged pieces, video compression. Eight piece sets were held out this time.

| Whole position right, of 200 | The first | The second | The second, plus the rules of chess | 
|---|---|---|---|
| Piece sets held out from training | 89 | 148 | 175 | 
| Piece sets it trained on | 107 | 198 | 199 | 

(I’ll get to that last column.)

Seven of the eight held-out sets came in at 22 to 25 boards right of 25. The eighth, a set called riohacha, managed 7 of 25. Its bishops don’t look like anyone else’s bishops, and the network read them as pawns. All of them, every time, confidently.

That turns out to be *the* characteristic failure on an unfamiliar piece set.
It isn’t random noise sprinkled across the board. It’s one piece type mistaken for another, consistently,
everywhere it appears. Which makes sense: the network has no idea it’s looking at a board.
It answers the same question 64 independent times, so if it’s wrong about a drawing,
it’s wrong about that drawing every time it shows up.

Having measured what I wanted to measure, I retrained on all 41 sets for the version that ships. That one reads 395 of those 400 correctly and is within two squares on the other five, though of course none of the sets are a fair test of it any more.

## The Third: Kaggle

All of this had been tested against my own generator’s idea of what a screenshot looks like, which is a little too convenient. I wanted pictures made by someone else.

Kaggle has a dataset called [Chess Positions](https://www.kaggle.com/datasets/koryakinp/chess-positions):
100,000 images of boards, 80,000 for training and 20,000 for testing, each 400 pixels square,
each with the position encoded right in its filename.
They’re rendered in dozens of board and piece styles that look nothing like my generator’s output. Marble.
Brushed metal. Red pieces on a pale blue board.

The second network read 1,420 of 2,000 of the test boards. Seventy-one percent.

So I cut the Kaggle training boards up into squares (the filename says what’s on each) and mixed them in: 300,000 Kaggle squares beside 1,100,000 of my own. They get lightly augmented too. My favourite trick there is rotating the hue of the whole square by a random amount some of the time, which turns a green board into a purple one and blue pieces into orange ones for free.

And the first result was… worse. Kaggle went to nearly perfect, but my own 400 screenshots dropped from 198 and 197 to 194 and 192.

My reflex was that the network had run out of capacity, having now been asked to learn a second visual world. So I trained a bigger one, 220,997 parameters. It scored 195 and 194 and took half again as long to read a board.

It wasn’t capacity. When I actually looked at the squares it was getting wrong, *every single one*
was an empty square where an arrow started or ended. An arrowhead, or the rounded butt of an arrow,
sitting in the middle of an empty square, read as a faint pawn or knight.
My generator had only ever drawn arrows passing *through* squares, clipping a corner on the way to somewhere else.
A blob of colour in the dead centre of a square was something it had only ever seen when there was a piece there.
Adding the Kaggle data hadn’t caused that. It had shifted the decision boundaries enough to expose it.

I added arrow starts and arrow heads to the generator, and went back to the original size.

|  | The second network | The third | 
|---|---|---|
| My screenshots, first 200: whole position right | 198 | 198 | 
| My screenshots, other 200 | 197 | 198 | 
| Kaggle test boards, of 2,000 | 1,420 | 1,998 | 

## The Fourth: The Bishop That Wasn’t

A day later a phone screenshot of a game review went through: chess.com’s review screen, where the queen had just gone from e2 to e8, and the app had drawn a wide yellow arrow straight up the e-file over the five empty squares in between.

The network reported a black bishop on e4 and another on e6.

Squint at it. A tall, narrow, vertical shape, centred on a light square. If you had never been shown one of those before and had to choose from thirteen options, bishop isn’t a crazy answer. And it had never been shown one. I’d drawn arrows clipping corners, and arrows starting and ending in the middle of a square. I had never drawn one running dead straight through the centre of a square from edge to edge. An arrow along a file, a rank or a long diagonal does exactly that.

So the generator learned to draw those, wider and in a few more colours, and the fourth network is the one on the site today. The screenshot reads correctly. Everything else scored within noise of the third.

## The Pattern

Four networks, and the architecture changed exactly once, early, and modestly. Every other improvement came from looking at specific pictures the network got wrong and working out what it had never been shown.

The one time I reached for a bigger model, it bought nothing. I knew this going in, in the way one knows things one has read. It lands differently when it’s your own 220,997 parameters sitting there doing no better than 130,565.

The other lesson is how misleading a comfortable test set is. 148 of 150 was a true number. It was also a measurement of how well the network handled the things I’d happened to think of.

# What the Network Doesn’t Decide

A fair amount of the accuracy comes from code that sits around the network and overrules it.

## The Rules of Chess

The network reads each square in isolation. But the 64 answers aren’t independent: together they have to be a chess position, and a chess position has exactly one king a side and never has a pawn on the first or last rank.

So after the network has had its say-

- If there are two white kings, the one it was less sure about becomes whatever its second choice was. Almost always a queen.
- If there’s no white king at all, the white piece that looks most like a king gets promoted.
- A pawn on the first or eighth rank becomes its next-best guess.

That’s the entire thing, and it took the held-out sets from 148 boards right to 175. It works because it uses information the network threw away. The network emits a probability for all thirteen classes, and when it mistakes a king for a queen, king is very often sitting right there in second place. A hard constraint is enough to pick it back up.

Any square that the rules had to overrule is flagged as uncertain, as is anything the network gave less than 70% to, and the editor highlights those squares for the user to check. In the demo they’re the blue and amber outlines.

## The Thing I Was Sure Would Work

Every knight on a given board is the same drawing. So you ought to be able to read a board *as a board*:
work out which squares hold identical drawings, and then even if the network is unsure about one of them,
its certainty about the others carries over. For a piece set the network has never seen, that should be a big win.

I tried it three ways. Comparing squares by the network’s own internal features. Comparing them by the piece’s silhouette, extracted by subtracting the square’s background colour. Clustering the silhouettes into groups first and then assigning each group one piece type.

The first two made things worse and the third was a wash.
The network’s internal features turn out to encode nearly as much about the *square* (its colour, its grain,
whether it’s highlighted) as about the piece on it. And the silhouette approach has a nasty failure mode:
when the network is wrong about a piece type throughout a set, it’s *confidently* wrong,
so propagating its confident answers to similar-looking squares just spreads the mistake to the few it had got right.

I’m leaving that in here because the idea still seems right to me, and I suspect the correct version is a second small network trained to answer “are these two squares the same drawing?”. But three attempts was enough for one weekend, and the page doesn’t do it.

## Plain Squares Are Empty

A screenshot of a board in a deep red, shaded so each square was a gradient, came back with 21 phantom pieces on its empty squares. The colours were simply outside anything in the training data.

Later networks were trained on such boards. But there’s also now a check that happens before the network is even believed. Take the middle half of the square and fit a plane to it, in each colour channel. If no pixel deviates from that plane by more than 8 levels out of 255, the square is a flat colour or a smooth gradient, and it’s empty no matter what the network thinks. On the test screenshots an empty square of that kind never deviates by more than 4, and a square with a piece on it never by less than 26, so there’s a comfortable gap to put the line in.

A dozen lines of arithmetic, and it makes an entire category of failure impossible instead of merely unlikely.
I’m a big fan of that trade wherever it’s on offer<sup>[12](#user-content-fn-belt)</sup>.

# Shipping 130,565 Numbers

PyTorch’s weights are 32-bit floats: 522KB. Before they go to the browser, each tensor is quantized to signed bytes with a single scale factor.

```
for name in ("c1", "c2", "c3", "f1", "f2"):
    for part in ("weight", "bias"):
        a = w[f"{name}.{part}"].astype(np.float32).ravel()
        scale = float(np.abs(a).max()) / 127 or 1.0
        out += struct.pack("<f", scale)
        out += np.clip(np.round(a / scale), -127, 127).astype(np.int8).tobytes()
```

Find the largest magnitude in the tensor, scale so that it maps to 127, round everything to the nearest integer. The file is ten tensors, each a four-byte float followed by its bytes: 130,605 bytes, or about 86KB once brotli has had a go at it. There’s no header and no format. The page knows the shapes.

I should be precise about what this is, because “int8 quantization” usually implies more. The *arithmetic*
isn’t quantized.
The page multiplies each byte by its scale when it loads the file and from then on everything is ordinary floating
point<sup>[13](#user-content-fn-floats)</sup>. The bytes are purely a way to make the download a quarter of the size.
Rounding each weight to one of 255 levels made no difference to accuracy that I could measure,
and I didn’t have to do anything clever like quantization-aware training to get there.

And here’s the inference engine, more or less in full-

```
// a round of looking (3 by 3, zero-padded), keeping what's positive, and halving by the greatest of each four
function conv(input, size, layer) {
    const [outs, ins] = layer.shape, half = size / 2;
    const out = new Float32Array(outs * half * half), row = new Float32Array(size * size);
    for (let o = 0; o < outs; o++) {
        row.fill(layer.b[o]);
        for (let i = 0; i < ins; i++) {
            const from = i * size * size, w = (o * ins + i) * 9;
            for (let ky = 0; ky < 3; ky++) for (let kx = 0; kx < 3; kx++) {
                const k = layer.w[w + ky * 3 + kx];
                if (!k) continue;
                const y0 = Math.max(0, 1 - ky), y1 = Math.min(size, size + 1 - ky);
                const x0 = Math.max(0, 1 - kx), x1 = Math.min(size, size + 1 - kx);
                for (let y = y0; y < y1; y++) {
                    const a = y * size, b = from + (y + ky - 1) * size + kx - 1;
                    for (let x = x0; x < x1; x++) row[a + x] += k * input[b + x];
                }
            }
        }
        for (let y = 0; y < half; y++) for (let x = 0; x < half; x++) {
            const a = 2 * y * size + 2 * x;
            out[o * half * half + y * half + x] = Math.max(0, row[a], row[a + 1], row[a + size], row[a + size + 1]);
        }
    }
    return out;
}
```

That one function is the convolution, the ReLU and the max-pool together. There’s a `dense` that’s four lines,
and a softmax. That’s it. No WebGL, no WebAssembly, no SIMD, no workers.

Is it fast? It’s about four million multiply-adds per square, so a quarter of a billion per board,
and it takes roughly 240 milliseconds on my laptop<sup>[14](#user-content-fn-timing)</sup>.
An optimized runtime would be ten or fifty times faster and I don’t care even slightly.
A person has just pasted a picture and is waiting to see a board appear.
A quarter of a second is indistinguishable from instant,
and the whole feature is a 131KB file and a script that aren’t even fetched until someone first hands the page a
picture.

I spend a lot of my professional life around people reaching for the heaviest tool available<sup>[15](#user-content-fn-heavy)</sup>. It’s nice,
once in a while, to ship the lightest one.

# Living In A Browser

Having just been smug about how little I needed, I should be straight about what the browser takes away, and what I’d do about each if a quarter of a second ever stopped being good enough.

## One Thread, And It’s The Important One

A page’s JavaScript runs on the same thread that paints the page and answers clicks.
Those 240 milliseconds of loops are 240 milliseconds in which nothing else happens at all. Scrolling stops.
A spinner wouldn’t spin. Even the word “Reading…” won’t appear unless you set the text,
hand control back to the browser so it can paint, and only then start the work<sup>[16](#user-content-fn-yield)</sup>.

For one screenshot that’s tolerable. For a video, where the network runs thousands of times, it isn’t, so that path does its reading in short bursts and yields in between.

The proper answer is a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API):
a real second thread, with the catch that it shares nothing with the page. No DOM, no shared variables, only messages.
That model happens to suit this problem perfectly. The 64 squares are independent of one another,
a square is 4,096 bytes, and a typed array’s buffer can be *transferred* to a worker instead of copied,
which costs nothing.

I haven’t shipped this, but I did measure it, with the same loops and the same weights, splitting the 64 squares evenly across a pool-

|  | Time to read 64 squares | Page frozen for | 
|---|---|---|
| Main thread, as shipped | 237 ms | 237 ms | 
| 1 worker | 237 ms | nothing | 
| 2 workers | 120 ms | nothing | 
| 4 workers | 65 ms | nothing | 
| 8 workers | 50 ms | nothing | 

The first row to the second is the one that matters: one worker is no faster, but the page stays alive.
After that it scales almost perfectly to four, and then flattens,
because this machine has four fast cores and the rest are efficiency cores<sup>[17](#user-content-fn-cores)</sup>.
Starting a worker and handing it its own copy of the weights took two or three milliseconds each, which surprised me.
I’d assumed it would eat the gain.

That phrase, *its own copy*, is the other catch. Each worker gets a private half-megabyte of weights.
To genuinely share memory between threads you need a `SharedArrayBuffer`,
and since Spectre browsers only hand those out to pages that
are [cross-origin isolated](https://developer.mozilla.org/en-US/docs/Web/API/Window/crossOriginIsolated):
two response headers that, once set, require everything the page embeds from elsewhere to explicitly opt in.
Payment widgets, fonts, embedded videos, the lot. For 522KB a copy it’s nowhere near worth the trouble.
For a model of a few hundred megabytes it’s the price of admission.

## No SIMD, No GPU, Unless You Ask Nicely

That inner loop does one multiply and one add per iteration.
The CPU underneath could do four or eight or sixteen at once, and JavaScript has no way to ask for that<sup>[18](#user-content-fn-simdjs)</sup>.

WebAssembly does, with 128-bit SIMD in every current browser,
and the same convolution compiled from C or Rust would probably run several times faster on one thread.
WebAssembly threads exist too, though they’re built on `SharedArrayBuffer`
and so drag in the cross-origin isolation business above. Beyond that there’s WebGPU,
which is how people run actual language models in a tab.

Those are, not coincidentally, exactly what ONNX Runtime Web and its peers are made of.
Which is the honest version of my boast earlier: I didn’t avoid needing that machinery by being clever.
I avoided it by having a problem small enough not to need it. For a network this size a GPU would likely be *slower*,
because compiling shaders and reading results back off the card costs more than the arithmetic.

## The Canvas Is Not A Bitmap Library

Everything the reader knows about the picture comes through a `<canvas>`: draw the image, read the pixels back.
That has more opinions than you’d expect.

**The browser may keep your canvas on the GPU.** Then every read-back is a round trip to the graphics card,
and the reader does hundreds of them. The `willReadFrequently`
hint when creating the context tells the browser to keep it in ordinary memory,
and it is the difference between fast and inexplicably slow.

**Every browser resizes differently.** Shrinking a 90 pixel square to 32 is done by the browser’s own resampling,
and the quality hint the reader sets is one that not every browser honours.
So the same screenshot becomes slightly different 32 by 32 inputs in Chrome, Safari and Firefox.
This is why the training generator resizes each square with one of four different filters chosen at random:
I can’t control which one a browser uses, so the network has seen them all. The arithmetic of the network,
on the other hand, is plain IEEE floating point and comes out identical everywhere.

**You can’t read pixels you didn’t get honestly.**
Draw an image from another site onto a canvas and the canvas becomes “tainted”: you can look at it,
but any attempt to read it back throws. This is why the feature takes a pasted,
dropped or chosen file and doesn’t offer “give me the URL of a picture”.

**Some browsers lie about the pixels on purpose.** Reading a canvas back is a classic way to fingerprint a browser,
so the privacy-hardened ones (Brave, Firefox with its resist-fingerprinting mode) add a little noise to what comes back.
I haven’t tested how the reader holds up under that.
A network trained on JPEG artefacts and video compression ought to shrug it off, but the “
is this square a perfectly plain colour” check from earlier, with its threshold of 8 levels,
is precisely the sort of thing a bit of deliberate noise could upset.

**Canvases have a maximum size**, and it’s smallest on phones, where asking for more than about sixteen million pixels gets you an empty canvas and no error.
A screenshot never comes close. A 48 megapixel photograph of a laptop screen sails past it.
Writing this entry is how I noticed the reader had that hole in it, and the fix was a few lines: the decoded picture is a bitmap, which has no such limit, so anything over twelve million pixels is drawn onto a proportionally smaller canvas.
Nothing is lost, since the board is looked for at 480 pixels across and its squares are read at 32.

## Video Is Worse

A `<video>` element is built to play, not to be interrogated. You can’t ask it for frame 412. You set `currentTime`,
wait for an event, draw whatever it’s showing, and hope that’s a new frame and not the one you already had.
The reader leans on `requestVideoFrameCallback` to find out which frame it actually got,
which is what stopped it counting a sliding pawn twice.
And whether a given file plays at all depends on which codecs that browser was built with.

The grown-up answer there is [WebCodecs](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API),
which hands you decoded frames one at a time, in order, as fast as you can take them, in a worker if you like.
It also means demuxing the container yourself,
and that was more than I wanted to take on for a feature I wasn’t yet sure anyone would use.

One more for the pile: timers in a background tab get throttled to about once a second. A loop that politely yields between bursts will crawl if the user switches tabs to go find the next video. A worker doesn’t have that problem either.

# What a Picture Doesn’t Say

A position is more than where the pieces are. You also need to know which way up the board is and whose move it is, and a picture is cagey about both. A few more bits of non-neural trickery pull out what they can.

**Which way up.** If the board has coordinates on it, they’ll tell you,
but I didn’t want to build an OCR engine to read eight digits. So the page doesn’t read them, it weighs them. A “1”
has far less ink in it than an “8”.
Find the little glyph in the corner of the top-left square and the one in the bottom-left square and compare how much
ink each has: if the top one is the heavy one, that’s the 8, and you’re looking from White’s side. Likewise the letters:
a, c and e are short, while b, d, f, g and h are tall.
Measure the heights along the bottom rank and see whether the short-tall pattern runs forwards or backwards.
When there are no coordinates to weigh, it falls back to assuming White’s pieces are mostly at White’s end of the board,
which is right far more often than not.

**Whose move.** Most sites highlight the square a piece just left and the one it landed on. So:
find the squares whose background colour is far from the other squares of their own shade. If exactly two are lit,
and one is empty and the other has a piece on it, that’s the last move, and the other side is to play.
If it’s anything else, nothing is claimed. On a test of 155 screenshots with a lit move,
it found 140 and got none wrong, and it never invented one that wasn’t there. Staying silent is fine.
Being wrong would be worse than useless.

**Unless someone’s in check.** A screenshot taken just after a queen delivered check, with nothing highlighted,
got set up as White to move with Black’s king hanging. That’s not a position that can exist,
and Stockfish rightly refused to look at it. So the reader now works out for itself whether either king is attacked,
and a side in check is the side to move, whatever the highlights say.

# Odds and Ends

What it can’t do, or what I haven’t tested well enough to claim-

- Photographs of physical boards. Different problem, as above.
- Chess.com’s own piece sets aren’t in the training data, since they aren’t mine to render. It has read them well in my own use, but that’s an anecdote and not a count. Kaggle’s set covers some lookalikes.
- An unusual piece set it’s never seen can still lose a whole piece type, riohacha-style. The rules catch some of that. The highlighted “not sure” squares catch more. The user checking the board before using it catches the rest, and the editor makes that the obvious next step.

Once a single picture could be read reliably, the obvious next thing was a lot of pictures in a row.
The same reader now takes a GIF or a screen recording of a game and reconstructs the moves,
by reading the board in every frame and finding the legal move that connects each position to the next<sup>[19](#user-content-fn-video)</sup>.
That had its own entertaining set of problems, such as a pawn that’s halfway through sliding to e4 being read,
quite reasonably, as standing on e3. It probably deserves its own entry.

If you want to try the real thing, it’s the [board editor at Chess Report](https://chessreport.ca/board).
Paste a screenshot.

## Footnotes

1. 
At least as of the publishing of this page, or if I get around to updating it. As I discover new edge cases I train new micromodels, benchmark them against what’s existing, and if they win the bake-off they replace the existing model. Several iterations have already occurred. [↩](#user-content-fnref-dated)
2. 
Usually accompanied by the unstated “…because I played something else and lost”. [↩](#user-content-fnref-chat)
3. 
As with the rest of that project, I leaned heavily on an LLM coding assistant through this (Claude, in this case), directing it, auditing what it produced and deciding what to try next. The pace in this entry, four networks and three abandoned ideas in a few days, is not a pace I’d have managed alone on a side project, and it’s the reason I bothered with the dead ends at all. It also means that writing this entry was partly an exercise in making sure I understand every piece of it, which I’d recommend to anyone working this way. [↩](#user-content-fnref-llm)
4. 
It’s a side project that’s doing well when it covers its own domain registration and the absolute minimum of compute. Everything is done to minify operating costs. [↩](#user-content-fnref-cheap)
5. 
People do work on it. The published approaches generally involve finding the board’s corners with one model, estimating the camera pose, and then classifying each square from a crop tall enough to include a piece that’s leaning over its neighbours in perspective. It’s a real research problem, where this is a weekend. [↩](#user-content-fnref-real)
6. 
Piece sets are artwork, and their licences are all over the map: some public domain, some copyleft, some non-commercial. The network was trained on renders of all of them, and none of the artwork is in this repository, on the site, or recoverable from 130,565 weights. Whether training on them is fine was my call to make and I’m comfortable with it. *Publishing* pictures of them in a blog post is a different question,
so the figures here use only the two sets the site itself ships.[↩](#user-content-fnref-licences)
7. 
OpenCV has had `findChessboardCorners` forever, but that’s for camera calibration targets: an empty,
high-contrast board with nothing on it. Put 32 pieces on the board, add a wood grain and a yellow highlight,
and it’s less happy. It’s also a multi-megabyte dependency I’d have had to compile to WebAssembly,
which rather defeats the purpose.[↩](#user-content-fnref-opencv)
8. 
Four points, inset about an eighth of a square from each corner, and the middle two of the four values are averaged. So one corner can be covered by a coordinate label, or by the ear of an ambitious knight, without it mattering. [↩](#user-content-fnref-corners)
9. 
I can’t overstate how much I value this. When the neural network gets something wrong, the diagnosis is a research project. When forty lines of arithmetic get something wrong, it’s much easier to reason with. [↩](#user-content-fnref-debug)
10. 
In most sets the queen is a crown with points and the king is a crown with a cross on top, and at 32 pixels the cross is about three pixels of information. King-for-queen is the network’s single most common error. [↩](#user-content-fnref-queen)
11. 
Label smoothing means telling the network the right answer is “95% knight” instead of “100% knight”. It discourages the absurd overconfidence these things develop, which matters here because the confidences get *used* afterwards: by the rules, and to decide which squares to flag for the user.[↩](#user-content-fnref-smoothing)
12. 
This is the same instinct as putting a `CHECK` constraint on a database column even though the application “
always” validates the value. The application doesn’t always.[↩](#user-content-fnref-belt)
13. 
Every one of which is, this being JavaScript, either a 32-bit float in a typed array or a 64-bit double in a local variable. I have [opinions about floats](https://dennisforbes.ca/blog/features/floating_point/understanding-floating-point-numbers) <sup>↗</sup> .[↩](#user-content-fnref-floats)
14. 
An M4 Mac, in Chrome. Finding the board is about 35ms and the straightening check about 20ms on top. Phones are slower, though not enough that it’s felt like waiting on any I’ve tried. [↩](#user-content-fnref-timing)
15. 
A Kubernetes cluster to serve a brochure site. A vector database for four hundred documents. You know the ones. [↩](#user-content-fnref-heavy)
16. 
The demos in this page do exactly that: set the status text, `await` a zero-length timeout, then start.
Skip the middle step and the status changes from nothing straight to the answer, a quarter of a second late.[↩](#user-content-fnref-yield)
17. 
`navigator.hardwareConcurrency` reports 10 on this machine and has no way of telling you that six of them are the slow ones.
Sizing a pool by that number is a guess. Half of it is usually a better guess.[↩](#user-content-fnref-cores)
18. 
There was a proposal to add SIMD types to JavaScript itself. It was abandoned in favour of doing it in WebAssembly, which was the right call and is no help at all when you’ve decided not to use WebAssembly. [↩](#user-content-fnref-simdjs)
19. 
Which, yes, means running the network on every frame of a video in JavaScript. A square that hasn’t changed since the previous frame isn’t read again, and on most frames of a chess video that’s nearly all of them. [↩](#user-content-fnref-video)

## See Also

- [Improving My Chess Abilities](https://dennisforbes.ca/blog/microblog/2026/10/chess_analysis_service)Sharing what I built to get better at the game
- [New Chess Features](https://dennisforbes.ca/blog/microblog/2026/11/new_chess_features)New games, lessons, and analysis functionality
- [Understanding Floating-Point Numbers](https://dennisforbes.ca/blog/features/floating_point/understanding-floating-point-numbers)Floating-Point Numbers are the foundation of the ML world, and understanding their operation is essential
