An ESP32-S3 is a two-core microcontroller you can buy for a few dollars. It runs a Neural Amp Modeler A2-Lite capture in single-precision float, live, with a guitar going in through a USB interface — and the samples it produces are bit-identical to the same capture rendered on a desktop. Not close. All 720,000 samples of the test render match, both files have the same hash, and subtracting one from the other gives digital zero.
It is also a whole pedal now: a 480×480 touch screen with seven effects around the amp, all of it on the same two cores.
As far as we can tell this is the first time a NAM A2 model has run on this chip at all. The version that ships is the floating-point one, which is not what we set out to build.
The obvious plan was to stop using floats #
The S3 has 128-bit vector instructions and they are excellent — at integers. The chip has a floating-point unit too, but no vector path into it, so float goes one multiply at a time while integers go sixteen.
So the obvious move is to stop using floats and run the network in 16-bit integers instead. That worked: about 20% faster than a well-optimised float build, comfortably inside two cores, and it sounded fine.
It also sounded slightly hissy in the gaps between notes.
The hiss was not ours #
The test material is fifteen seconds of a dry guitar riff, with 28 dB between its peak and its average level.
That DI is never digitally silent. In its quietest passages it sits around −67 dBFS, which is the room and the pickups of the original take. Then put a 5150 with an overdrive in front of it — about 42 dB of gain — and that noise comes up with everything else.
Measured across exactly those quiet windows, the integer engine, the float three-channel engine and the full eight-channel float engine all put out the same level, within 0.7 dB of each other, all carrying the same tiny DC offset, which is the model's own output bias. The hiss was in the recording, and the amp was doing what an amp does to it.
We also swept a proper gate in front of the amp to see whether gating would help, and it does not: the best it ever bought was under a decibel, and past that it started eating the playing. Quantisation error does not live in silence. It lives in mid-level sustain and decay — inside the notes — where no threshold can reach it. A gate is still worth having for the amplified room noise. It is not a fix for arithmetic.
What actually fixed the integer engine #
Three changes, and the order they happened in matters, because two of them look worthless on their own.
Every quantisation scale had been rounded down to a power of two and padded, which throws away most of a bit per layer. Only one of those shifts is a hardware shift and actually has to be a power of two; the rest are ordinary multiplies and can take their exact values. On its own this moved the measurement by 0.2 dB. It matters only because it stops wasting the range the next change needs.
Then each layer got its own exponent, so the integer grid slides up when the signal drops instead of sitting frozen at whatever the loudest possible passage would require. This is where the 6 dB came from, after two failures worth recording:
Driving that exponent from the loudest sample in the current block made it dramatically worse — and the reason is a nice trap. The convolution does not read the block. It reads up to 1,200 samples of history. A quiet block right after a loud one has a tiny peak and an enormous convolution result, so scaling up on the block's peak guarantees you clip. The exponent has to follow a sliding peak over the history the layer will actually read, with the current block only ever allowed to pull it down — the direction that cannot overflow.
And without a retreat rule, a layer can park at an exponent that clips on every single block with nothing to push it back. Adding "if the output gets within 15% of full scale, drop a step" took the clipping count from 188,508 to a few thousand, and tightening that ceiling took it the rest of the way to zero.
Finally the input moved up one bit. A guitar DI never exceeds full scale, so half the input range was reserved for headroom that never gets used. The output deliberately did not move, because an amp's output does pass unity on transients, and clipping those is far worse than a bit of noise floor.
That last one was the biggest single jump of the three, and only because the other
two came first: while the layers were still noisy, a wasted bit on the input was
buried underneath them. Together the three took the quantisation error 26 dB down,
for about 10% more CPU — and the version that made the noise measurement worse
along the way was a log2
call, which cost 13 points of CPU to produce bit-identical output. An integer count-leading-zeros does the same job for free.
Where the integer engine landed, and why it stopped mattering #
At the end of all that, the integer engine was exactly as far from the full eight-channel model as the float three-channel engine is. The conversion to integers had stopped contributing anything measurable; the entire remaining difference was the small model standing in for the big one, which is a modelling choice and not an arithmetic one.
One caveat about that kind of measurement, because it is easy to over-read. A 17 dB null sounds catastrophic and is not: it means the two waveforms are 99% correlated, and turning one of them up by a single decibel would produce a residual the same size. Null depth measures waveform difference, not audibility, and for distorted signals the two diverge badly. It is the right tool for catching a bug and the wrong tool for predicting what you will hear.
So listen to it instead
Play the eight-channel render and then the integer one; they are hard to tell apart. Then open the differences. The first two are the model gap, and they are recognisably a guitar — fizz, pick attack, the top end a smaller network cannot reproduce. The third is the entire quantisation error, played at its true level like everything else here, and there is nothing to hear: it sits some 49 dB below the program material.
And then float won anyway #
The integer engine was good. Then we went back to the float engine, because the number it had lost to was from a generic optimised build and nobody had hand-written its inner loop.
A leaf kernel now evaluates several consecutive convolutions at once, sharing every weight load between them, with separate accumulation chains so the frames never wait on each other. It ended up about 30% faster than the generic float build — and slightly faster than the integer engine it was competing with, in real single-precision float, with no exponents to manage, no scales to tune, and no quantisation error at all.
It also agrees bit-for-bit with the plain C++ version of itself, over random weights and both kernel sizes. Everything below depends on that.
Two things about it caught us out.
It has to live in instruction RAM. The 314-byte hot loop, left in external flash-backed memory where the display code also runs, put the model in contention with the panel and cost fourteen points of CPU. Fourteen points, for moving 314 bytes.
The engine used to keep a second copy of its convolution history so a read could never wrap around the end of the buffer. Wrapping the pointer properly instead costs 3% more CPU and returns 77 KB of internal RAM — and on this chip that is not a memory-efficiency nicety. Internal RAM is what the display's DMA buffer and the USB endpoints are competing for, and when they lose, USB simply fails to start. A 3% slower engine that leaves room for the peripherals is the faster pedal.
Bit exact, proven on the board #
There is a way to check whether the chip computes the same values as a desktop that takes almost everything else out of the question. A diagnostic image flashes the exact input file and the model onto the board, runs all 720,000 samples through the same engine and the same assembly the live firmware uses, writes the raw output back to flash, and reads it out. No converters, no analog loop, no interface clock — so no converter noise, gain uncertainty or cable in the measurement.
The first comparison against the host came back at −108 dB, and that number turned out to be about the reference rather than about the chip.
The board writes its result back as raw 32-bit float. The host's render was a 24-bit integer WAV — the correct format for something you intend to listen to, and one that rounds every sample onto a 24-bit grid on the way in. Decoding it back to float cannot recover what was discarded. So the comparison was float against decoded-24-bit, and −108 dB was the cost of that round trip through a listening format. The S3 could have been perfect and the number would not have budged.
It was. Once the host wrote raw float too, all 720,000 samples matched, both files hashed identically, and the difference was digital zero at unity gain and gain-matched. Both render at −15.7 LUFS and −2.6 dBTP.
The general form of that is the most portable thing in this post: your reference is part of the measurement. A null can only be as deep as the weaker of the two files feeding it, and a perfectly good listening format is a lossy step in the chain.
The model adds a third of a millisecond. The interface adds the rest. #
The inference costs 0.333 ms — one 16-sample block at 48 kHz — and that is the entire algorithmic latency the amp model contributes.
The round trip is about 4.7 ms, and the other 4.4 of it is USB. The iRig HD 2 is a full-speed device, which means the host may talk to it once per millisecond, and 48 kHz for one millisecond is exactly 48 audio frames. That single fact sets the floor for everything downstream, and no amount of firmware changes it: the packet size is written into the interface's own descriptor.
What firmware could change was the software stacked on top of that floor, and there was a lot of it — one 48-frame packet per transfer instead of four, a trimmed startup backlog, and two queued playback transfers rather than one. Between them they removed something like 13 ms of pure buffering. The one that surprised me was the playback queue: with a single transfer in flight the host stack cannot recycle it before the next USB frame is due, and playback collapses to a quarter speed while capture carries on happily at 48 kHz. Two alternating transfers restore it completely.
Which is why the engine is around 7% of the round trip on this chip, and why the P4 sits somewhere else entirely — high-speed USB is scheduled eight times as often, so the same work carries far less transport around it.
And then seven effects moved in #
The amp model is the expensive part but it was never the whole pedal. The touch build runs eight blocks, the amp among them — gate, compressor, two modulation slots, drive, delay and reverb — on the same two cores, at the same 16-sample block.
Every block opens into an editor.
A chain is serial, so it cannot be split by sample. It is split by stage instead: core 0 runs the pre-effects and the first third of the network, core 1 runs the rest of the network and the head, core 0 takes the finished block for the second modulation slot and the delay, and core 1 produces the stereo reverb and packs the output. Pairs of stages share tasks, so the fourth stage needed neither another DSP stack nor two threads inside one effect.
And then the hand-off between them had two slots, which is not enough — with one slot being filled and one being worked on there is never a free one to start the next block into, so the stages serialise instead of overlapping. That looks like healthy load figures, every deadline met, and about 38,000 dropped audio frames every five seconds. Four slots fixed it, for 2,320 bytes. The same trap caught us on the P4, on entirely different hardware.
With all of that running live, the shipping preset sits at about two thirds of one core and 92% of the other. Forcing every effect on at once — including two modulation instances, which no sane preset does — takes it to the high eighties and low nineties, still with no missed deadlines, no dropped frames and no USB errors. That configuration exists so the scheduler gets measured against something worse than the product.
The reverb had to be written twice
The first compact plate was a four-line feedback network with two input allpasses and loops between 6 and 15 milliseconds. It met its deadline comfortably and it sounded metallic: too few recurrences, too short, too regular.
What replaced it is a half-rate Dattorro-style plate — four input diffusers into two cross-coupled tanks, each tank a modulated allpass, a long delay, a damping filter and then another of each, with four decorrelated output taps to hide the physical loop periods. It runs at 24 kHz, and the longer eight-line tails behind it at 12 kHz, because that is what fits.
Room, plate, hall and ambient are also no longer one tank with four sets of constants. Room is a sparse early-reflection pattern into a short unmodulated network; hall is a wider early field into a modulated one with separate decay rates for low and high frequencies; ambient uses heavier input diffusion and the longest tail. Spring maps to the plate, because the real dispersion model measured about 125 µs per block — more than a third of the entire block budget, for one effect.
The mix control was wrong in a way only playing through it found. It was an ordinary crossfade, so even a modest amount of reverb audibly pulled the direct guitar down. It is now a send: the dry signal stays at exactly unity until halfway up the control.
And like the convolution history before it, the reverb's long delay lines had to move out to external RAM with only its hot core kept internal. On this chip the scarce resource is not cycles.
Traps, recorded so they cost someone else less #
The full eight-channel processor silently does nothing when you hand it more frames than it was allocated for. No error, no partial output — the buffer comes back untouched. Our very first render "worked" and was just the input. Autocorrelation pitch detection octave-errors on decaying plucked notes and will confidently report every guitar as a bass. YIN is what you want. Guard every null with a lag probe first. All of these were confirmed time-aligned before being trusted, so a null is a tone difference rather than a latency artifact. A null cannot be deeper than its reference file. See −108 dB above.
So was the integer work wasted? #
On this chip, in the end, yes. The float engine is faster and exact, and it is what ships on the S3.
Except that the quantiser turned out to be the only way to run the big model on the other chip. The ESP32-P4's vector unit is fourteen times faster than its floating-point unit, and its 410 instructions are all integer, and eight-channel A2-Full in float needs more instructions per sample than both of its cores have cycles. In integers, on that vector unit, the whole eight-channel model fits and runs the pedal.
That is its own post, the one before this. Block floating point, the sliding peak over the history, the retreat rule and the rounding fix all carried over directly, and then it needed one more idea on top of them.