{"slug": "running-nam-a2-lite-natively-on-an-esp32-s3", "title": "Running Nam A2-Lite Natively on an ESP32-S3", "summary": "An ESP32-S3 microcontroller now runs a Neural Amp Modeler A2-Lite capture natively in single-precision float, producing bit-identical output to a desktop render, with all 720,000 samples matching by hash and digital subtraction. The developer achieved this by optimizing the integer engine with per-layer exponents and exact quantisation scales, fixing hiss that was traced to the original recording, not arithmetic. This marks the first time a NAM A2 model has run on this chip.", "body_md": "# Running NAM A2-Lite natively on an ESP32-S3\n\nAn ESP32-S3 is a two-core microcontroller you can buy for a few dollars. It runs\na Neural Amp Modeler A2-Lite capture in single-precision float, live, with a\nguitar going in through a USB interface — and the samples it produces are\nbit-identical to the same capture rendered on a desktop. Not close. All 720,000\nsamples of the test render match, both files have the same hash, and subtracting\none from the other gives digital zero.\n\nIt is also a whole pedal now: a 480×480 touch screen with seven effects around\nthe amp, all of it on the same two cores.\n\nAs far as we can tell this is the first time a NAM A2 model has run on this chip\nat all. The version that ships is the floating-point one, which is not what we\nset out to build.\n\n## The obvious plan was to stop using floats\n\nThe S3 has 128-bit vector instructions and they are excellent — at integers. The\nchip has a floating-point unit too, but no vector path into it, so float goes one\nmultiply at a time while integers go sixteen.\n\nSo the obvious move is to stop using floats and run the network in 16-bit\nintegers instead. That worked: about 20% faster than a well-optimised float\nbuild, comfortably inside two cores, and it sounded fine.\n\nIt also sounded slightly hissy in the gaps between notes.\n\n## The hiss was not ours\n\nThe test material is fifteen seconds of a dry guitar riff, with 28 dB between its\npeak and its average level.\n\nThat DI is never digitally silent. In its quietest passages it sits around −67\ndBFS, which is the room and the pickups of the original take. Then put a 5150\nwith an overdrive in front of it — about 42 dB of gain — and that noise comes up\nwith everything else.\n\nMeasured across exactly those quiet windows, the integer engine, the float\nthree-channel engine and the full eight-channel float engine all put out the same\nlevel, within 0.7 dB of each other, all carrying the same tiny DC offset, which is\nthe model's own output bias. The hiss was in the recording, and the amp was doing\nwhat an amp does to it.\n\nWe also swept a proper gate in front of the amp to see whether gating would help,\nand it does not: the best it ever bought was under a decibel, and past that it\nstarted eating the playing. Quantisation error does not live in silence. It lives\nin mid-level sustain and decay — inside the notes — where no threshold can reach\nit. A gate is still worth having for the amplified room noise. It is not a fix for\narithmetic.\n\n## What actually fixed the integer engine\n\nThree changes, and the order they happened in matters, because two of them look\nworthless on their own.\n\n**Every quantisation scale had been rounded down to a power of two and padded**,\nwhich throws away most of a bit per layer. Only one of those shifts is a hardware\nshift and actually has to be a power of two; the rest are ordinary multiplies and\ncan take their exact values. On its own this moved the measurement by 0.2 dB. It\nmatters only because it stops wasting the range the next change needs.\n\n**Then each layer got its own exponent**, so the integer grid slides up when the\nsignal drops instead of sitting frozen at whatever the loudest possible passage\nwould require. This is where the 6 dB came from, after two failures worth\nrecording:\n\nDriving that exponent from the loudest sample in the current block made it\ndramatically *worse* — and the reason is a nice trap. The convolution does not\nread the block. It reads up to 1,200 samples of history. A quiet block right after\na loud one has a tiny peak and an enormous convolution result, so scaling up on\nthe block's peak guarantees you clip. The exponent has to follow a sliding peak\nover the history the layer will actually read, with the current block only ever\nallowed to pull it *down* — the direction that cannot overflow.\n\nAnd without a retreat rule, a layer can park at an exponent that clips on every\nsingle block with nothing to push it back. Adding \"if the output gets within 15%\nof full scale, drop a step\" took the clipping count from 188,508 to a few\nthousand, and tightening that ceiling took it the rest of the way to zero.\n\n**Finally the input moved up one bit.** A guitar DI never exceeds full scale, so\nhalf the input range was reserved for headroom that never gets used. The output\ndeliberately did *not* move, because an amp's output does pass unity on\ntransients, and clipping those is far worse than a bit of noise floor.\n\nThat last one was the biggest single jump of the three, and only because the other\ntwo came first: while the layers were still noisy, a wasted bit on the input was\nburied underneath them. Together the three took the quantisation error 26 dB down,\nfor about 10% more CPU — and the version that made the noise measurement worse\nalong the way was a `log2`\n\ncall, which cost 13 points of CPU to produce\nbit-identical output. An integer count-leading-zeros does the same job for free.\n\n## Where the integer engine landed, and why it stopped mattering\n\nAt the end of all that, the integer engine was exactly as far from the full\neight-channel model as the *float* three-channel engine is. The conversion to\nintegers had stopped contributing anything measurable; the entire remaining\ndifference was the small model standing in for the big one, which is a modelling\nchoice and not an arithmetic one.\n\nOne caveat about that kind of measurement, because it is easy to over-read. A\n17 dB null sounds catastrophic and is not: it means the two waveforms are 99%\ncorrelated, and turning one of them up by a single decibel would produce a\nresidual the same size. Null depth measures waveform difference, not audibility,\nand for distorted signals the two diverge badly. It is the right tool for catching\na bug and the wrong tool for predicting what you will hear.\n\n### So listen to it instead\n\nPlay the eight-channel render and then the integer one; they are hard to tell\napart. Then open the differences. The first two are the model gap, and they are\nrecognisably a guitar — fizz, pick attack, the top end a smaller network cannot\nreproduce. The third is the entire quantisation error, played at its true level like\neverything else here, and there is nothing to hear: it sits some 49 dB below\nthe program material.\n\n## And then float won anyway\n\nThe integer engine was good. Then we went back to the float engine, because the\nnumber it had lost to was from a *generic* optimised build and nobody had\nhand-written its inner loop.\n\nA leaf kernel now evaluates several consecutive convolutions at once, sharing\nevery weight load between them, with separate accumulation chains so the frames\nnever wait on each other. It ended up about 30% faster than the generic float\nbuild — and slightly faster than the integer engine it was competing with, in real\nsingle-precision float, with no exponents to manage, no scales to tune, and no\nquantisation error at all.\n\nIt also agrees bit-for-bit with the plain C++ version of itself, over random\nweights and both kernel sizes. Everything below depends on that.\n\nTwo things about it caught us out.\n\n**It has to live in instruction RAM.** The 314-byte hot loop, left in external\nflash-backed memory where the display code also runs, put the model in contention\nwith the panel and cost fourteen points of CPU. Fourteen points, for moving 314\nbytes.\n\n**The engine used to keep a second copy of its convolution history** so a read\ncould never wrap around the end of the buffer. Wrapping the pointer properly\ninstead costs 3% more CPU and returns 77 KB of internal RAM — and on this chip\nthat is not a memory-efficiency nicety. Internal RAM is what the display's DMA\nbuffer and the USB endpoints are competing for, and when they lose, USB simply\nfails to start. A 3% slower engine that leaves room for the peripherals is the\nfaster pedal.\n\n## Bit exact, proven on the board\n\nThere is a way to check whether the chip computes the same values as a desktop\nthat takes almost everything else out of the question. A diagnostic image flashes the exact input file and the model onto the\nboard, runs all 720,000 samples through the same engine and the same assembly the\nlive firmware uses, writes the raw output back to flash, and reads it out. No\nconverters, no analog loop, no interface clock — so no converter noise, gain\nuncertainty or cable in the measurement.\n\nThe first comparison against the host came back at −108 dB, and that number turned\nout to be about the reference rather than about the chip.\n\nThe board writes its result back as raw 32-bit float. The host's render was a\n24-bit integer WAV — the correct format for something you intend to listen to, and\none that rounds every sample onto a 24-bit grid on the way in. Decoding it back to\nfloat cannot recover what was discarded. So the comparison was float against\ndecoded-24-bit, and −108 dB was the cost of that round trip through a listening\nformat. The S3 could have been perfect and the number would not have budged.\n\nIt was. Once the host wrote raw float too, all 720,000 samples matched, both files\nhashed identically, and the difference was digital zero at unity gain and\ngain-matched. Both render at −15.7 LUFS and −2.6 dBTP.\n\nThe general form of that is the most portable thing in this post: **your reference\nis part of the measurement.** A null can only be as deep as the weaker of the two\nfiles feeding it, and a perfectly good listening format is a lossy step in the\nchain.\n\n## The model adds a third of a millisecond. The interface adds the rest.\n\nThe inference costs 0.333 ms — one 16-sample block at 48 kHz — and that is the\nentire algorithmic latency the amp model contributes.\n\nThe round trip is about 4.7 ms, and the other 4.4 of it is USB. The iRig HD 2 is a\nfull-speed device, which means the host may talk to it once per millisecond, and\n48 kHz for one millisecond is exactly 48 audio frames. That single fact sets the\nfloor for everything downstream, and no amount of firmware changes it: the packet\nsize is written into the interface's own descriptor.\n\nWhat firmware *could* change was the software stacked on top of that floor, and\nthere was a lot of it — one 48-frame packet per transfer instead of four, a\ntrimmed startup backlog, and two queued playback transfers rather than one.\nBetween them they removed something like 13 ms of pure buffering. The one that\nsurprised me was the playback queue: with a single transfer in flight the host\nstack cannot recycle it before the next USB frame is due, and playback collapses\nto a quarter speed while capture carries on happily at 48 kHz. Two alternating\ntransfers restore it completely.\n\nWhich is why the engine is around 7% of the round trip on this chip, and why the\nP4 sits somewhere else entirely — high-speed USB is scheduled eight times as\noften, so the same work carries far less transport around it.\n\n## And then seven effects moved in\n\nThe amp model is the expensive part but it was never the whole pedal. The touch\nbuild runs eight blocks, the amp among them — gate, compressor, two modulation\nslots, drive, delay and reverb — on the same two cores, at the same 16-sample\nblock.\n\nEvery block opens into an editor.\n\nA chain is serial, so it cannot be split by sample. It is split by stage instead:\ncore 0 runs the pre-effects and the first third of the network, core 1 runs the\nrest of the network and the head, core 0 takes the finished block for the second\nmodulation slot and the delay, and core 1 produces the stereo reverb and packs the\noutput. Pairs of stages share tasks, so the fourth stage needed neither another\nDSP stack nor two threads inside one effect.\n\nAnd then the hand-off between them had two slots, which is not enough — with one\nslot being filled and one being worked on there is never a free one to start the\nnext block into, so the stages serialise instead of overlapping. That looks like\nhealthy load figures, every deadline met, and about 38,000 dropped audio frames\nevery five seconds. Four slots fixed it, for 2,320 bytes. The same trap caught us\non the P4, on entirely different hardware.\n\nWith all of that running live, the shipping preset sits at about two thirds of one\ncore and 92% of the other. Forcing every effect on at once — including two\nmodulation instances, which no sane preset does — takes it to the high eighties\nand low nineties, still with no missed deadlines, no dropped frames and no USB\nerrors. That configuration exists so the scheduler gets measured against\nsomething worse than the product.\n\n### The reverb had to be written twice\n\nThe first compact plate was a four-line feedback network with two input allpasses\nand loops between 6 and 15 milliseconds. It met its deadline comfortably and it\nsounded metallic: too few recurrences, too short, too regular.\n\nWhat replaced it is a half-rate Dattorro-style plate — four input diffusers into\ntwo cross-coupled tanks, each tank a modulated allpass, a long delay, a damping\nfilter and then another of each, with four decorrelated output taps to hide the\nphysical loop periods. It runs at 24 kHz, and the longer eight-line tails behind\nit at 12 kHz, because that is what fits.\n\nRoom, plate, hall and ambient are also no longer one tank with four sets of\nconstants. Room is a sparse early-reflection pattern into a short unmodulated\nnetwork; hall is a wider early field into a modulated one with separate decay\nrates for low and high frequencies; ambient uses heavier input diffusion and the\nlongest tail. Spring maps to the plate, because the real dispersion model measured\nabout 125 µs per block — more than a third of the entire block budget, for one\neffect.\n\nThe mix control was wrong in a way only playing through it found. It was an\nordinary crossfade, so even a modest amount of reverb audibly pulled the direct\nguitar down. It is now a send: the dry signal stays at exactly unity until halfway\nup the control.\n\nAnd like the convolution history before it, the reverb's long delay lines had to\nmove out to external RAM with only its hot core kept internal. On this chip the\nscarce resource is not cycles.\n\n## Traps, recorded so they cost someone else less\n\n**The full eight-channel processor silently does nothing** when you hand it more\nframes than it was allocated for. No error, no partial output — the buffer comes\nback untouched. Our very first render \"worked\" and was just the input.\n**Autocorrelation pitch detection octave-errors** on decaying plucked notes and\nwill confidently report every guitar as a bass. YIN is what you want.\n**Guard every null with a lag probe first.** All of these were confirmed\ntime-aligned before being trusted, so a null is a tone difference rather than a\nlatency artifact.\n**A null cannot be deeper than its reference file.** See −108 dB above.\n\n## So was the integer work wasted?\n\nOn this chip, in the end, yes. The float engine is faster and exact, and it is\nwhat ships on the S3.\n\nExcept that the quantiser turned out to be the only way to run the *big* model on\nthe other chip. The ESP32-P4's vector unit is fourteen times faster than its\nfloating-point unit, and its 410 instructions are all integer, and eight-channel\nA2-Full in float needs more instructions per sample than both of its cores have\ncycles. In integers, on that vector unit, the whole eight-channel model fits and\nruns the pedal.\n\nThat is its own post, the one before this. Block floating point, the sliding peak\nover the history, the retreat rule and the rounding fix all carried over directly,\nand then it needed one more idea on top of them.", "url": "https://wpnews.pro/news/running-nam-a2-lite-natively-on-an-esp32-s3", "canonical_source": "https://playtaurus.com/blog/running-nam-a2-lite-natively-on-an-esp32-s3", "published_at": "2026-08-18 21:56:38+00:00", "updated_at": "2026-08-18 22:10:58.587817+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-products"], "entities": ["ESP32-S3", "Neural Amp Modeler", "NAM A2-Lite"], "alternates": {"html": "https://wpnews.pro/news/running-nam-a2-lite-natively-on-an-esp32-s3", "markdown": "https://wpnews.pro/news/running-nam-a2-lite-natively-on-an-esp32-s3.md", "text": "https://wpnews.pro/news/running-nam-a2-lite-natively-on-an-esp32-s3.txt", "jsonld": "https://wpnews.pro/news/running-nam-a2-lite-natively-on-an-esp32-s3.jsonld"}}