I taught myself integer-only neural net training in Rust — and ran into a real quantization problem

I'm a 10th grade student teaching myself Rust by building something instead of just reading theory. The project is called Green-AI — an attempt to explore more energy-efficient AI training by using only integer arithmetic, no floating point anywhere except for timing measurements.

Turns out this breaks in a genuinely interesting way, and fixing it taught me more about numerical precision than any tutorial did.

The problem: a "dead zone" in fixed-point training

A standard weight update in my integer neuron looks like this:

fn update(&self, error: i32, input: i32) -> i32 {
    (((error as i64) * (input as i64)) >> 14) as i32
}

That >> 14 is doing the job floating-point division normally does — it scales the update down to a sane range. The problem: once the error gets small enough, (error * input) >> 14 rounds down to 0, even though the error isn't actually zero. The weights just... stop updating. Training silently stalls.

I benchmarked this "Standard" approach over 1000 runs: it converges fast, but lands with a final weight error of 77 compared to the true target weights. It gets stuck in the dead zone before it ever gets close.

My fix: ABSL (Adaptive Bitshift Learning)

My fix was to make the shift amount adaptive to the size of the error — so updates never round away to nothing:

fn update(&self, error: i32, input: i32) -> i32 {
    let abs_error = error.abs();

    let shift = match abs_error {
        e if e > 15000 => 17,
        e if e > 8000  => 16,
        e if e > 4000  => 15,
        e if e > 1500  => 14,
        e if e > 500   => 12,
        e if e > 200   => 10,
        e if e > 80    => 8,
        e if e > 20    => 7,
        _ => 6,
    };

    (((error as i64) * (input as i64)) >> shift) as i32
}

Large errors get damped more (bigger shift, prevents overshoot), small errors get amplified more (smaller shift, prevents stalling).

I also tried stochastic rounding as an alternative fix, which is closer to what's actually used in quantized-training resaerch. You probabilistically round the truncated fraction up or down instead of always down, so updates accumulate correctly on average.

Results

Algorithm

Avg time/run

Final weight error

Converges after

Standard (fixed shift)

132.5 µs

77

1928 steps

Stochastic Rounding

206.8 µs

9

846 steps

AdaptiveShift (ABSL)

143.6 µs

5

2230 steps

ABSL ends up the most accurate of the three, and noticeably faster per step than stochastic rounding. The trade-off: it takes more steps to fully settle, likely because it keeps making meaningful updates near the target instead of stalling out early — accuracy over speed-to-convergence.

What I'm still figuring out

The shift thresholds (15000, 8000, 4000...) are hand-tuned magic numbers for this one neuron's weight scale. I don't yet know how to make them scale automatically to different layer sizes.

Everything so far is tested on a single neuron, not a real multi-layer network with backprop.

Is "adaptive step size based on error magnitude" a known technique outside of fixed-point contexts? It feels conceptually close to things like Rprop, but I got here from a completely different angle (avoiding integer truncation).

Repo's here if you want to poke at the code: https://github.com/Mojo0869/green-ai

Feedback very welcome, especially from anyone who's worked with quantized or fixed-point training before — I'd love to know what I'm missing.