Zero Compute AI
A paper on the next step in the evolution of Neural Networks
by
Patience Strong
Introduction
Neural Networks, in their current form, require vast amounts of compute / electrical power to function. They require this degree of power in order to emulate a technology that, if expressed as hardware, would require no computational power at all.
In short, the compute power is consumed by the emulation process, not by the network or the task that network is processing. Therefore, if we stop emulating the circuit, and instead build it, no conventional computer at all would be required to achieve the same result.
It is the above observation, and tentative proposals on how to achieve the next iteration of this technology, that is the subject of this paper.
Current Technology
Neural Networks are built around the concept of the artificial neurone, like this:
Image 1: A simple neurone.
The massive compute demand comes from the host servers needing to perform very simple calculations billions of times per second in order to emulate an array of these neurones. So, although the computation is trivial, it is the scale of the task that consumes the compute power.
Essentially, the software is reading and summing the values referred to as “weights” in figure 1, a trivial task, then offsetting that summed value by the value of a constant called a “bias”, another trivial task, then using a conditional statement to decide if the final computed value is greater than the trigger threshold for the output.
Again, very trivial coding, but when you have billions of neurones in the network, and you need to compute these trivial tasks for every one of them in real time, then you require a ridiculous amount of compute / electrical power to do so.
The Proposal
This paper is proposing that instead of using all this compute power to emulate the circuit, we actually build it as a semiconductor device.
The intent here is to simply demonstrate that it is not nearly as difficult to express this network as a semiconductor device, as it is for example, to make a GPU.
An example of a viable approach
The weights and Bias
The weights and Bias shown could be realised as Digital to Analog Converters, (DAC). Each of which would have its register implemented as non-volatile memory, so the value of the weight or bias can survive power loss.
The registers would use standard address and data buses to allow them to be updated each time a new iteration of the neural network is released.
The Activation Function
The output from the weights and bias feed into a fast switching transistor that turns on when the current or voltage is above the activation threshold, and remains off when below it.
No doubt an expert in semiconductor design would come up with something even simpler.
It is worth noting that this device will process its neurones in parallel, so it should be considerably faster than a GPU, as well as draw far less power.
Conclusion
If the neural network is implemented as an electrically programable device, then successive releases of an AI can be overwritten on the non-volatile chip hosting it. So parity with current upgrade practices can be maintained. Because each neurone handles the decision whether to trigger its output or not in real time, no computer is required. The only need for computers would be to handle incoming requests and returning the results.
I hope you find this document useful and that it sparks a productive conversation.
Yours sincerely.
Patience