Large scale machine learning models are based on mathematical abstractions of biological neural networks. The highly successful GPT-3 model basically models the synapses of more than 8 million neurons (with preposterously large matrix calculations).
As a former psychology major, I know that there is more to the brain than synapses. Neurons have nuclei, membranes, and a trunk (axion) with branches (dendrites). And lots of chemicals inside and out. (And who knows what quantum effects we don't know about.)
This makes me wonder what is being left out of the commonly used models, and why.
I'm pretty sure the "why" has a lot to do with practical considerations of contemporary engineering. The synaptic activity is modelled in discretized vectors that block together groups of synapses in blocks of time. The propagation of values (which natural neurons do via the other anatomical parts) is represented by big matrix computations. The output represents a spatial layout of synapse, which encodes the result.
This computational model works pretty well, though power consumption is becoming prohibitive.
An alternative design models the behavior of the axon, which propagates sequences of pulses. This is represented in a specialized chip, confusingly called a "neuromorphic chip". (This is "morphic" only for selected aspects of a neuron—there could be a lot of different kinds of "neuromorphic chips".) The output is a temporal pattern (I htink), which represents the behavior over time of each neuron.
This winter Stanford researcher Kwabena Boahen describes another concept: "dendrocentric" computation [1]. This approach models the behavior of dendrites, which, if I recall correctly connect the axon with one or more synapse.
There is much I don't understand about the mathematical representation here. Biological dendrites have a complicated array of electrochemical behaviors which, presumably encode natural computations. If I have it right, the model represents the sequence of electrochmical pulses arriving along the dendrite.
The point of this approach is that there is a lot more information encoded in these pulse trains than in the values of synapses. This is a critical advantage: Boahen estimates that this could reduce power consumption orders of magnitude compared to GPT-3 like models [2].
Stepping back, all this makes me wonder exactly what we are doing here. How can emulating only one part of a complex natural system produce reasonable results? If three different partial emulations all work, at least to a degree, what does this mean? And what would happen if we combined them into a more comprehensive model?
Above all, it looks to me like there is a missing theory, a theory of how natural neural nets work. In particular, I think there should a mathematical theory of "learning networks". Naturally evolved neural networks will be an optimal sub-case, specialized for embodied bio-nets. But there will be other optima that work completely different but are very efficient and effective for particular cases. (And who knows what exotic special cases might fall out of the math—microbial networks, "networks" of stars, a theory of human language, a theory of music.)
Phew.
Anyway, I'm intrigued by this growing field of specialized hardware for machine learning. I admit that my inner software engineer is offended by the inelegant, brute force success of "deep learning". Throwing massive amounts of ten year old hardware (and electricity) at a problem may work, but it isn't satisfying.
So, yes, please. Give me some more weird new chip designs.
- Kwabena Boahen, Dendrocentric learning for synthetic intelligence. Nature, 612 (7938):43-50, 2022/12/01 2022. https://doi.org/10.1038/s41586-022-05340-6
- Charles Q. Choi, Dendrocentric AI Could Run on Watts, Not Megawatts, in IEEE Spectrum - Artificial Intelligence, December 20, 2022. https://spectrum.ieee.org/dendrocentric-learning
No comments:
Post a Comment