Despite the wilderness success of ChatGPT and other major language models, the artificial neural networks (ANNs) that underlie these systems may be on the wrong track.
First, ANNs are “superpower hungry,” he said Cornelia Fermuller, a computer scientist at the University of Maryland. “And the other problem is [their] lack of transparency.” Such systems are so complicated that no one really understands what they do, or why they work so well. This, in turn, makes it almost impossible to get them to reason analogously, which is what humans do: use symbols for objects, ideas and the relationships between them.
Such shortcomings likely stem from the current structure of ANNs and their building blocks: individual artificial neurons. Each neuron receives input, performs calculations, and produces output. Modern ANNs are extensive networks of these computational units, trained to perform specific tasks.
Yet the limitations of ANNs have long been apparent. For example, think of an ANN that keeps circles and squares apart. One way to do this is to have two neurons in the output layer, one indicating a circle and one indicating a square. If you want your ANN to also recognize the color of the shape, say blue or red, you need four output neurons: one for the blue circle, the blue square, the red circle, and the red square. More functions mean even more neurons.
This cannot be how our brains perceive the natural world, with all its variations. “You have to imagine having a neuron for all combinations,” he said Bruno Olshausen, a neuroscientist at the University of California, Berkeley. “So you would have in your head, [say,] a purple Volkswagen detector.”
Instead, Olshausen and others state that information in the brain is represented by the activity of numerous neurons. So the perception of a purple Volkswagen is encoded not as the actions of a single neuron, but as those of thousands of neurons. The same set of neurons, firing differently, could represent an entirely different concept (perhaps a pink Cadillac).
This is the starting point for a radically different approach to computation known as hyperdimensional computation. The key is that each piece of information, such as the idea of a car or its make, model, or color, or all together, is represented as a single entity: a hyper-dimensional vector.
A vector is simply an ordered sequence of numbers. For example, a 3D vector consists of three numbers: the X, j, And z coordinates of a point in 3D space. For example, a hyperdimensional vector, or hypervector, could be a sequence of 10,000 numbers representing a point in 10,000-dimensional space. These mathematical objects and the algebra to manipulate them are flexible and powerful enough to take modern computers beyond some of the current limitations and promote a new approach to artificial intelligence.
“This is what I’ve been most excited about, practically in my entire career,” said Olshausen. For him and many others, hyper-dimensional computing promises a new world where computing is efficient and robust and machine decisions are completely transparent.
Enter high-dimensional spaces
To understand how hypervectors enable computing, let’s go back to images with red circles and blue squares. First, we need vectors to represent the SHAPE and COLOR variables. Then we also need vectors for the values that can be assigned to the variables: CIRCLE, SQUARE, BLUE and RED.
The vectors must be different. This distinction can be quantified by a property called orthogonality, meaning that they are at right angles. In 3D space, there are three vectors that are perpendicular to each other: one in the X direction, another in the j, and a third in the z. In a 10,000-dimensional space there are 10,000 such mutually orthogonal vectors.