The holy grail of computational neuroscience: Invariance

There are quite a few problems that computational neuroscientists need to solve in order to achieve a true theoretical understanding of biological intelligence. But I’d like to talk about one problem that I think is the holy grail of computational neuroscience and artificial intelligence: the quest for invariance. From a purely scientific and technological perspective I think this is a far more important and interesting problem than anything to do with the “C-word”: Consciousness. 🙂

Human (and animal) perception has an extraordinary feature that we still can’t fully emulate with artificial devices. Our brains somehow create and/or discover invariances in the world. Let me start with a few examples and then explain what invariance is.

Invariance in vision

Think about squares. You can recognize a square irrespective of it’s size, color, and position. You can even recognize a square with reasonable accuracy when viewing it from an oblique angle. This ability is something we take for granted, but we haven’t really figured it out yet.

Now think about human faces. You can recognize a familiar face in various lighting conditions, and under changes of facial hair, make-up, age, and context. How does the brain allow you to do things like this?

Invariance in hearing

Think about a musical tune you know well. You will probably be able to recognize it even if it is slowed down, sped up, hummed, whistled, or even sung wordlessly by someone who is tone-deaf. In some special cases, you can even recognize a piece of music from its rhythmic pattern alone, without any melody. How do you manage to do this?

Think about octave equivalence. A sound at a particular frequency sounds like the same note as a sound at double the frequency. In other words, notes an octave apart sound similar. What is happening here?

What is invariance?

How does your brain discover similarity in the midst of so much dissimilarity? The answer is that the brain somehow creates invariant representations of objects and patterns. Many computational neuroscientists are working on this problem, but there are no unifying theoretical frameworks yet.

So what does “invariance” mean? It means “immunity to a possible change”. It’s related to the formal concept of symmetry. According to mathematics and theoretical physics, an object has a symmetry if it looks the same even after a change. a square looks exactly the same if you rotate it by 90 degrees around the center. We say it is invariant (or symmetrical) with respect to a 90 degree rotation.

Our neural representations of sensory patterns somehow allow us to discover symmetries and using them for recognition and flexible behavior. And we manage to do this implicitly, without any conscious effort. This type of ability is limited and it varies from person to person, but all people have it to some extent.

Back to the examples

We can redefine our examples using the language of invariance.

The way human represent squares and other shapes is invariant with respect to rotation, as well as with respect to changes in position, lighting, and even viewing angle.
The way humans represent faces is invariant with respect to changes in make-up, facial hair, context, and age. (This ability varies from person to person, of course.)
The way humans represent musical tunes is invariant with respect to changes in speed, musical key, and timbre.
The way humans represent musical notes is invariant with respect to doubling of frequency ( which is equivalent to shifting by an octave.)

All these invariances are partial and limited in scope, but they are still extremely useful, and far more sophisticated than anything we can do with artificial systems.

Invariance of thought patterns?

The power of invariance is particularly striking when we enter the domain of abstract ideas — particularly metaphors and analogies.

Consider perceptual metaphors. We can touch a surface and describe it as smooth. But we can also use the word “smooth” to describe sounds. How is it that we can use texture words for things that we do not literally touch?

Now consider analogies, which are the more formal cousins of metaphors. Think of analogy questions in tests like the GRE and the SATs. Here’s an example

Army: Soldier :: Navy : _____

The answer is “Sailor”.

These questions take the form “A:B::C:D”, which we normally read as “A is to B as C is to D”. The test questions normally ask you to specify what D should be.

To make an analogy more explicit, we can re-write it this way: “R(x,y) for all (x,y) = (A,B) or (C,D)”. The relation “R” holds for pairs of words (x,y), and in particular, for pairs (A,B) as well as (C,D).

In this example, the analogical relationship R can be captured in the phrase “is made up of”. An army is made up of soldiers and a navy is made up of sailors. In any analogy, we are able to pick out an abstract relationship between things or concepts.

Here’s another example discussed in the Wikipedia page on analogy:

Hand: Palm :: Foot: _____

The answer most people give is “Sole”. What’s interesting about this example is that many people can understand the analogy without necessarily being able to explain the relationship R in words. This is true of various analogies. We can see implicit relationships without necessarily being able to describe them.

We can translate metaphors and analogies into the language or invariance.

The way humans represent perceptual experiences allows us to create metaphors that are invariant with respect to changes in sensory modality. So we can perceive smoothness in the modalities of touch, hearing and other senses.
The way humans represent abstract relationships allows us to find/create analogies that are invariant with respect to the particular things being spoken about. The validity of the analogy R(x,y) in invariant with respect to replacing the pair (x,y) with (A,B) or (C,D).

The words “metaphor” and “analogy” are essentially synonyms for the word “invariant” in the domains of percepts and concepts. Science, mathematics and philosophy often involve trying to make explicit our implicit analogies and metaphors.

Neuroscience, psychology and cognitive science aim to understand how we form these invariant representations in the first place. In my opinion doing so will revolutionize artificial intelligence.

Further reading:

I’ve only scratched the surface of the topic of invariance and symmetry.

I talk about symmetry and invariance in this answer too:

Mathematics: What are some small but effective theses or ideas in mathematics that you have came across? [Quora link. Sign-up required]

I talk about the importance of metaphors in this blog post:

Metaphor: the Alchemy of Thought

I was introduced to many of these ideas through a book by physicist Joe Rosen called Symmetry Rules: How Science and Nature Are Founded on Symmetry. It’s closer to a textbook that a popular treatment, but for people interested in the mathematics of symmetry and group theory, and how it relates to science, this is an excellent introduction. Here is a summary of the book: [pdf]

Relatively recent techniques such as deep learning have helped artificial systems form invariant representations. This is how facial recognition software used by Google and Facebook work. But these algorithms still don’t have the accuracy and generality of human skills, and the way they work, despite being inspired by real neural networks, is sufficiently unlike real neural processes that these algorithms may not shed much light on how human intelligence works.

Notes:

This post is a slightly edited form of a Quora answer I wrote recently.

In the comments section someone brought up the idea that some invariants can be easily extracted using Fourier decomposition. This is what I said is response:

Good point. Fourier decomposition is definitely part of the story (for sound at the very least), but it seems there is a lot more.

Some people think that the auditory system is just doing a Fourier transform. But this was actually shown to be partially false a century ago. The idea that pitch corresponds to the frequencies of sinusoids is called Ohm’s acoustic law.

From the wiki page:

For years musicians have been told that the ear is able to separate any complex signal into a series of sinusoidal signals – that it acts as a Fourier analyzer. This quarter-truth, known as Ohm’s Other Law, has served to increase the distrust with which perceptive musicians regard scientists, since it is readily apparent to them that the ear acts in this way only under very restricted conditions.
—W. Dixon Ward (1970)

This web page discusses some of the dimensions other that frequency that contribute to pitch:

Introduction to Psychoacoustics – Module 05

There are interesting aspects of pitch perception that render the Fourier picture problematic. For example, there is the Phenomenon of the missing fundamental: “the observation that the pitch of a complex harmonic tone matches the frequency of its fundamental spectral component, even if this component is missing from the tone’s spectrum.”

Evidence suggests that the human auditory system uses both frequency and time/phase coding.

Missing fundamental: “The brain perceives the pitch of a tone not only by its fundamental frequency, but also by the periodicity of the waveform; we may perceive the same pitch (perhaps with a different timbre) even if the fundamental frequency is missing from a tone.”

This book chapter also covers some of the evidence: [pdf]

” One of the most remarkable properties of the human auditory system is its ability to extract pitch from complex tones. If a group of pure tones, equally spaced in freque ncy are presented together, a pitch corresponding to the common frequency distance between the individual components will be heard. For example, if the pure tones with frequencies of 700, 800, and 900 Hz ar e presented together, the result is a complex sound with an underlying pitch corresponding to that of a 100 Hz tone. Since there is no physical energy at the frequency of 100 Hz in the complex, such a pitch sensation is called residual pitch or virtual pitch (Schouten 1940; Schouten, Ritsma and Cardozo, 1961). Licklider (1954) demonstrated that both the plac e (spectral) pitch and the residual (virtual) pitch have the same properties and cannot be auditorally differentiated.”

The status of Fourier decomposition in vision might be more controversial. Spatial frequency based models have their adherents, but also plenty of critics. One of my professors says that claiming the visual system does spatial Fourier amounts to confusing the object of study with the tools of study. 🙂 We still don’t whether and how the brain performs spatial Fourier decomposition.

A very recent paper reviews this issue:

The neural bases of spatial frequency processing during scene perception

“how and where spatial frequencies are processed within the brain remain unresolved questions.”

Vision scientists I know often talk about how the time domain cannot be ignored in visual processing.

A general point to be made is that even if we have mathematical solutions that are invariant, computational neuroscientists haven’t quite figured out how neural networks achieve such invariant representations. The quest for invariance is more about plausible neural implementation than mathematical description per se.

Do mirror neurons explain understanding, or is it the other way round?

(Alternate title: In Soviet Russia, Mirror Neurons Explain YOU!)

A draft of this post has been sitting around for a few weeks, and while I’m happy with today’s sanity check, I still can’t help but suspect that I am missing something in the debate on “action understanding”. So I am happy to be convinced that I have completely misunderstood some key aspect of the mirror neuron story.

Mirror neuron “theory” strikes me as an odd mix of interesting experimental results and ambiguous reasoning. Nevertheless, I think the popularity of the mirror neuron idea serves as an opportunity to talk about some ideas that are hard to work into neuroscience writing aimed at a general, non-mathematical audience. People interested in mirror neurons may have intuitively found their way to a problem that computational neuroscientists consider one of the central challenges in our quest to understand biological intelligence: how does the brain detect and create invariants out of the formless and ever-changing sensory raw material available to it? As we shall see, the activity of a canonical mirror neuron displays a kind of invariance — its firing reflects the recognition of an action irrespective of the agent performing the action.

First of all, what exactly are mirror neurons? They are supposedly a “class” of neurons that fire when a monkey is performing a goal-oriented action, and also when another individual (human or monkey) is performing the same action. A group of Italian researchers discovered them in the premotor cortex of rhesus macaques [1]. Since these neurons were found in premotor cortex the researchers inferred that they were instrumental in “understanding” the goal-directed action, irrespective of the agent. Some researchers link this “embodied” understanding with imitation, since it appears to be based in the brain areas that trigger motor responses.

People often refer to mirror neurons as the basis of empathy, based on the idea that our understanding of other people’s actions and feelings depends on our ability to imitate or emulate the feelings of others. There are two independent lines of research being merged here: (i) psychological studies in humans linking imitation with empathy, and (ii) mirror neuron studies in monkeys linking brain activity with imitation or action understanding. But there are many more connections that need to be made before even hinting at a neural theory of imitation-based empathy. First of all, the evidence for mirror neurons in humans is still being debated. And even if a consensus emerges that they exist as described, we still don’t have a very clear picture of the neural mechanisms linking the processes that are presumably involved in empathy: sensation, perception, pattern recognition, motor control and subjective feeling. Ignoring these and other major holes in the story, VS Ramachandran felt confident enough to assert that mirror neurons were “the driving force behind the great leap forward in human evolution”. I have cited some work that points out flaws in the way the experimental results have been interpreted [2], but for the purpose of this post, let’s just take the results as given.

I don’t deny that mirror neurons seem to be linked to fascinating phenomena, but we should not be surprised at all to find these phenomena reflected in the brain. Where else could they be? So let’s investigate what we already know — but rarely state explicitly — about the human mind-brain-behavior continuum. This will allow us to put the mirror neuron hype in perspective, and perhaps find out where the real scientific puzzles are lurking.

Let’s think about a typical goal-directed action — aiming at a target with a bow and arrow. When you see Robin Hood aiming at the target, you can say, “Robin Hood is aiming at the target”. When you see yourself aiming at the target, you can say, “I am aiming at the target.” This is beyond obvious. Now, unless you suspect that language is controlled by the stomach or the pancreas, you already knew — and I am just reminding you — that the concept of “aiming” that you are able to identify in other people and in yourself is somewhere in your brain.

I see what you’re doing there!

Convinced that concepts are represented somewhere in the brain (either in a localized or distributed manner), we can be reasonably confident that the concept of “aiming” can affect the brain processes that control your mouth, lungs and vocal chords, allowing you to say the word “aiming”, or mirror the action. And it is clear that the concept must be independent of who is holding the bow and arrow. Imagine the alternative. You might call Robin Hood’s act of aiming “trapooling” and you might call your own act of aiming “caduffing”. If you were unable to see the similarity between what you do and what anyone else does, you would be unable to replace “trapooling” and “caduffing” with a single word that describe the action you and Robin Hood were both performing. If we were really plagued by a blindness to similarity between different experiences, we would be unable to communicate about them at all, because our perceptions would be unique and unrepeatable, and therefore private and inaccessible.

Language depends on the ability to recognize and categorize things, processes, and abstract concepts. So we can safely assume that some neurons or groups of neurons or patterns of neural activity will appear to be “mirroring”, because they can access your muscles and trigger aiming when you see someone (anyone!) engaged in aiming at a target. And since concepts must have access the motor system, it’s no surprise to find neurons that correlate with them in motor areas of the brain. They would have to affect the motor system at some stage, right?

It’s not hard to see why people often associate “understanding” with motor responses. How do we tell if someone has understood what we are saying? We can ask them to imitate the action we’re talking about. Or we can ask them to point to what we are talking about. Either way, to say that someone else understands a concept is to say the he or she can act on it appropriately. [See note 2 for some comments on why this kind of understanding is not merely a feature of language.]

All this becomes apparent when we witness a child learning the meaning of a word. When my cousin’s son was 2 years old he was still slowly figuring out how to use colour words. He would pick up a coloured Duplo block and ask us, “Blue?” If the block was in fact blue one of us would nod and smile, and he would be very pleased with himself. But sometimes he picked up a red block and asked “Blue?” And we would gently shake our heads and correct him. Clearly it took some trial-and-error for him to grasp the meaning of the word. Now how do we know he has understood? We don’t typically use brain scans or mind reading. We infer his understanding from his ability to use the word “blue” in the appropriate contexts. If he can use the word blue only when it’s appropriate, and pick out blue things when they are pointed out to him, most people will happily admit that he has understood the concept of blue. And the same goes for action words like running, jumping and aiming. Even for someone who is paralyzed or in a vegetative state, we infer understanding from the ability to respond to stimuli– to move an eyelid or finger, or to modulate internal brain activity that is detected by a scanner.

Very little neuroscience is needed to support the assertion that the brain is central to these phenomena. Neuroscience is really needed to explain how the brain does all this. We still have no idea. If we did, you would be able to teach your smartphone a new board game by just having it listen in while you read out the rules.

So what then is the radiant how question obscured by the mirror neuron hype cloud? I think it might be about invariance. In physics and mathematics, invariance (or symmetry) means insensitivity to a change or transformation [3]. For example, a perfect uniform square is invariant with respect to rotations of 90 degrees. If you rotate a square by 90 degrees, it looks the same. Let’s say you’re playing that toddler’s puzzle of slotting wooden blocks of various shapes into a board with corresponding holes. There’s a hole for each block. You pick up a square block and try to place it in the square hole. Even if you rotate it by 90 degrees, it will fit in the hole. An equilateral triangle, on the other hand, cannot be rotated by 90 degrees and still fit in its designated hole. But it does have an invariance — you can rotate it by 120 degrees and it will fit. “Rotation by 90 degrees” and “Rotation by 120 degrees” are examples of transformations. So a square is invariant to rotation by 90 degrees, and an equilateral triangle is invariant to rotation by 120 degrees. The board with the holes in it is like a rudimentary pattern recognition system. It recognizes shapes of particular sizes, for a handful of orientations. You could imagine a high-tech version of this puzzle, in which successfully fitting a square block in the corresponding hole triggers a computerized voice that says “You have found a square!”

Humans can make use of far more invariants that the board with the holes. When it comes to shape recognition, the average humans is better than most advanced computer programs. Our ability to detect squares, for instance, is invariant or near-invariant to changes in size, orientation, viewing angle, texture, and so on. And consider how well we recognize other people! You can often spot a person you know even if he or she is far away, in disguise, in an unexpected place, wearing different make-up or sporting a new hairstyle. So your internal representation of that person is invariant with respect to a variety of transformations.

Similarly, your internal representation of an action is, to varying degrees, invariant to a change in the agent performing the action, the nature of the goal or target, and the context in which the action is taking place. Thus you can recognize “running”, whether it’s a human running on a track, a chicken running in a field, or a bull running through the streets of Pamplona. And you can recognize “aiming” whether you are doing it or Robin Hood is doing it, so your internal representation of “aiming” is invariant with respect to the change of agent. How the brain facilitates finding and creating this sort of invariance is a major open problem, and is being investigated from various angles by psychologists, cognitive scientists, artificial intelligence researchers, and neuroscientists.

Mirror neuron activity appears to be invariant with respect to a change of agent. This is clearly an interesting experimental finding, and gives us possible physical correlates of a complex psychological phenomenon. But does it deserve all the recent hype? After all, (a) we know from human behavior that the brain must show invariants, and (b) we don’t know how particular invariants arise.

Both the hype and the ambiguous reasoning I mentioned before seems to stem from using mirror neurons as an explanation for understanding, rather than a consequence of brain processes that remain to be explained. In popular articles, and even in the technical literature, mirror neurons are starting to sound like magical antennas that pick up what is happening, and somehow “know” that the same goal-oriented action is being performed irrespective of who is doing it. And if you lack these magical antennas, so the story goes, you may become autistic! Neuroscientists often make fun of the idea of the homunculus — a little man inside your brain that sits in a control room and “sees” what you see, “hears” what you hear and so on. The homunculus was dismissed because it simply passes the explanatory buck down one level, forcing us to ask how the homunculus sees and hears in the first place. But the mirror neuron “system” somewhat resembles a homunculus — it does all the hard work of recognizing goal-oriented actions and determining that they are similar in the first place.

So the real question to be asked is: how are mirror neurons able to do what they do? Mirror neurons are not magical antennas. The invariants they are able to create or detect must therefore be a product of their intrinsic properties, their inputs, and their interactions with other cells. If they seem like antennas then we must understand how they can “tune-in” to particular aspects of stimuli and not others. It is the goal of computational neuroscience to understand how the great radio that is the brain helps an organism tune-in to various “frequencies” in the world in order to play the music that is its behavior. To do so we investigate the biophysical mechanisms by which the neural networks act together to control the state of the whole body. Simply saying that the mirror neurons “do empathy” doesn’t help much.

______________

Blogospheric critiques of mirror neuron theory

Mirror Neurons — The unfalsifiable theory — Talking Brains

What’s So Special About Mirror Neurons? — SciAm

Mirror Neurons: The Most Hyped Concept in Neuroscience? — Psychology Today

Notes

Here’s a somewhat unfair and exaggerated analogy that gets at the inadequacy of mirror neurons as an explanation of anything. Imagine if physicists explained the elliptical orbits of planets by simply stating that the planets contain innate “ellipse particles” that are predisposed to making elliptical paths. So ellipses are caused by ellipse particles. This would be a useless explanation. On the one hand, we already knew about the ellipses. On the other, the explanation doesn’t tell us anything about how the ellipse particles work. Solving the real puzzle requires understanding how the planets interact with the sun and each other, and how these interactions result in elliptical motion. The task of neuroscience is to understand how interactions between neurons in neural networks allows for mirroring behavior to emerge. And if we want to talk about empathy, we will have to do even more work, connecting sensory signals with perception, pattern recognition and subjective feeling. In the interim we are saying not much more that this: “Mirroring is caused by mirror neurons.” We could easily reverse this, and say “The faculty of mirroring results in mirror-neuron-like firing patterns.” Rather than an explanation of empathy or understanding, mirror neuron theory thus far seems to be little more than a re-description of the phenomena with no added explanatory power.
In order to learn the names of new concepts, you must be able to recognize phenomena in the world in agent-independent ways. For example, let’s say you recently learned the name of a particular yoga position — padmasana, for instance. Before discovering the name, you must be able to recognize the position as something that multiple people, including yourself, might be capable of performing. Otherwise when someone points to a person performing padmasana, you will not know what is being pointed to, and therefore will not be able to recognize the concept in other people and yourself. Without this recognition, naming would be impossible. So even though I use language to illustrate this process of mirroring or invariance-finding, the process is “prelinguistic” and seems to form part of the scaffolding necessary for language development.

References

[1] Rizzolatti G, Fogassi L, & Gallese V (2001). Neurophysiological mechanisms underlying the understanding and imitation of action. Nature reviews. Neuroscience, 2 (9), 661-70 PMID: 11533734 [pdf]

This is the group that discovered mirror neurons. This paper reviews the idea of mirror neuron theory as an account of “action understanding”. They say that “By action understanding, we mean the capacity to achieve the internal description of an action and to use it to organize appropriate future behaviour”. They claim that mirror neuron theory supports the idea that “an action is understood when its observation causes the motor system of the observer to ‘resonate’.”

[2] Hickok, G. (2009). Eight problems for the mirror neuron theory of action understanding in monkeys and humans. Journal of Cognitive Neuroscience,21(7), 1229-1243. doi:10.1162/jocn.2009.21189 [PubMed]

Here are some key problems listed by Greg Hickok (who also wrote the Talking Brains blog post I linked to). As you can see, they are much more specific to the experimental methodology, the specific location where the purported mirror neurons were found, and the relevance of mirror neurons to “action understanding”.

There Is No Evidence in Monkeys That Mirror Neurons Support Action Understanding
Action Understanding Can Be Achieved via Nonmirror Neuron Mechanisms
The Relation between Macaque Mirror Neurons and the ‘‘Mirror System’’ in Humans Is Either Nonparallel or Undetermined
Action Understanding in Humans Dissociates from Neurophysiological Indices of the Human ‘‘Mirror System’’
Damage to the Inferior Frontal Gyrus Is Not Correlated with Action Understanding Deficits

[3] Rosen, J. (1995). Symmetry in Science: An Introduction to the General Theory. [goodreads link]

This is a highly recommended textbook explaining the concept of mathematical symmetry, and its relevance to science. In fact, Rosen argues very convincingly that science is the quest to uncover symmetry in nature.

Star Trek and Tin Man pics courtesy Wikipedia.

Robin Hood Clipart courtesy FCIT. [Source: Louis Rhead. Bold Robin Hood and His Outlaw Band (New York and London: Harper & Brothers, 1912) 29. Retrieved March 13, 2013, from http://etc.usf.edu/clipart/8100/8191/robin_hood_8191.htm]