When Numbers have Colours

I just finished my latest Coursera on-line course on Visual perception and the Brain. As part of the course assignments, I had to write a final essay on a topic related to visual perception. I have long been fascinated by synaesthesia, so I researched it and wrote a little about it. What follows is an adapted version, suitable for a blog.

Synesthesia2Daniel Tammet has synaesthesia. To Daniel, numbers on a page have a unique three-dimensional shape and unique colour. That is, the digit 1 is a hazy glow of white light. The number 4 is a blue boomerang. The 5 is a yellow cross-hatched square. The number 6 is a tiny black hole. Daniel “sees” these shapes and colours both when looking at the number on a printed page as well as when he is simply imagining the number. Number-colour synaesthetes, like Daniel, have a unique multi-modal experience of the world. A stimulus to their visual sense triggers an associated experience in another of their senses (eg. colour, auditory, haptic or olfactory). In Daniel’s brain, as in the brains of approximately 0.5-2% of the world’s population, there is an unusual amount of cross-talk between sensory systems that are usually somewhat isolated from one another. This gives rise to one sensory modality triggering another (eg, a graphical representation of a number causing a sensation of colour). There are several different sub-types of synaesthesia where one modality of perception triggers another modality. The most common is the number–colour synaesthesia (like Daniel) in which numbers have shape and colour. The literature also documents word-colour, sound-colour, texture-colour, sound-taste and several other forms of inter-modality triggering. In a recent TED talk, Daniel used painting to create a pictorial representation of the colours, emotions, textures he experiences when the thinks of the first 20 digits of PI. Very cool.

Why Number-Colour? Why Not Number-Smell?

Could any human sense cross-talk with any other? Could a number have a particular smell, for instance? It seems not. Adjacency in the brain is among the stronger theories of why synaesthetes have certain sensory modalities paired with other modalities. Cross-connections are more likely in adjacent areas than they would be in part of the brain that were distant from one-another. In a 2001 paper by the neuro-psychologists Ramachandran and Hubbard, the authors propose that the number-colour type of synaesthesia may result from hyperconnectivity between nerve systems in the fusiform gyrus and/or in the angular gyrus areas of the brain. These locations in the brain are known as integration points for different visual-perceptual and emotional systems. Dr. Jamie Ward of University of Sussex points out that our neural systems for taste are close to those of systems which support spoken language in Broca’s area of the Brain. Thus, there are synaesthetes who pair the sound of words with a particular taste.

Hereditary Basis of Synaesthesia

There is some evidence that synaesthesia is hereditary (passed along the X chromosome). Thus, one theory postulates that synaesthesia could arise due to a genetic mutation that inhibits the pruning of neural connections between perceptual areas of the brain during fetal development. Reinforcing this observation, Ramachandran and Hubbard refer to experiments where researchers found much larger neural feedback from inferior temporal areas to the V4 visual area of the brain in prenatal monkeys than they did in adult monkeys. They speculate that should connection pruning during fetal development fail to eliminate these links, then connections between the number-grapheme area and the V4 area would persist into adulthood supporting the ability to the experience of colour when viewing numbers or letters. While genetic factors appear to be part of the picture, genetics by itself cannot account for the whole phenomenon. The neural cross-connections merely permit a number to evoke a colour. Learning is an essential reinforcing element because people are not born with an understanding of number and letter symbols. Ramachandran and Hubbard point out the fact that different synaesthetes have different colours evoked by the same numbers. For example, to one synaesthete 3 is red while to another it is blue.

The “Inverse Problem” in Visual Cognition

It is a bit hard to believe, but our eyes do not perceive the world as it is.The Inverse Problem The image that impinges on the back of our retinas is in fact an amalgam, a blend, of a number of distortions that occur as light from (say) the sun bounces off an object and the photons are picked up by your eye. It works like this … light from the sun traverses the turbulent atmosphere, is partially absorbed by the object in front of us and partly reflected by it, is reflected by near-by objects that blend with the main reflection, traverses yet more atmosphere, gets inverted and bent by your eye’s cornea and lens and then finally ends up on the rods and cones that make up your retina. These are only sensitive to certain wavelengths of light and their response changes with time. By the time the image is being picked up by your retinal cells and sent to your brain, it is a mere approximation of what you are looking at.

Figuring out what is ‘really’ out there in the physical world is called The inverse problem in visual cognition. The crux of the inverse problem stems from the fact that the same image on the retina can be generated by different physical objects with different shape, colour, motion or orientation. The classic example illustrating this is that if the sun was a little brighter and the paper a little darker, our retina would register the same image as when the sun was duller and the paper brighter. Just using our visual system, we can’t know which of these two scenarios are in fact ‘reality’. Grey_square_optical_illusion-300x232The reverse is also true. We may see the same patch of colour as lighter or darker depending on the surrounding area in the image. For example, the patches A and B in the image to the right are in fact the same shade of gray — we just see them differently due to the context.

This fundamental limitation causes ambiguity in that we cannot be sure what physical source is causing the visual stimuli we see. Is that a tiger hiding in the bushes or is it just the play of shadow and light on the leaves? Our retinal images are the same in both cases. Answering such questions quickly and accurately can be the difference between living to procreate and being somebody’s lunch. Dr. Dale Purves, in the Coursera course I’m taking called Visual Perception and the Brain, makes the point that images are only representations of the physical world. They only exist for us in our perceptual and cognitive systems. Our visual perception of the physical world is distorted due to variances in luminance, reflectance, transmittance and the physiological limitations of our perception systems. As such, as we try to make sense of what our eyes see, our brain must make a number of assumptions and extrapolations based on heuristics and memory. As synaesthetes have a special kind of integrated milti-sensory perception, could they have an evolutionary advantage over non-synaesthetes in their ability to make sense of the physical world? For example, Dr. Purves has shown that the same patch of colour in the physical world is experience differently by our perceptual system depending on the context that the patch appears in.

Could There be an Evolutionary Advantage to Synaesthesia?

Could someone with synaesthesia, using senses triggered by that patch of colour, have memories and experiences which could afford him/her an evolutionary advantage? Dr. Jamie Ward, in a recent video interview outlines a number of theories as to why their might be an advantage to being a synaesthete. For example, he relates a study where synaesthetes have been shown to have an ability to remember certain shapes or colours better than non-synaesthetes. This ability to use other sensory systems to help remember more visual information and remember it more accurately could be an adaptation that could be selected for by natural selection. Having more and better data in which to associate images with phenomenon in the physical world could help address the inverse problem in visual perception and thus could result in better survival and reproduction. That being said, Dr. Ward asserts that more work is needed to better understand the phenomenon because there are different forms of synaesthesia with different genetic and phenotypic characteristics.

The phenomenon of synaesthesia has interested me for years. I can’t help but feel a bit impoverished knowing that there are people who have a more integrated and rich experience of the world than I do.

P.S. Did you know that Leonard Bernstein, Richard Feynman and Vincent Van Gogh were all synaesthetes? So were a lot of creative forces in history (here is a list of more famous people with synaesthesia).