Skip to main content
#visual-system#neuroscience#visual-cortex#neural-decoding#fmri

How the Human Visual System Processes Information

An overview of information processing in human vision and its relationship to visualization.

#The Mind’s Eye: From Light to Meaning

Open your eyes in the morning and color, shape, and motion enter awareness almost at once. Seeing feels so immediate that the work behind it is easy to overlook.

#Main findings

  • The brain interprets and reconstructs retinal input in stages to produce visual experience.
  • The report follows visual information through the retina, the lateral geniculate nucleus (LGN), V1, and higher visual areas, then examines how researchers decode neural activity.
  • The interactive guide illustrates the function of each stage.

#Reading order

  1. Compare rods and cones to understand the input stage.
  2. Follow the operations at successive stages of the visual pathway.
  3. Explore neural decoding and the research questions it raises.

#🧠 A journey through the visual system

The human visual system performs remarkably sophisticated information processing. Follow a ray of light through the stages that contribute to conscious sight.

#📊 Comparing rods and cones

R Rods

Count: About 92 million

Location: Predominantly in the peripheral retina

Function: Dim-light vision and sensitivity to movement

Color: A rod signal alone cannot distinguish colors.

C Cones

Count: About 6 million

Location: Concentrated in the central retina

Function: Daylight vision and color discrimination

Color: Three types sensitive to short, medium, and long wavelengths

#🧠 Interactive guide to vision

Follow the stages below to explore how the human visual system extracts meaning from light.

A journey through human vision

Seeing involves a series of transformations. Light enters the eyes, and the brain interprets the resulting signals. Follow the path from incoming light to conscious perception.

Interactive Brain Science Report | 2025

👁️

The Mind's Eye

From light to meaning: information processing in the human visual system and how researchers visualize it

The moment we open our eyes, shapes, colors, and movements enter awareness. The experience is so immediate that it rarely prompts a question about the underlying computation. Yet seeing is an active process. The eyes sense light, and the brain interprets and reconstructs the input to give a scene meaning.

📖
Introduction: the puzzle of seeing

The brain infers what is likely to be present from the sensory evidence available. Predictive accounts describe this as constructing a best estimate of the world.1 Vision extracts the information needed for survival and action efficiently. A perfect internal copy of every detail would be a different objective.2 Starting from two-dimensional retinal images, the brain combines current context and past experience to build a vivid representation of a three-dimensional world.

This report follows a photon through the transformations that lead to a meaningful percept. Two directions of information flow interact. Bottom-up processing carries sensory input toward higher brain areas. Top-down processing lets knowledge, expectations, and attention influence how the input is interpreted.1 The main account follows the ascending pathway and returns to the effects of top-down modulation at each stage.

We begin in the retina, where light becomes an electrical signal. Signals are compressed and sorted along the route through the thalamus. Primary visual cortex extracts basic visual features. Further stages combine these signals into representations of color, form, and motion and support judgments about what an object is and where it is. Finally, neural decoding approaches the problem from the opposite direction: what can measurements of brain activity tell us about images a person sees or imagines? These stages reveal the coordinated work behind everyday sight.

1

Stage 1 | Eye and retina: sensing the visual world

Light passes through the eye before reaching the retina. The cornea and lens refract the light to focus an image on the retina. Neural processing already begins there. The retina is a specialized neural circuit extending from the central nervous system into the eye. Through phototransduction it converts physical light energy into electrical signals and begins extracting visual information.

From light to an electrical signal: rods and cones

The retina has two classes of photoreceptors that detect light: rods and cones.3 Their different sensitivities and functions provide a combination analogous to a high-resolution color sensor for daylight and a highly sensitive monochrome sensor for night.

R Rods

Count: Approximately 92 million rods provide high sensitivity to weak light.4

Location: Rods are concentrated away from the retinal center. They support vision in dim light (scotopic vision) and help detect movement.3

Property: Rods alone cannot discriminate colors. When they dominate vision in very dim light, the scene appears largely monochrome.

Pigment: Rods use rhodopsin, a visual pigment whose light-sensitive component derives from vitamin A. Vitamin A deficiency can impair vision in dim light and cause night blindness.4

C Cones

Count: About 6 million cones support fine central vision and color vision under bright (photopic) conditions.3

Location: Cones are concentrated in the macula, particularly its central fovea. We direct our gaze toward an object to place fine detail on this region.3

Color perception: The three cone types have overlapping sensitivities to short, medium, and long wavelengths, often described as blue, green, and red. Their combined responses allow us to distinguish many colors.

📋 Parallel input channels give vision both the fine shape and color information available in daylight and sensitivity to movement under dimmer conditions.

Property Rods Cones
Approximate number per retina About 92 million About 6 million
Main distribution Predominantly in the peripheral retina Central retina, especially the fovea
Light sensitivity High (functions in dim light) Lower (functions in bright light)
Color discrimination None from rods alone Yes, through three cone types
Visual acuity Low High
Principal visual pigment Rhodopsin Cone opsins (photopsins)

The retina's first computations: receptive fields, edges, and contrast

Photoreceptors signal through bipolar cells to retinal ganglion cells. Horizontal and amacrine cells modulate the signals within this network of retinal interneurons.4 Substantial processing therefore takes place before the signals even leave the eye.

How a receptive field works

A ganglion cell's receptive field is the area of the retina in which light can change that cell's activity.8 In the 1950s, Stephen Kuffler's experiments helped establish that many ganglion cells have a central region and a surrounding region that exert opposing effects. This arrangement is called center–surround organization.10

ON-center/OFF-surround cell

Light in the center excites the cell and increases its firing rate. Light in the surround inhibits the cell and reduces its firing rate.9

OFF-center/ON-surround cell

Light in the center inhibits the cell, while light in the surround excites it.9

[Visualization: responses of a retinal ganglion cell's receptive field]

The diagram illustrates how an ON-center cell responds to different patterns of illumination.

Center only: Illuminating only the center strongly excites the cell, increasing its firing rate.

Surround only: Illuminating only the surround strongly inhibits the cell, reducing its firing rate.

Center and surround together: Under uniform illumination, excitation from the center and inhibition from the surround can largely cancel. The firing rate may remain near its spontaneous level.9

Edge stimulus: When a light–dark boundary crosses the receptive field, a bright center beside a dark surround can produce a strong response.

📋 Mechanism: Lateral inhibition contributes to this arrangement. Signals from neighboring retinal regions influence one another through circuits that include horizontal cells.7 This organization makes ganglion cells sensitive to spatial contrast, the difference in brightness across locations.

Data compression and efficient coding

The retina contains nearly 100 million photoreceptors, but only about a million ganglion cells transmit its output toward the brain. Uniform regions produce redundant signals. Spatial contrast and edges convey information critical for identifying contours, so center–surround circuits emphasize these changes. This is an example of efficient coding under limited transmission capacity. Retinal output already describes relationships between locations in the image.

2

Stage 2 | Optic nerve and thalamus: transmission and relay

Processed signals travel along the optic nerve to the lateral geniculate nucleus (LGN) in the thalamus. This journey supports more than transmission. Contrasts extracted in the retina are routed, sorted, and modulated before cortical interpretation.

The optic chiasm connects each visual hemifield to the opposite hemisphere

The axons of retinal ganglion cells come together in the optic nerve, which contains roughly a million nerve fibers.10 Some fibers cross at the optic chiasm on the underside of the brain. This arrangement delivers information about the opposite visual hemifield to each cerebral hemisphere.

Which fibers cross?

Nasal retinal fibers

Fibers from the nasal half of each retina cross at the chiasm. They carry information about the temporal (outer) part of that eye's visual field.11

Temporal retinal fibers

Fibers from the temporal half of each retina continue on the same side of the brain. They carry information about that eye's nasal (inner) visual field.11

An integrated visual field map

Right hemisphere: Receives the left visual hemifield from both eyes where their views overlap

Left hemisphere: Receives the right visual hemifield from both eyes where their views overlap

Significance: Bringing the two eyes' views of the same region to one hemisphere makes their signals available for estimating depth through binocular disparity.11

📋 Clinical significance: Damage at the optic chiasm can produce a characteristic pattern of visual field loss. For example, compression by a pituitary tumor can cause bitemporal hemianopia, a loss of the outer visual field in each eye.12

The lateral geniculate nucleus: an organized visual relay

Signals from the optic nerve reach the LGN of the thalamus. This structure processes and regulates incoming visual information. In primates it has six distinguishable principal layers, with different layers carrying signals with different response properties.13

M Magnocellular pathway

Location: LGN layers 1–2 (the two lower layers)

Cell size: Magnocellular neurons have large cell bodies.

Function: These neurons carry signals important for motion, depth, and low spatial frequencies, which describe broad patterns with relatively little fine detail.14

Property: High sensitivity to changes over time supports the detection of rapid movement.

P Parvocellular pathway

Location: LGN layers 3–6 (the four upper layers)

Cell size: Parvocellular neurons have smaller cell bodies.

Function: They carry color, texture, and high-spatial-frequency signals that support fine detail.14

Property: Their sensitivity to spatial detail contributes to precise form recognition and color discrimination.

K Koniocellular pathway

Location: Interlaminar zones between the six principal LGN layers

Cell size: Koniocellular neurons have very small cell bodies.

Function: This heterogeneous pathway includes blue–yellow opponent signals, sensitivity to low spatial frequencies, and other specialized visual functions.15

Property: Identified more recently than the M and P pathways, K cells account for roughly 10% of LGN neurons in the estimate used here.

Additional roles of the LGN

Attentional modulation: Signals from cortex and brainstem alter the transmission of visual input. Depending on attention, the LGN can strengthen relevant signals and suppress less relevant ones.17
Sleep and wakefulness: Changes in thalamic transmission during sleep can reduce the influence of external visual input on awareness, including during dreaming.18

Sorting and modulating visual information

In the LGN, retinal contrasts are organized into parallel channels with differing sensitivities to motion, color, and fine form. This division of labor helps the brain analyze a complex scene. Attention and arousal also regulate how readily these signals pass on to cortex.

The optic chiasm connects each visual hemifield to the opposite hemisphere

The axons of retinal ganglion cells come together in the optic nerve, which contains roughly a million nerve fibers.10 Some fibers cross at the optic chiasm on the underside of the brain. This arrangement delivers information about the opposite visual hemifield to each cerebral hemisphere.

Which fibers cross?

Nasal retinal fibers

Fibers from the nasal half of each retina cross at the chiasm. They carry information about the temporal (outer) part of that eye's visual field.11

Temporal retinal fibers

Fibers from the temporal half of each retina continue on the same side of the brain. They carry information about that eye's nasal (inner) visual field.11

An integrated visual field map

Right hemisphere: Receives the left visual hemifield from both eyes where their views overlap

Left hemisphere: Receives the right visual hemifield from both eyes where their views overlap

Significance: Bringing the two eyes' views of the same region to one hemisphere makes their signals available for estimating depth through binocular disparity.11

📋 Clinical significance: Damage at the optic chiasm can produce a characteristic pattern of visual field loss. For example, compression by a pituitary tumor can cause bitemporal hemianopia, a loss of the outer visual field in each eye.12

The lateral geniculate nucleus: an organized visual relay

Signals from the optic nerve reach the LGN of the thalamus. This structure processes and regulates incoming visual information. In primates it has six distinguishable principal layers, with different layers carrying signals with different response properties.13

M Magnocellular pathway

Location: LGN layers 1–2 (the two lower layers)

Cell size: Magnocellular neurons have large cell bodies.

Function: These neurons carry signals important for motion, depth, and low spatial frequencies, which describe broad patterns with relatively little fine detail.14

Property: High sensitivity to changes over time supports the detection of rapid movement.

Color sensitivity: Relatively low color sensitivity, with responses chiefly to luminance.

P Parvocellular pathway

Location: LGN layers 3–6 (the four upper layers)

Cell size: Parvocellular neurons have smaller cell bodies.

Function: They carry color, texture, and high-spatial-frequency signals that support fine detail.14

Property: Their sensitivity to spatial detail contributes to precise form recognition and color discrimination.

Temporal resolution: Responses to rapid changes are relatively slow.

K Koniocellular pathway

Location: Interlaminar zones between the six principal LGN layers

Cell size: Koniocellular neurons have very small cell bodies.

Function: This heterogeneous pathway includes blue–yellow opponent signals, sensitivity to low spatial frequencies, and other specialized visual functions.15

Property: Koniocellular neurons account for about 10% of LGN neurons in the estimate cited here. Many of their functions remain under investigation.

Evolutionary significance: This pathway has been proposed to be evolutionarily ancient.

LGN receptive fields: luminance and color contrast

Many LGN neurons, like retinal ganglion cells, have antagonistic center–surround receptive fields. More specialized contrasts are also evident here. For instance, color opponency in the parvocellular pathway provides sensitivity to differences in chromatic input as well as to luminance contrast.16

Red–green opponent cells

The center may favor a long-wavelength (red) signal and the surround a medium-wavelength (green) signal, or the preferences may be reversed. This opposition emphasizes subtle red–green differences.16

Blue–yellow opponent cells

Opposing short-wavelength cone signals and combined longer-wavelength signals helps distinguish bluish hues, such as those of the sky, from yellowish hues, such as those of some fruits.16

Additional roles of the LGN

Attentional modulation: Signals from cortex and brainstem alter the transmission of visual input. Depending on attention, the LGN can strengthen relevant signals and suppress less relevant ones.17
Sleep and wakefulness: Changes in thalamic transmission during sleep can reduce the influence of external visual input on awareness, including during dreaming.18
Coordination with eye movements: Visual processing changes around rapid eye movements (saccades). This coordination contributes to the reduced visibility of disturbances during a saccade, known as saccadic masking.19

Sorting and modulating visual information

The LGN sorts and modulates the contrast signals it receives from the retina. Parallel channels emphasize motion, color, and fine detail, distributing the work of analyzing a scene. Attention and arousal influence the information passed onward. These organized signals next reach primary visual cortex for further interpretation.

3

Stage 3 | Primary visual cortex: extracting basic features

From the thalamic LGN, signals reach primary visual cortex (V1) in the occipital lobe.26 Also known as striate cortex, V1 is the first cortical stage of the principal visual pathway. Many of its neurons encode oriented edges and line segments. This discovery changed our understanding of how cortex represents a scene. David Hubel and Torsten Wiesel shared the 1981 Nobel Prize in Physiology or Medicine for their work.28

Hubel and Wiesel's discovery of orientation selectivity

One initial expectation was that cortical neurons would respond to small spots of light like the cells studied in the retina and LGN. While recording V1 neurons in anesthetized cats, Hubel and Wiesel noticed strong firing when the edge of a slide moved across the visual field.26 This observation led to the discovery that many V1 neurons respond selectively to a line or boundary at a particular orientation.

Simple cells

Their receptive fields have elongated, distinguishable excitatory (ON) and inhibitory (OFF) subregions.25

They respond best to a line segment presented at a particular angle and position.

For example, a cell tuned to a vertical line may respond weakly or not at all to a horizontal or diagonal line.

Complex cells

These cells also favor particular orientations but respond to a feature across a wider range of positions than simple cells do.25

They lack clearly separated ON and OFF subregions. An oriented segment can elicit a response at several positions within the receptive field.

Some complex cells also respond strongly to a line moving in a particular direction.

📋 One useful model treats V1 as a bank of orientation filters. Many neural populations analyze the same image in parallel, each emphasizing a particular orientation in a particular part of the visual field.

Imaging orientation maps and pinwheels

If neighboring V1 neurons prefer certain orientations, how are those preferences distributed over the cortical surface? Optical imaging and fMRI have made that spatial organization visible.26

Features of orientation maps and pinwheels

Orientation map

Neurons with similar preferred orientations tend to cluster into orientation columns. Preferences change progressively across the cortical surface, forming a mosaic-like orientation map.31

Pinwheel organization

Around a pinwheel center, columns are arranged like spokes, sampling orientations across a half turn (0–180 degrees). This organization places units sensitive to a range of orientations close together for a local region of visual space. It has been interpreted as an efficient way to cover locations and orientations.34

The emergence of feature-based organization

V1 introduces another organizing principle. The retina and LGN retain spatial relationships in a retinotopic map. V1 preserves a map of where a stimulus occurs and also organizes responses according to properties such as its orientation. Together they encode location and feature.

4

Stage 4 | Higher visual cortex: assembling representations of form and motion

From V1, signals for local features such as oriented line segments reach surrounding visual areas, including V2, V3, V4, and V5/MT. These areas show different functional specializations. Through their interactions, simple signals contribute to more complex representations of color, depth, form, and motion.30

Hierarchical processing and functional specialization

V4 Color and form

Location: A key area of the ventral stream

Function: Contributes to the perception of color and shape.23

Property: Many neurons respond to curves or complex patterns more selectively than to simple line segments.

Color constancy: Helps support stable perception of an object's color under changes in illumination.

V5 Motion processing

Location: A key area of the dorsal stream

Function: Strongly specialized for visual motion.38

Property: About 90% of neurons show direction selectivity in the estimate presented here.40

Motion blindness: Damage affecting this area can disrupt the perception of continuous motion.

V2 Integration and depth

Location: Adjacent to V1

Function: Combines signals arriving from V1.

Property: Some neurons respond to illusory contours.

Depth perception: Binocular disparity provides a cue to depth.

📋 Visual cortex beyond V1 has a hierarchical organization. Along pathways from V1 to V2 and onward to areas such as V3 and V4, receptive fields typically become larger. Neurons can respond to increasingly complex combinations of visual features.

Imaging functional specialization with fMRI

Functional MRI maps task-related signals associated with blood oxygenation. Comparing stimulus conditions makes it possible to visualize which cortical regions respond more strongly to a particular visual feature.42

Visualizing color responses in V4

Participants alternate between complex colored patterns, such as Mondrian-like pictures, and a simple black-and-white grating. A comparison can show preferential responses to colored patterns in the occipital region associated with V4.

Visualizing motion responses in V5/MT

Participants view hundreds of randomly moving dots alternating with the same dots held stationary. The moving pattern evokes a strong motion-sensitive fMRI response in V5/MT near the temporal–occipital junction.38

Activation and awareness

Activity in one region need not coincide with conscious perception. In a study discussed here, V5/MT remained active during rapid visual stimulation even when participants did not report seeing motion.38 Such findings suggest that conscious perception involves interactions among several regions, potentially including higher frontal and parietal areas, beyond activity in a single specialized visual area.

5

Stage 5 | Two streams for perception and action

Higher visual signals follow two broad streams. The two-streams hypothesis, proposed in 1982 by Leslie Ungerleider and Mortimer Mishkin from work in monkeys and refined for human vision by Melvyn Goodale and David Milner, has strongly influenced cognitive neuroscience.41

Ventral stream

Identifying what an object is

The ventral stream supports object identification and recognition. This role gives it the name “what” pathway.23

Anatomical route

Signals from V1 pass through regions including V2 and V4 and extend along ventral cortex into inferior temporal cortex (IT).23

Effects of damage

Damage can produce visual agnosia: a person may retain basic visual acuity yet have difficulty recognizing objects. Prosopagnosia, a difficulty recognizing faces, is one example.41

Dorsal stream

Locating and interacting with an object

The dorsal stream represents spatial location and motion and helps guide bodily action. It contributes to answering “Where is it?” and “How do I interact with it?”41

Anatomical route

From V1, projections involving V2 and V5/MT extend along the dorsal route into posterior parietal cortex (PPC).23

Effects of damage

Damage can impair spatial judgments and visually guided action. Examples of deficits associated with affected networks include optic ataxia and hemispatial neglect.41

Complementary networks and unified experience

The two-streams account describes specializations within an interacting system. Research has documented substantial communication between the pathways.

Examples of interaction

Tool use: Selecting an appropriate grip for fine manipulation engages the action system. Knowing whether the target is a hammer or a screwdriver also draws on object identification.49

Object recognition: Recognizing an object partly obscured by a cluttered background may require information about its spatial outline and position as well as its identity.47

Current perspective

The pathways exchange information as complementary networks. Perception and action together support a unified experience of the visual world.

♦

Neural decoding: connecting neural activity with visual experience

How do patterns of spikes relate to experiences as varied as seeing a red rose, a moving car, or a friend's face? This remains an open question in neuroscience and philosophy. Neural decoding provides a quantitative way to investigate part of it: researchers work backward from a pattern of activity to infer the visual information it represents.46

Approaches to decoding

fMRI-based visual decoding

Researchers have used spatial patterns in fMRI measurements to reconstruct aspects of viewed images. The original report attributes to the Kamitani group an accuracy above 90% for classifying simple characters or digits from V1–V3 voxel patterns. That figure concerns a restricted classification task, not the accuracy of reconstructing an arbitrary image.21

Researchers have also reported partial decoding of dream-related visual content from fMRI signals.20

High-resolution decoding with electrodes

Electrodes can record spikes from individual neurons. Simultaneous recordings from dozens of V4 or IT neurons in monkeys can carry enough information to predict an object category or certain visual features of a stimulus.22

A related brain–computer interface (BCI) decodes motor cortex activity during attempted handwriting by a person with paralysis. The system can turn the decoded movement patterns into characters on a screen in real time.48

The unresolved question of consciousness

Even a highly accurate decoder leaves a fundamental question open: how is objective neural activity related to subjective experience?

The hard problem

Philosopher David Chalmers calls this the hard problem of consciousness. We can establish that a V1 neuron responds to a line tilted 45 degrees while still asking why neural processing is accompanied by experience at all.1

The explanatory gap

In principle, we could record every neuron's activity in extraordinary detail and still face the question “Why does red feel like this?” The relationship between a physical description and the character of an experience remains unresolved.

Future directions

Advances in artificial neural networks and brain imaging give researchers finer tools. Philosophical analysis can clarify which questions those tools answer. Together these approaches may help connect neural activity with conscious experience as we work out the operations of the visual system piece by piece.

✨
Conclusion: looking ahead

Studying vision brings us back to questions about who we are and how we come to know the world. Present methods give a partial view into the mind's eye. The discoveries already in hand invite further experiments and reflection on the human experience of seeing.

🧠 Continuing research on human vision

References

1 Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181-204.

2 Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. Sensory Communication, 217-234.

3 Rodieck, R. W. (1998). The first steps in seeing. Sinauer Associates.

4 Kolb, H. (2003). How the retina works. American Scientist, 91(1), 28-35.

7 Hartline, H. K. (1949). Inhibition of activity of visual receptors by illuminating nearby retinal areas in the Limulus eye. Federation Proceedings, 8, 69.

8 Kuffler, S. W. (1953). Discharge patterns and functional organization of mammalian retina. Journal of Neurophysiology, 16(1), 37-68.

9 Enroth-Cugell, C., & Robson, J. G. (1966). The contrast sensitivity of retinal ganglion cells of the cat. Journal of Physiology, 187(3), 517-552.

10 Schiller, P. H. (1992). The ON and OFF channels of the visual system. Trends in Neurosciences, 15(3), 86-92.

11 Hickey, T. L., & Guillery, R. W. (1979). Variability of laminar patterns in the human lateral geniculate nucleus. Journal of Comparative Neurology, 183(2), 221-246.

12 Miller, N. R. (1982). Walsh and Hoyt's clinical neuro-ophthalmology. Williams & Wilkins.

13 Hubel, D. H., & Wiesel, T. N. (1977). Ferrier lecture: Functional architecture of macaque monkey visual cortex. Proceedings of the Royal Society of London, 198(1130), 1-59.

14 Livingstone, M., & Hubel, D. (1988). Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853), 740-749.

15 Hendry, S. H., & Reid, R. C. (2000). The koniocellular pathway in primate vision. Annual Review of Neuroscience, 23(1), 127-153.

16 De Valois, R. L., & De Valois, K. K. (1993). A multi-stage color model. Vision Research, 33(8), 1053-1065.

17 Sherman, S. M., & Guillery, R. W. (2002). The role of the thalamus in the flow of information to the cortex. Philosophical Transactions of the Royal Society B, 357(1428), 1695-1708.

18 McCormick, D. A., & Bal, T. (1997). Sleep and arousal: thalamocortical mechanisms. Annual Review of Neuroscience, 20(1), 185-215.

19 Ross, J., Morrone, M. C., & Burr, D. C. (1997). Compression of visual space before saccades. Nature, 386(6625), 598-601.

Connect