Berkeley, CA, USA. Researchers succeeded in decoding and reconstructing the dynamic visual experiences of people, paving the way for reproduction of dreams and memories.

So far, the technology blends brain imaging and computer simulation to reconstruct movie clips people have already viewed, such as Hollywood movie trailers, but offers some startling possibilities.


The breakthrough opens up a variety of futuristics scenarios, such as accurate recovery of early childhood memories, examining the minds of stroke and/or coma patients, or watching one's own dreams. "This is a major leap toward reconstructing internal imagery," said Professor Jack Gallant, a neuroscientist at the University of California, Berkeley. "We are opening a window into the movies in our minds." The findings appear in the journal Current Biology.

Adobe Flash Player not installed or older than 9.0.115!

Video courtesy of Nishimoto, et al., University of California, Berkeley. Time: 00:00:29.

Reconstruction from brain activity.

The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI.

The procedure is as follows:

[1] Record brain activity while the subject watches several hours of movie trailers.

[2] Build dictionaries (regression model) to translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points in the brain at which brain activity was measured.

[3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions.

[4] Build a random library of ~18,000,000 seconds of video downloaded at random from YouTube (that have no overlap with the movies subjects saw in the magnet).

[5] Put each of these clips through the dictionaries to generate predictions of brain activity. Select the 100 clips whose predicted activity is most similar to the observed brain activity.

[6] Average those clips together. This is the reconstruction.Eventually, practical applications of the technology could lay the groundwork for brain-machine interface so that people with cerebral palsy or paralysis, for example, can guide computers with their minds.

However, researchers point out that the technology is decades from allowing users to read others' thoughts and intentions, as portrayed in such sci-fi classics as Brainstorm, in which scientists recorded a person's sensations so that others could experience them.

Previously, Gallant and fellow researchers recorded brain activity in the visual cortex while a subject viewed black-and-white photographs. They then built a computational model that enabled them to predict with overwhelming accuracy which picture the subject was looking at.

Their latest experiment solved a much more difficult problem by actually decoding brain signals generated by moving pictures. "Our natural visual experience is like watching a movie," said Shinji Nishimoto, lead author of the study and a post-doctoral researcher in Gallant's lab. "In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences."

Nishimoto and two other research team members served as subjects for the experiment, because the procedure requires volunteers to remain still for hours at a time inside the functional Magnetic Resonance Imaging (fMRI) scanner.

They watched two separate sets of Hollywood movie trailers, while fMRI was used to measure blood flow through the visual cortex, the part of the brain that processes visual information. On the computer, the brain was divided into small, three-dimensional cubes known as volumetric pixels, or "voxels."

"We built a model for each voxel that describes how shape and motion information in the movie is mapped into brain activity," Nishimoto said.
The brain activity recorded while subjects viewed the first set of clips was fed into a computer program that learned, second by second, to associate visual patterns in the movie with the corresponding brain activity.

Brain activity evoked by the second set of clips was used to test the movie reconstruction algorithm. This was done by feeding 18 million seconds of random YouTube videos into the computer program so that it could predict the brain activity that each film clip would most likely evoke in each subject.

Finally, the 100 clips that the computer program decided were most similar to the clip that the subject had probably seen were merged to produce a blurry yet continuous reconstruction of the original movie.

Reconstructing movies using brain scans has been challenging because the blood flow signals measured using fMRI change much more slowly than the neural signals that encode dynamic information in movies, researchers said. For this reason, most previous attempts to decode brain activity have focused on static images.

"We addressed this problem by developing a two-stage model that separately describes the underlying neural population and blood flow signals," Nishimoto said.

Ultimately, Nishimoto said, scientists need to understand how the brain processes dynamic visual events that we experience in everyday life. "We need to know how the brain works in naturalistic conditions," he said. "For that, we need to first understand how the brain works while we are watching movies."

Name

Email

Website

Title

Comment

smaller | bigger

I have read and agree to the Terms of Usage.

Add Comment