Osaka ’s Graduate School of Frontier Biosciences’ new paper details how they use Stable Diffusion, an AI image generation program, to translate brain activity into corresponding visual representation. Although there have been many previous, similar thought-to-computer image experiments, this test is the first to employ Stable Diffusion. For additional system training, researchers linked thousands of photos’ textual descriptions to volunteers’ brain patterns detected when viewing the pictures via functional magnetic resonance imaging (fMRI) scans.

Blood flow levels fluctuate within the brain depending on which areas are being activated. Blood traveling to humans’ temporal lobes, for example, helps with decoding information about image “contents.” while the occipital lobe handles dimensional qualities like perspective, scale, and positioning. An existing online dataset of fMRI scans generated by four humans looking at over 10,000 images was fed into Stable Diffusion, followed by the images’ text descriptions and keywords. This allowed the program to “learn” how to translate the applicable brain activity into visual representations.

During the testing, a human looked at the image of a clock tower. The brain activity registered by the fMRI corresponded to Stable Diffusion’s previous keyword training, which then fed the keywords into its existing text-to-image generator. From there, a recreated clock tower was further detailed based on the occipital lobe’s layout and perspective information to form a final, impressive image.

The team’s augmented Stable Diffusion image generation is limited only to the four person image database—further testing will require additional testers’ brain scans for training purposes. That said, the team’s groundbreaking advancements show immense promise in areas such as cognitive neuroscience, and as Science notes, could help researchers delve into how other species perceive their environments.