The days of keeping your thoughts to yourself may be numbered.
Scientists have unveiled a groundbreaking ‘mind-reading’ AI that can recreate images people are looking at simply by analysing their brain scans.
As part of a study, participants were asked to look at photos ranging from images of a baseball game to a dog leaning out of a car and a group trekking across a snowy plain.
The AI programme analysed their brain activity patterns through a scan and reproduced the pictures – which it had never seen before – with remarkable accuracy.
‘There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well,’ Professor Michal Irani, from the Weizmann Institute of Science, said.
‘However, they tend to make mistakes in basic features such as composition and colour.
‘The new model we developed outperforms them in reconstructing both the content of the image and its details.
‘What’s more, while every other model requires dozens of hours of brain scans to learn to “read” a new person, our model needs only one hour.’
Scientists have unveiled a groundbreaking ‘mind-reading’ AI that can recreate images people are looking at simply by analysing their brain scans. Pictured: The photos people viewed (left) and the AI reconstruction (right)
The AI programme analysed their brain activity patterns through a scan and reproduced the pictures – which it had never seen before – with remarkable accuracy (referred to as RECON here)
To build the system, called Brain-IT, the scientists fed it thousands of brain scans collected while eight volunteers looked at different images.
This allowed the programme to learn how specific patterns of neural activity correspond to colours, shapes and objects.
It was so accurate that it even learned to predict what a brain scan would look like from an image.
By combining data from multiple studies, the researchers identified brain regions that appear to perform similar functions across different people.
For example, one area consistently responded to images of food while another was activated by pictures of sport. ‘During training, the encoder naturally identified 128 functional regions that are shared by all people and perform specific roles in image processing,’ Professor Irani said.
‘Some of them are familiar to neuroscientists, but others are entirely new.
‘For example, we discovered a division of roles within the brain region that processes images of places – the PPA – with one part responding to indoor scenes and another to outdoor scenes.’
When given a new brain scan, the AI was able to generate a remarkably accurate reconstruction of the original image.
The system was fed thousands of brain scans collected while eight volunteers looked at different images. When it was then tested, it could recreate a set of new images (RECON) the participants were looking at
Participants were asked to look at photos ranging from images of a baseball game (left) to a dog leaning out of a car and a group trekking across a snowy plain. Brain-IT could then reproduce the picture (right)
Other AI ‘mind reading’ tools require about 40 hours of brain scan data on anyone new before they can be used to predict what a person is seeing.
But this new decoder only needs 60 minutes of data, the team explained.
To prove this, they compared the images generated by Brain-IT when trained with one hour of data compared to the same system trained with 40 hours, and the results are remarkably similar.
They also compared their system’s image recreation results with those of other programmes, revealing it generated much more accurate reconstructions.
Professor Irani’s lab is now working to extend these ‘mind-reading’ methods to the decoding of auditory information as well.
Video could be another frontier.
‘What remains especially challenging is decoding video – for example, during dreaming,’ she explained.
‘Dozens of images change every second, while an fMRI scan takes about two seconds. If we overcome all these obstacles, it’s possible that in the future we may even be able to read dreams.’
The team compared the images generated by Brain-IT when trained with one hour of data compared to the same system trained with 40 hours, and the results are remarkably similar
They also compared their system’s image recreation results with those of other programmes, revealing it generated a much more accurate reconstructions
The team are also working towards similar systems that decode brain activity recorded using electroencephalography (EEG).
This technique measures the brain's electrical signals through sensors placed on the scalp, sometimes via a cap or even specially designed headphones.
As AI models become more sophisticated, scientists expect it will become increasingly easy to interpret the brain data collected in this way, rather than through MRI scans.
The findings were presented at the Cognitive Computational Neuroscience conference in New York last month.