We have different ways of seeing what the brain is up to, from low-resolution electrodes that track waves of activity coursing through the brain to implanted electrodes that can track the activity of individual cells. Combined with a detailed knowledge of which regions of the brain are involved in certain processes, we were able to achieve remarkable things such as robotic arm.
But today researchers announced a new piece of mind-reading that is impressive in its scope. By combining fMRI imaging of the brain with a system similar to cell phone predictive text, they extracted the core of the phrases a person hears in near real time. While the system doesn’t get the exact words right and makes a fair number of mistakes, it’s also flexible enough to reconstruct an imaginary monologue that runs entirely in a person’s head.
Making functional MRI work
Functional MRI is a way to see which parts of the brain were active. By adjusting the imaging sensitivity to capture differences in blood flow, it’s possible to identify areas in the brain that are replenishing their energy after processing some information. It has been extremely useful for understanding how the brain works, but it also has some significant limitations.
First, the person whose brain is being imaged has to be put in an MRI scanner for it to work, so it’s not useful for real-world applications. In addition, spatial resolution is somewhat limited, so activities involving only a small number of cells can be difficult to resolve. Finally, the response to activity picking up fMRI is somewhat delayed; it begins to rise a few seconds after neural activity, builds to a peak, and does not fade until a full 10 seconds after the neurons are active.
This delay makes tracking a continuous activity – reading a few paragraphs, for example – very difficult, since some areas activated by a word may partially overlap, both physically and temporally, with those activated by subsequent words.
The researchers behind the new work (a small team at the University of Texas at Austin) decided to skip identifying individual words and focused on developing a system that can identify a sequence of words. And they developed an amazing data set to train the recognition system: three people sat in MRI tubes for 16 hours while text was read to them.
What are you listening to right now?
The system works by constantly updating a list of candidate phrases that match brain activity registered by the fMRI. The computerized part of things was based on a generative neural network language model that had been trained on the English language and narrowed down the word combinations to be considered. (It keeps the system from evaluating the possibility of users hearing “aimless hyena mauve” or something similarly nonsensical.)
This still leaves a wide range of valid English phrases for the system to take into account. To handle this in near real time, the researchers turned to what is known as a beam search algorithm. As the neural response to a spoken word is processed, the system creates a list of potential high-scoring matches. When the next-word answer is added, it converts these into scored lists of two-word combinations that both fit the neural signals well and make sense because of their training in English. As each new response comes in from the fMRI, it further refines these lists until it has values ​​for whole phrases.
By managing these as lists rather than as a single answer, the system avoids discarding a good match if it misidentifies a single word. And by restricting itself to common English usage, the system avoids getting bogged down in tracking nonsensical phrases.
#MRI #enable #ability #mind #hears
