First patient fitted with new ‘mind-reading’ device uses brain signals to write
A man who developed paralysis and lost his ability to speak following a stroke can now communicate using a system that translates electrical signals from his brain into individual letters, allowing him to create whole words and sentences in real time.
To use the device, which receives signals from electrodes implanted in his brain, the man silently tries to say code words that replace the 26 letters of the alphabet, according to a new report, published Tuesday (November 8th) in the log Nature Communication (opens in a new tab). These code words come from the NATO phonetic alphabet, in which “alpha” represents the letter A, “bravo” for B and so on.
“The NATO phonetic alphabet was developed for communication over noisy channels”, Sean Metzger (opens in a new tab), first author of the study and a doctoral candidate at the University of California, Berkeley and the University of California, San Francisco Bioengineering Graduate Program, told Live Science. “That’s kind of the situation we’re in, where we’re in this noisy environment of neural recordings.” The researchers first tried using individual letters instead of code words, but their system had trouble distinguishing between phonetically similar letters, such as B, D, P, and G.
By silently speaking the NATO code words, the user generates brain activity which can then be decoded by algorithms which assemble the desired letters and insert spaces between the words as they are formed. To complete a sentence, the user attempts to shake their right hand; this produces distinct brain activity that tells the device to stop decoding.
Related: What happens in our brain when we “hear” our own thoughts?
In recent tests, the man could produce sentences from a vocabulary of over 1,150 words at a speed of 29.4 characters per minute, or about seven words per minute. The decoder sometimes made errors when translating its brain activity into letters, showing a median character error rate of 6.13%.
This marks an improvement over a previous test of the system, which was described in a 2021 report in The New England Journal of Medicine (opens in a new tab). In this test, the man constructed sentences by trying to say whole words aloud from a defined vocabulary of 50 words. The device could decode about 18 words per minute with a median accuracy of 75% and a maximum accuracy of 93%.
“It was great, but limited,” in terms of vocabulary and what the user tried to say the words out loud, Metzger said. The latest trial of the system shows that the system was still running quietly and that by using an orthographic approach, a user can significantly expand the available vocabulary. In the future, the two approaches could be easily combined: users could rely on the whole-word decoder to quickly generate common words, and they could use the single-letter decoder to spell less common words, Metzger explained. .
The man featured in both studies is the first participant in the Arm and Voice Brain-Computer Interface Restoration Trial (BRAVO) (opens in a new tab), which takes place at UC San Francisco. The trial is open to adults who have lost significant speech and motor control due to conditions such as stroke, amyotrophic lateral sclerosis (ALS) and muscular dystrophy.
At age 20, the participant suffered a severe stroke that cut off blood flow to a part of the brainstem called the pons. This structure acts as a bridge between the brain and the spinal cord, and following his stroke, the participant lost much of his ability to move his head, neck and limbs and all of his ability to produce speech. intelligible. In general, humans now communicate by using their limited head mobility to select letters on a screen using a physical pointer or head-controlled cursor.
The man entered the BRAVO trial at the age of 36, when he underwent surgery to have a web of 128 electrodes laid on the surface of his brain. Basically, these electrodes rest on a region of the wrinkled cerebral cortex which controls the muscles of the vocal tract, ordering them to move and thus produce specific sounds. It also covers the area of the brain involved in hand movement.
Related: Can we think without using language?
Currently, to connect to the set-top box, the test participant must be physically plugged into the device via a port that crosses the skin of his scalp. Ideally, the system in the future will be completely wireless, Metzger said.
To calibrate the decoder, the researchers asked the participant to silently try to say each of the NATO code words and practice shaking their right hand. Over time, they also made him spell arbitrary words and copy entire sentences, letter for letter. Eventually, after spending about 11 hours practicing with the system, the man was able to spell his own original sentences and produce answers to specific questions.
A system limitation is that there is a 2.5 second time window allocated for each letter; meanwhile, the user silently says a code word, and the system records and decodes the resulting brain signals. Reducing that time window and making the decoding rate more flexible will be key to both increasing system speed, Metzger said.
Although the new study includes only one participant, it is “still a groundbreaking study”, said Jun Wang (opens in a new tab), an associate professor in the Departments of Speech, Language, and Hearing Sciences and Neurology at the University of Texas at Austin. More research is needed to find out if the same approach will work for other patients, or if it will need to be tailored somewhat for each person, Wang told Live Science in an email.
To be suitable for everyday use, these devices will need to be easy to use without assistance for patients and their caregivers, and they will need to interface with other computer software, Wang said. The technology would be particularly useful for patients in a locked-in state, who are almost completely paralyzed but retain full cognitive function and some ability to move their eyes, he said.