Science to restore speech is moving fast. Here’s what’s next

[Audio] The best part of sleep is falling asleep. That’s the only part you’re there for.

Jules: Their styles are different, but I have always gravitated towards their authenticity.

Sanders: There’s another comedian who is extra meaningful for Jules: Conan O’Brien.

[Audio] Today is a bit of a different sort of episode. I’ll be talking to Jules and Maria.

Sanders: After MA had reached out, Conan invited the couple onto his podcast, where they connected about comedy. Conan still checks in on Jules every so often.

[Audio] It’s incredible to me that in our conversations, you’re very frank about the pain that you go through, and how intense it is. But you’re also always so positive and funny and, frankly, you give me a lot of [beep]. Which I find to be, well I hate it, by the way, first of all. Oh, get over it, Conan.

Jules: Conan is very special to me. Watching and listening to him helped me disconnect from the harsh reality of my life. And when I got to meet him and chat with him, it meant the world to me.

Maria Aleandra Fernandez: Having the most normal conversation with Conan O’Brien is something I will never get over.

[Audio] So he has a bunch of banked phrases. He’s going to play a few of our favorites. OK. I’m fine. I’m perfectly [beep] fine. [laughter] Come on man, are we doing this or not? Oh my God, oh my God. I’m destroying this marriage.

MA: He really is so genuine and like, not only willing to talk to us, but really looks forward to talking. Like, I love talking to you guys. I’m not just doing it to be a nice person, which feels, you know, feels wonderful.

[Audio] You have my love, my respect and we’re thinking about you guys all the time.

Sanders: As you were talking earlier, it made me think about what it means to make somebody laugh and to entertain them, and like how actually very, very special that must be. Just kind of, at its heart, it’s, it is joy and humans connecting. And so, extra special.

MA: That’s the other thing I’ve learned mainly through Jules, about humor and comedy. And who were we watching yesterday? The NPR interview with Dave Chappelle.

[Audio] Dave Chappelle says he is very clear about his role as a comedian.

MA: And Dave Chappelle said something that is so spot-on. He said that comedians are how people metabolize things.

[Audio] We’re like the nation’s kidney. We help everyone metabolize not just facts, but feelings around facts, or ideas. And jokes are just a shorthand for all of that.

MA: And I’m also a nerd and I’ve been reading some Greek tragedies and studying about how, you know, that’s also like the artists that help people metabolize feelings and emotions. And when he said that, I was like, that’s exactly it. And that’s what Jules has done.

It’s something that heals. And it’s not even like it’s self-deprecating. It’s not like, oh, let’s make fun. And the way he does it is so smart and genuine, but also not taking himself seriously. It just disarms everything. You’re just like, oh, OK. And that idea of metabolizing the really hard feelings that come around grief, mortality, losing a person, seeing how he’s trapped in his body, but he chooses not to be trapped in that really big way is so profound.

Sanders: Science is iterative. It builds. It may take a wrong turn or two, but overall, it’s moving toward a deeper understanding of our world and of ourselves.

By using their voices, by telling their stories, by telling their jokes, Jules and MA are helping me, and maybe you too, metabolize the pain – and the joy – of living. Welcome to The Deep End, Talk to Me.

[Voices] Hey mom, can I tell you something? Hi. Do you want to talk? That added layer of it being you. Hello? Your voice. Hi.

Sanders: I’m Laura Sanders and today, we’re going to look ahead to what’s coming next for the human voice and efforts to restore it.

[Voice] Your voice is you.

Sanders: In some ways, this podcast is a story of progress, of figuring out, little by little, more about the brain, and more about technology and more about people. Today, Jules can type with his eyes, and his son can hear his dad’s voice say I love you. In earlier episodes, you heard from Casey Harrell. Casey can think about saying a word to his daughter, and boom, they’re off playing a game together.

Casey Harrell: So this has kept me still in touch with people and the world, and it is not hyperbole to frame it this way.

Sanders: Casey has ALS and can no longer speak. He volunteered for a clinical trial, and now he has a device implanted in his brain that detects neural patterns as he tries to say a word. This system has changed his life.

Casey: It is the ability to parent my daughter. It is the ability to connect with my wife. It is the ability to make a wry joke with my friends. Timing is everything, and that is why having nothing, this is completely revolutionary.

Sanders: But there aren’t many people like Casey. Leigh Hochberg is the director of the BrainGate clinical trials consortium, the trial that Casey is participating in. I asked him how many people might have participated in similar sorts of research projects. How many people have electrodes in their brain that can help them speak?

Leigh Hochberg: In terms of actually decoding attempted speech, I’m almost sure it’s still less than 10. It may be closer to five or six, so far, and that number undoubtedly is going to increase fairly quickly.

Sanders: So this is a small but growing number. Growing, because we now have a solid scientific understanding of the brain’s speech signals, and other signals too. And neurotech companies are very interested.

Hochberg: About 10 years ago or so, the academic research had de-risked this field sufficiently that it allowed for the entry of now likely a dozen or more companies that are actively pursuing their own version of an implantable brain-computer interface.

What I anticipate will happen, what I guess I hope will happen, is that the companies will continue to very carefully, thoughtfully and in a very focused manner, create their systems, and test them rigorously. And hopefully they’ll be demonstrated to be safe and effective. And at the same time, the academic groups have the opportunity to continue to demonstrate what’s possible, to push beyond that, and to show hopefully what those medical devices, once they’re actually available to market, might be able to do next.

Sanders: I talked with Vikash Gilja, a neural engineer who works at a company called Paradromics. He sees this momentum that Hochberg describes too.

Vikash Gilja: And, you know, we can see all the pieces coming together. These tools weren’t available even five years ago. And what we’re seeing is like, you know, month over month, there are new tools available. So we’re able to leverage those and move faster.

Sanders: Paradromics has developed its own brain-computer interface for communication. The device is about the size of a dime, and it sits on the motor cortex. That’s the part of the brain that creates speech signals. The chip then detects these signals as a person thinks about saying a word.

Matthew Willsey: So the cortical module has 421 tiny little sensors.

Sanders: That’s neurosurgeon and biomedical engineer Matthew Willsey, of the University of Michigan Health. He’s describing the device from Paradromics here.

Willsey: They’re about one and a half millimeters long. And you know, it’s smaller than your thumbnail, like kind of all the way around. It’s like a cylinder. And at the bottom of the cylinder are the electrodes, is what we call them. And we get them really close to the brain. And then when we deploy it, what it does is it pushes those electrodes down into the brain.

Sanders: Those tiny sensors are listening to the nearby neurons that control the muscles required for speech.

Willsey: The principle is to try and get these things as small as possible to disrupt the least amount of tissue as possible, and then to get them really, really close to the individual cells, the neurons themselves that have the information we’re trying to record.

Sanders: The goal here is to restore a person’s ability to talk.

Willsey: So in the ideal situation, the electrodes would sit, you know, right next to those neurons. This type of technology that’s very intuitive, that can record activity from the brain in kind of a high-throughput way, can put the words up at a good speed and a good accuracy, allows them to kind of regain the ability to communicate with others close to how they maybe were before they had the illness. So that truly is very powerful.

Sanders: You might remember that Casey’s implant, called the BrainGate 2 system, requires ports on his head that are connected to external wires.

Casey: What is annoying about this device is really easy to answer. It is not portable, and it is not implanted only inside my head.

Sanders: Unlike Casey’s implant, the Paradromics device is fully implantable. That means it’s wireless. This summer, Willsey implanted Paradromic’s device in a volunteer’s brain. The brain belonged to a 68-year-old woman who lives in Michigan. She has primary lateral sclerosis, a disease that has taken away most of her ability to talk. So far, the Paradromics system is up and running, and the volunteer is testing her abilities out.

Here she is talking with Willsey and other researchers in a video provided by Paradromics. You’ll hear her natural voice, which is hard to understand, and then a digital voice speaking the words her brain implant decoded.

[Audio] I love the change of seasons in Michigan. And my favorite one is the fall. I love the change of seasons in Michigan. And my favorite one is the fall. Yeah, it’s good.

Sanders: This volunteer has used the device to talk to her grandkids on the phone. But this is just the very beginning for this research project. There’s so much more to test.

Casey is still working on his system, too — particularly, streamlining the process. When Casey uses his brain-computer interface, it decodes speech signals from his brain, then puts words on a screen. Casey then selects a button, using his eye movements or his brain-computer interface, to say these words out loud with a digital voice. Here’s Casey.

Casey: It is not fast enough to provide enough instant conversation with other people that need to slow down to talk to me.

Sanders: But scientists want to skip this middle part — the words written on a screen. They want instead to go straight to speech, and perhaps make conversations faster. Casey says this goal to go directly from brain signals to a speaking digital voice would be his top priority.

Casey: That one is the holy grail of what this will become in terms of speech. It will allow you to have a seamless conversation. I am talking about, yes, I will intend to talk, and then it will just translate that automatically into speech, like we do, and you are doing right now.

Sanders: In an earlier episode, we talked about attempts to do just this with Casey’s system, to push it to do what they call brain-to-voice. Early attempts were hard to decipher. But it has already improved, by a lot. In the latest tests, this brain-to-voice process produces words that are understandable – reaching about 95 percent accuracy for listeners. Here’s Casey speaking test sentences. It comes from a preprint manuscript posted on the website bioRxiv.

[Audio] The day is not over yet. Please come in with me.

Sanders: Scientists have another thing on their to-do list. And this one goes beyond the words themselves. Here’s Gilja, from Paradromics.

Gilja: Let’s say I identify whether a sentence is a statement or a question. Well, now you can pass that information on to speech synthesis, so that you get these two different versions.

Sanders: The same words could be a statement or a question. You see what he means. You see what he means? And that information can be pulled out from a person’s brain activity.

Gilja: If I’m decoding prosodic information, if I’m decoding speed, when a word is meant to appear, that can also be passed on to modern speech synthesis systems, so that you’re more accurately capturing the prosody that somebody intends. Same thing with loudness, right? So you can see how as that information becomes available from the neural channels, it can be provided to the AI speech synthesis to get closer to the intended voice in the moment that the individual wants to generate.

Sanders: All of these signals can be collected and used to build a more accurate voice — one that says things exactly how a person intends to say them, like a Minnesotan’s ooftah or a slow southern hey y’all. Now, today’s voice clones can add these emotional layers onto words already. Here’s what Jules told me when I asked him about whether his voice clone can do it.

Jules: Oh, there are plenty of things it does that are all the above. But one thing in particular is how it inserts tone and inflections on words. Sometimes my wife will get mad because my response can come off with some sass.

Sanders: So there’s loudness, emphasis, pitch — all of these things contribute to what words means. But imagine if a brain implant can pick up these cues directly — no typing required. Here’s Hochberg.

Hochberg: There’s a lot that can be conveyed through the emphasis, through the tone, through even the humor that can sometimes be conveyed by the way that somebody who’s able-bodied may speak a particular sentence. All of those, I think, really can be captured by these implantable brain-computer interfaces by harnessing the rich neural activity that’s available from the brain. We should be able to restore a lot of that.

Sanders: Sergey Stavisky is one of the UC Davis researchers working with Casey and Hochberg. He and his colleagues are working on some of these ideas.

Stavisky: And even now we’re continuing to improve that and actually adding additional layers. So we have one forward-looking project where we can maybe decode the kind of emotional valence of how he’s trying to say it. Is he trying to say something in an angry way or in a surprised way or in a happy-sounding way, to kind of further make it more expressive.

Sanders: They’re even trying to decode singing. Well, OK, it’s really more of a hum at this point, he says. But those sorts of tone shifts could, in theory, be deciphered.

[Audio] Hums.

Stavisky: We definitely have our work cut out for us on the singing and melody side of things.

[Audio] Hums.

Sanders: All sorts of voice signals could be pulled from brain activity.

Stavisky: Ultimately, we want people to be in full control of what they’re communicating. This should come from their intent. So, yeah, we are trying to decode all of the aspects of his voice, whether it’s the pitch, the timbre, the loudness, the intonation, which word he’s stressing. All of that, ideally, we would decode from brain activity, including kind of all of that emotional nuance.

Sanders: Voice clones can speak with various emotions. You heard my voice clone do just that in earlier episodes. But the difference here is that a person wouldn’t have to type in the instructions. They would come straight from their brain, seamlessly. Other nonverbal signals can be pulled from the brain, too.

Samantha Brosler: In the past five years or so, there have been a lot of advancements in the field of BCIs for restoring individual functions to people with paralysis.

Sanders: That’s Samantha Brosler, a bioengineer at the University of California, San Francisco. Some brain-computer interfaces can pick up movements. We haven’t talked much about them, but these implants have their own very successful research history. They’ve allowed people to control a robotic arm, for instance.

Brosler: Most of these devices so far have been designed to restore one function at a time, whereas in reality, frequently people who are paralyzed are missing both the ability to speak and perform body movements. And so in this study, we asked whether a single brain implant could be used to decode both speech and movement at the same time.

Sanders: It turns out, it could. She and her colleagues were able to decode gestures for communication — things like thumbs up, or a head nodding yes — and words. This combo approach is a more natural way of getting a point across.

Brosler: We naturally use gestures alongside our speech. So for example, some gestures can substitute for speech entirely, like nodding yes, shaking your head no, waving, shrugging, giving a thumbs up. But they can also add or change the meaning of what we’re saying. So for example, if you say maybe combined with nod head yes, that has a very different, you know, meaning than if you say maybe and shrug or shake your head no.

Sanders: My kids know that a slow maybe with a skeptical eyebrow raise is much closer to a no than my fast upbeat maybe with a little head nod. There is so much subtext in our body language.

Brosler: So I think of, you know, gestures and body movements as an additional communication channel that can work together with speech to add expressivity to what we’re trying to say.

Sanders: In the study, the output was a digital avatar who looked like the person. Success looked like the avatar moving and getting the word right.

Brosler: And the idea is that we enable the participants to use speech and gestures alone or together. So in theory, someone could say hello while waving.

Sanders: The researchers started simple, with just a few phrases — hello, bye, good idea, how’s it going, let’s go — and just a few gestures, like a shrug or a point, a fist pump, thumbs up.

Brosler: Of course, ultimately, we’re hoping to expand this to even larger vocabulary and more range of body movements across the whole body to help allow people more expression in their communication.

Sanders: The sheer number of different approaches to restoring communication is a sign that this neurotechnology is coming fast. Here’s Willsey.

Willsey: I think it’s good. There’s a lot of players that are entering into the field that have a variety of different ideas where they can run the clinical trials and see how these different devices stack up.

Sanders: There are lots of reasons to be optimistic about the future here, and it’s a future that isn’t even that far off. Here’s David Brandman, one of the researchers at UC Davis who works with Casey. He’s also the neurosurgeon who implanted Casey’s device.

Brandman: I think I’m about five years away from being able to meet a patient with ALS and having the following conversation, and saying, you know, Ms. Smith, I’m sorry you’ve been diagnosed with ALS. I don’t have a cure for you, but I do have a device that I can implant that can help you communicate despite facing a future of paralysis. It’s a device that would be implanted. It’s fully wirelessly. It’s cosmetically invisible, and you’d be able to use it to communicate your needs and have a meaningful conversation with friends and family. So I think that kind of conversation is about five to 10 years away.

Sanders: It’s a beautiful vision. When you’ve been a reporter as long as I have, you’re skeptical when scientists say something big is just five to 10 years away. But in this case, I see it.

It’s easy to get drawn into the technical details here. Different types of implants, the decoding processes, AI-powered voice clones and what they can do and can’t do — what they should do and shouldn’t do. But if you step back and really look, you’ll see that this research is all propelled by a very simple and powerful fact: The human voice is too important to lose.

Casey: What does my voice mean to me? It means so much. I think it is the most important thing that I have lost in this whole disease.

Sanders: So now we are here, in this moment, as Jules would say, right here, right now. And right now, the human voice is shifting and changing, expanding in ways that were completely unthinkable even just a few years ago. The future of the human voice and its many replicas is anybody’s guess.

Sanders: In one of Jules’ comedy sets, the one he did in November of 2024, he wheels out onto the stage as the song “Walking on Sunshine” plays.

As the crowd settles and finishes their clapping and woo-ing, Jules pauses. You can’t see this, but he wiggles his eyebrows — a lot. The crowd goes crazy every time. He hasn’t said a single word yet. After another pause, he deadpans with his voice clone: “That is all the time I have. My Make-A-Wish covered only a few minutes. Goodnight, everybody.”

And with that, Jules takes control and he’s off on a tear, ripping through stories of a weatherman who seems too happy about standing out in hurricanes, the exquisite guilt trips lovingly crafted by his Cuban parents, and the indignities of having body parts shaved with his sister right there in the room. Jules has the audience laughing so hard. It’s magnificent.

MA: And being backstage and then seeing him come out, like roll out in his wheelchair and everybody go nuts because they’re like, what is going on here? Like, this doesn’t make sense. And every time I was like, this is like life. This is love. This is passion. This is like, what else is there? Like, I don’t know that there’s anything else more powerful than what this guy is doing here and what I can help him do.

Sanders: I get what MA meant when she said that comedy is something that heals. When his set actually does come to a close, he tells the audience.

[Audio] Now you have to watch me slowly get off this stage. Goodnight, everybody.

Sanders: It’s hilarious. It’s happy. It’s sad. It’s perfect.

This is The Deep End, Talk to Me. I’m Laura Sanders. If you like this podcast, tell your friends and leave us a review. It helps the show a lot. Send us your questions and your comments at podcasts@sciencenews.org.

The Deep End is a production of Science News. It’s based on original reporting by me, Laura Sanders. This episode was produced and mixed by Ella Rowen. Our project manager is Ashley Yeager. Nancy Shute is our editor in chief. Our music is by Blue Dot Sessions. The podcast is made possible in part by the Burroughs Wellcome Fund, with support from PRX.

This podcast was produced with support from the Burroughs Wellcome Fund and in partnership with PRX. The podcast logo was created by Neil Webb.