Data Drop: Pehuén Moure on Learning to Control Artificial Vision Through Brain Activity

Interviewer (Michael Beyeler): The paper moves from predicting what electrical stimulation does to the brain toward actually controlling the resulting neural activity. Why is that extra step important for a visual prosthesis?

Dr. Moure: Predicting the response to electrical stimulation tells us how the brain is reacting to a given pattern; in this study we called this the forward problem. But what we ultimately care about is the reverse. We want to choose a stimulation pattern that produces a particular neural response, which means inverting the forward model. That inverse problem is highly nonlinear, and it depends on what the brain is doing on a given day.

Targeting the neural response lets us close the loop in a self-supervised manner. Most modern multi-electrode arrays can stimulate and record at the same time, so the device can measure how close it got to the target on every trial without having to ask the person what they saw. For visual neuroprosthesis this helps solve an important calibration problem, as the recordings drift we can adapt the models automatically to these changes.

Interviewer: You had stimulation parameters, simultaneous population recordings, and perceptual reports from the same participant. What did putting those three pieces together reveal about the link between what we stimulate, what the cortex does, and what a person actually sees?

Dr. Moure: We tried predicting what the person had reported from each of the three pieces. The neural responses predicted the reports better than the stimulation parameters did, for whether he saw a phosphene at all and for its brightness and color. Intuitively this makes sense: as the brain drifts, the percept follows the cortex, not the stimulation input.

The models also did better the more we gave them. Combining the stimulation, the resting activity, and the evoked response gave the best predictions. But most of the gain came from the neural activity, so that is the piece we believe is worth targeting.

Interviewer: The evoked responses were complex and state-dependent, but they were also confined to a relatively low-dimensional manifold. How should we think about that tension? Does it make the cortex easier to control, or does it reveal hard limits on what electrical stimulation can produce?

Dr. Moure: The latent factor analysis does show that the evoked activity is lower-dimensional than the stimulation. That tells us some target responses may simply not be reachable, and we saw this directly. The further a target sat from that low-dimensional space, the harder it was to evoke.

So it does limit what we can produce, but it also makes the problem more tractable. The model can leverage the fact that many different stimulation patterns produce the same cortical output and find a stimulation pattern that better fits the needs of the implant. What is still open is where the limit comes from; it could be a property of cortical electrical stimulation itself, things like current spread or the underlying circuitry.

Interviewer: When you let the model search for stimulation patterns that would drive the cortex toward a target response, did it find anything you would not have designed by hand? What did you learn from the solutions it came up with?

Dr. Moure: One thing we saw is that the model-based methods reached their targets with lower overall current than the conventional approaches. It does have a clear clinical upside, since we generally want to minimize the charge delivered, though current alone did not explain why these patterns worked better. The other lesson was about the forward model itself. Its simulations were optimistic, but they ranked targets the same way the in vivo results did, so it works as a meaningful simulator to optimize over before evaluating in vivo.

Interviewer: So what comes next for you? What would you most like to test or build on from this work?

Dr. Moure: I'm most interested in how we keep improving calibration and personalization for models used in neural interfaces. One opportunity is sampling more efficiently on a new day. Rather than collecting a large dataset, we could use the model's uncertainty or the baseline activity from that morning to pick the stimulation patterns that would lead to the most improvement. The other is to bring perception into the loop directly. Right now we optimize for a target neural response, but if the perception decoders become part of the optimization, we can target what the person actually sees and fully close the loop. That same question of how a deep learning model adapts to an individual in biological applications is what I'm pursuing in my postdoc at Cornell Tech.