Controlling Biological Neuronal Networks with Reinforcement Learning
Description see below. "Click" to view full-size image.
“The highly dynamic nature of neuronal networks and changes with disease progression create an urgent need of a change in the stimulation control from an open to a closed-loop system”, is PhD student Jan Wülfing from the Neurorobotics Lab convinced. This means: The control system does not only send electrical pulses into the brain, but also receives feedback about possible changes in brain activity after stimulation and is then able to react to them. Essential prerequisite for this: The controller can autonomously learn. But how?
Reinforcement Learning (RL) is a promising tool to address such challenges, but has rarely been used for the long-term control of live, plastic neural networks yet. This is a difficult problem since it requires the controller to adapt to poorly characterized non-stationary background processes that alter stimulus-response relations over time.
The research team captured these challenges. Jan Wülfing and his colleague Dr. Sreedhar Kumar from the Bernstein Center Freiburg and IMTEK have succeeded in showing that reinforcement learning methods are indeed able to control certain features of the activity in biological neural networks to a predefined target level and can also autonomously adjust the control when the network changes its activity. For this, they defined a novel control task as clamping response strengths to predefined levels over long durations in a living model system, namely generic BNNs in vitro grown on microelectrode arrays.
Their results: “We can show that by defining appropriate state-action spaces and employing powerful nonlinear RL methods such as Deep RL, adaptivity to non-stationary background processes can be achieved in a series of experiments”, Jan Wülfing says. In 27/29 networks, the learned controllers were able to improve performance compared to a random and a (linear) controller by a large margin. “It would be possible that these findings could be a step forward to optimize the control of deep brain stimulation in future”, sums the young scientist up.
Deep brain stimulation
To influence brain activity through electrical impulses via implanted electrodes
Learning by positive or negative reinforcement
The learning organism or system is rewarded for actions that are wanted. Actions that are not wanted are ignored or have negative consequences.
Figure legend
Activity features of biological neural networks are controlled by means of electrical stimulation in a closed loop. The control law is learned with Reinforcement Learning methods.
Original Publication
Jan M. Wülfing, Sreedhar S. Kumar, Joschka Boedecker, Martin Riedmiller, Ulrich Egert, Adaptive Long-term Control of Biological Neural Networks with Deep Reinforcement Learning, Neurocomputing, 2019, ISSN 0925-2312.
Read also
Artificial neural networks learn to control biological neuronal networks
Contact
Jan Wülfing
University of Freiburg
Department of Computer Science
Neurorobotics Lab
Georges-Koehler-Allee 080
79110 Freiburg im Breisgau
E-mail: wuelfj@informatik.uni-freiburg.de
Dr. Sreedhar Saseendran Kumar
via
Prof. Dr. Ulrich Egert
University of Freiburg
Bernstein Center Freiburg &
Biomicrotechnology
Dept. for Microsystems Engineering, IMTEK
Georges-Koehler-Allee 102
79110 Freiburg
Tel .: +49 761 203-7524
E-mail: egert@imtek.uni-freiburg.de