Extended Data Fig. 5: Real-time extraction of brain-wide neuronal activities and ensemble-triggered virtual reality experiments.

a, Experimental design of ensemble activity-triggered feedback control. A reference frame was generated by averaging 50 frames, and regions of interest (ROIs) corresponding to neurons across the imaged plane were segmented using a watershed algorithm. The larva then experienced locomotor-triggered virtual reality, while activities were extracted in real-time (RT) for each ROI. K-means clustering were performed to group the neurons into distinct clusters within 10 min. Different weights were assigned to each neuron based on similarity of their activities to the cluster mean, and the weighted average from all neurons were calculated to represent the ensemble activity. A seed cluster, with the ensemble activity correlated with the swimming were selected to trigger VR (red dash box). In ensemble activity-triggered virtual reality experiment, the ensemble activity (red trace) directly controlled the grating velocity presented to the zebrafish. Neuronal activities from all the groups were displayed and updated in real time during the experiment (inset). OMR, optomotor response. b, Anatomical locations of the neurons composing each cluster are shown in the plane in focus. The cluster identity is color-coded. Scale bar, 100 μm. c, Activities from all neurons across the imaging plane for VR, ordered by the cluster ID. d, Anatomical locations of the neurons composing the motor-related seed cluster (located in the cerebellum, CB, and hindbrain, HB), and a randomly selected neuron ensemble with the same number of neurons as the seed cluster are shown using different colors. Scale bar, 100 μm. e,f, Gain adaptation in closed-loop control by the population neuronal activity of the seed cluster (e), and the randomly selected ensemble (f). The frequency and amplitude of the seed cluster neuronal activities (upper and middle panels) in the low gain condition were significantly larger than those in the high gain condition (f). In contrast, such modulation effect was not absent with the closed-loop control by the randomly selected ensemble (f). g,h, There was no significant difference in the modulation of activity frequency (Frequency ratio between low and high gain, g) or grating velocity (Grating velocity ratio between high and low gain, h) between the neuronal ensemble activity-triggered (“Neuron”) and locomotor-triggered (“Locomotor”) closed-loop controls. Two-sided paired t-test, n = 9 fish. In the box plots, the central lines mark the medians, the box limits mark the upper and lower quartiles, and the whiskers mark the ±1.5× interquartile range. Data from each individual fish were marked with filled circles and connected with a line.