Fig. 5: The deep behavioral features have highly-structured dynamics.

a, Example dynamics of deep behavioral features computed by the neural network in Fig. 3a. The features have been sorted along the y axis using a one-dimensional t-SNE embedding⁴⁵. b, Inferred states using an HMM. c, Reconstructed features using the inferred states on a test trial. d, State transition matrix of the HMM. Self-transitions were set to 0 and the rows were renormalized to 1. States have been sorted to maximize the sum of transition probabilities above the diagonal, using the Rastermap algorithm⁴⁶. e, Same as d for HMMs fit directly to the keypoint data. f, Simulated states using the HMM fit to the deep behavioral features. g, Simulated features from the HMM. h, Distribution of inferred state lifetimes using the self-transition probabilities of the HMM, averaged across n = 5 recordings from five mice, and error bars represent s.e.m. See Methods for description of controls for all panels. i, Probability of transitions to n-nearest states as a function of n. The average is taken over all initial states, and the line represents the average across n = 5 recordings from five mice. j, Schematic for k–m. For each pair of states with a high transition probability, certain other transition probabilities are reported. Each dot represents transitions from a different animal, averaged across all high-probability pairs. k, Average probability of reverse transitions. Baseline is computed as the average transition probability across all state transitions. l, Probability of two-state transitions. m, Probability of two-state backward transitions. n, Distribution of ‘forward’ sequence lengths, where the forward direction is defined as higher indices in the Rastermap sorting of states from d, averaged across n = 5 recordings from five mice, and error bars represent s.e.m. o, Neural populations tuned to 19 selected states (of 50 total). The top 300 most selective neurons were chosen on train trials, and their average on test trials is shown. Vertical lines indicate trial onsets, while the second jagged line indicates trial offsets.