Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Multi-view manifold learning of human brain-state trajectories

Nature Computational Science volume 3, pages 240–253 (2023)Cite this article

Subjects

A preprint version of the article is available at bioRxiv.

Abstract

The complexity of the human brain gives the illusion that brain activity is intrinsically high-dimensional. Nonlinear dimensionality-reduction methods such as uniform manifold approximation and t-distributed stochastic neighbor embedding have been used for high-throughput biomedical data. However, they have not been used extensively for brain activity data such as those from functional magnetic resonance imaging (fMRI), primarily due to their inability to maintain dynamic structure. Here we introduce a nonlinear manifold learning method for time-series data—including those from fMRI—called temporal potential of heat-diffusion for affinity-based transition embedding (T-PHATE). In addition to recovering a low-dimensional intrinsic manifold geometry from time-series data, T-PHATE exploits the data’s autocorrelative structure to faithfully denoise and unveil dynamic trajectories. We empirically validate T-PHATE on three fMRI datasets, showing that it greatly improves data visualization, classification, and segmentation of the data relative to several other state-of-the-art dimensionality-reduction benchmarks. These improvements suggest many potential applications of T-PHATE to other high-dimensional datasets of temporally diffuse processes.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: T-PHATE procedure.
Fig. 2: Visual validation of T-PHATE embeddings.
Fig. 3: Alternative embeddings featuring time.
Fig. 4: HMM procedure and analysis.
Fig. 5: Evaluating event segmentation within-subject.
Fig. 6: Out-of-sample event boundary fit.

Similar content being viewed by others

Data availability

The Sherlock dataset was downloaded from the Dataspace Public Repository at the following link: http://arks.princeton.edu/ark:/88435/dsp01nz8062179. The StudyForrest dataset was accessed via DataLad68 from: https://github.com/psychoinformatics-de/studyforrest-data. Steps to reproduce our preprocessing pipeline and ROI extraction are available here: https://github.com/ericabusch/tphate_analysis_capsule (ref. 69). Source Data are provided with this paper.

Code availability

Data analysis code is written as custom Python scripts (v.3.6.13) based on sci-kit learn v.0.23.2 (https://scikit-learn.org/), nilearn v.0.9.2 (https://nilearn.github.io), nibabel v.4.0.1 (https://github.com/nipy/nibabel), PHATE v.1.0.7 (https://phate.readthedocs.io/en/stable/)28 and Brainiak v.0.11 (https://brainiak.org/)62. T-PHATE is available as a Python package at: https://github.com/KrishnaswamyLab/TPHATE (ref. 70). The pipeline to replicate all of the analyses presented here is available at: https://github.com/ericabusch/tphate_analysis_capsule (ref. 69).

References

  1. Averbeck, B. B., Latham, P. E. & Pouget, A. Neural correlations, population coding and computation. Nat. Rev. Neurosci. 7, 358–366 (2006).

    Article  Google Scholar 

  2. Laurent, G. Olfactory network dynamics and the coding of multidimensional signals. Nat. Rev. Neurosci. 3, 884–895 (2002).

    Article  Google Scholar 

  3. Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Ryu, S. I. & Shenoy, K. V. Cortical preparatory activity: representation of movement or first cog in a dynamical machine? Neuron 68, 387–400 (2010).

    Article  Google Scholar 

  4. Chang, L. & Tsao, D. Y. The code for facial identity in the primate brain. Cell 169, 1013–1028.e14 (2017).

    Article  Google Scholar 

  5. Freiwald, W. A. & Tsao, D. Y. Functional compartmentalization and viewpoint generalization within the macaque face-processing system. Science 330, 845 (2010).

    Article  Google Scholar 

  6. Jazayeri, M. & Ostojic, S. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Curr. Opinion Neurobiol. 70, 113–120 (2021).

  7. Hennig, J. A. et al. Constraints on neural redundancy. eLife 7, e36774 (2018).

    Article  Google Scholar 

  8. Nieh, E. H. et al. Geometry of abstract learned knowledge in the hippocampus. Nature 595, 80–84 (2021).

    Article  Google Scholar 

  9. Cunningham, J. P. & Yu, B. M. Dimensionality reduction for large-scale neural recordings. Nat. Neurosci. 17, 1500–1509 (2014).

    Article  Google Scholar 

  10. Norman, K. A., Polyn, S. M., Detre, G. J. & Haxby, J. V. Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends Cogn. Sci. 10, 424–430 (2006).

    Article  Google Scholar 

  11. Cox, D. D. & Savoy, R. L. Functional magnetic resonance imaging (fMRI) ‘brain reading’: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage 19, 261–270 (2003).

    Article  Google Scholar 

  12. Kamitani, Y. & Tong, F. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685 (2005).

    Article  Google Scholar 

  13. Haxby, J. V. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).

    Article  Google Scholar 

  14. Haynes, J.-D. & Rees, G. Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7, 523–534 (2006).

    Article  Google Scholar 

  15. Polyn, S. M., Natu, V. S., Cohen, J. D. & Norman, K. A. Category-specific cortical activity precedes retrieval during memory search. Science 310, 1963–1966 (2005).

    Article  Google Scholar 

  16. Peelen, M. V., Atkinson, A. P. & Vuilleumier, P. Supramodal representations of perceived emotions in the human brain. J. Neurosci. 30, 10127–10134 (2010).

    Article  Google Scholar 

  17. Yeshurun, Y., Nguyen, M. & Hasson, U. Amplification of local changes along the timescale processing hierarchy. Proc. Natl Acad. Sci. USA 114, 9475–9480 (2017).

    Article  Google Scholar 

  18. Davatzikos, C. et al. Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. NeuroImage 28, 663–668 (2005).

    Article  Google Scholar 

  19. Birn, R. M., Smith, M. A., Jones, T. B. & Bandettini, P. A. The respiration response function: the temporal dynamics of fMRI signal fluctuations related to changes in respiration. NeuroImage 40, 644–654 (2008).

    Article  Google Scholar 

  20. Turk-Browne, N. B. Functional interactions as big data in the human brain. Science 342, 580–584 (2013).

    Article  Google Scholar 

  21. Gao, S., Mishne, G. & Scheinost, D. Nonlinear manifold learning in functional magnetic resonance imaging uncovers a low-dimensional space of brain dynamics. Hum. Brain Mapp. 42, 4510–4524 (2021).

    Article  Google Scholar 

  22. Olszowy, W., Aston, J., Rua, C. & Williams, G. B. Accurate autocorrelation modeling substantially improves fMRI reliability. Nat. Commun. 10, 1220 (2019).

    Article  Google Scholar 

  23. Golesorkhi, M. et al. The brain and its time: intrinsic neural timescales are key for input processing. Commun. Biol. 4, 1–16 (2021).

    Article  Google Scholar 

  24. Ito, T., Hearne, L. J. & Cole, M. W. A cortical hierarchy of localized and distributed processes revealed via dissociation of task activations, connectivity changes, and intrinsic timescales. NeuroImage 221, 117141 (2020).

    Article  Google Scholar 

  25. Shine, J. M. et al. The dynamics of functional brain networks: integrated network states during cognitive task performance. Neuron 92, 544–554 (2016).

    Article  Google Scholar 

  26. Shine, J. M. et al. Human cognition involves the dynamic integration of neural activity and neuromodulatory systems. Nat. Neurosci. 22, 289–296 (2019).

    Article  Google Scholar 

  27. Allen, E. A. et al. Tracking whole-brain connectivity dynamics in the resting state. Cerebral Cortex 24, 663–676 (2012).

    Article  Google Scholar 

  28. Moon, K. R. et al. Visualizing structure and transitions in high-dimensional biological data. Nat. Biotechnol. 37, 1482–1492 (2019).

    Article  Google Scholar 

  29. Tenenbaum, J. B., De Silva, V. & Langford, J. C. A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000).

    Article  Google Scholar 

  30. Roweis, S. T. & Saul, L. K. Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000).

    Article  Google Scholar 

  31. Salhov, M., Bermanis, A., Wolf, G. & Averbuch, A. Approximately-isometric diffusion maps. Appl. Comput. Harmon. Anal. 38, 399–419 (2015).

    Article  MathSciNet  MATH  Google Scholar 

  32. Becht, E. et al. Dimensionality reduction for visualizing single-cell data using UMAP. Nature Biotechnol. 37, 38–44 (2019).

    Article  Google Scholar 

  33. Gotts, S. J., Gilmore, A. W. & Martin, A. Brain networks, dimensionality, and global signal averaging in resting-state fMRI: hierarchical network structure results in low-dimensional spatiotemporal dynamics. NeuroImage 205, 116289 (2020).

    Article  Google Scholar 

  34. Casanova, R. et al. Embedding functional brain networks in low dimensional spaces using manifold learning techniques. Front. Neuroinform. 15, 740143 (2021).

  35. Mannfolk, P., Wirestam, R., Nilsson, M., Ståhlberg, F. & Olsrud, J. Dimensionality reduction of fMRI time series data using locally linear embedding. Magn. Res. Mater. Phy. 23, 327–338 (2010).

    Article  Google Scholar 

  36. Van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).

  37. Radvansky, G. A. & Zacks, J. M. Event boundaries in memory and cognition. Curr. Opin. Behav. Sci. 17, 133–140 (2017). Memory in time and space.

    Article  Google Scholar 

  38. Zacks, J. M., Speer, N. K., Swallow, K. M. & Maley, C. J. The brain’s cutting-room floor: segmentation of narrative cinema. Front. Hum. Neurosci. 4, 168 (2010).

    Article  Google Scholar 

  39. Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S. & Reynolds, J. R. Event perception: a mind-brain perspective. Psychol. Bull. 133, 273–293 (2007).

    Article  Google Scholar 

  40. Zacks, J. M., Tversky, B. & Iyer, G. Perceiving, remembering, and communicating structure in events. J. Exp. Psychol. Gen. 130, 29–58 (2001).

    Article  Google Scholar 

  41. Kurby, C. A. & Zacks, J. M. Segmentation in the perception and memory of events. Trends Cogn. Sci. 12, 72–79 (2008).

    Article  Google Scholar 

  42. Baldassano, C. et al. Discovering event structure in continuous narrative perception and memory. Neuron 95, 709–721.e5 (2017).

    Article  Google Scholar 

  43. Lee, C. S., Aly, M. & Baldassano, C. Anticipation of temporally structured events in the brain. eLife 10, e64972 (2021).

    Article  Google Scholar 

  44. Baldassano, C., Hasson, U. & Norman, K. A. Representation of real-world event schemas during narrative perception. J. Neurosci. 38, 9689–9699 (2018).

    Article  Google Scholar 

  45. MacDonald, I. L. & Zucchini, W. Hidden Markov and Other Models for Discrete-Valued Time Series Vol. 110 (CRC, 1997).

  46. Yates, T. S. et al. Neural event segmentation of continuous experience in human infants. Proc. Natl Acad. Sci. USA 119, e2200257119 (2022).

    Article  Google Scholar 

  47. Speer, N. K., Zacks, J. M. & Reynolds, J. R. Human brain activity time-locked to narrative event boundaries. Psychol.Sci. 18, 449–455 (2007).

    Article  Google Scholar 

  48. DuBrow, S. & Davachi, L. The influence of context boundaries on memory for the sequential order of events. J. Exp. Psychol. Gen. 142, 1277–1286 (2013).

    Article  Google Scholar 

  49. DuBrow, S. & Davachi, L. Temporal binding within and across events. Neurobiol. Learn. Mem. 134, 107–114 (2016).

    Article  Google Scholar 

  50. Ezzyat, Y. & Davachi, L. Similarity breeds proximity: pattern similarity within and across contexts is related to later mnemonic judgments of temporal proximity. Neuron 81, 1179–1189 (2014).

    Article  Google Scholar 

  51. Chen, J. et al. Shared memories reveal shared structure in neural activity across individuals. Nat. Neurosci. 20, 115–125 (2017).

    Article  Google Scholar 

  52. Hasson, U., Yang, E., Vallines, I., Heeger, D. J. & Rubin, N. A hierarchy of temporal receptive windows in human cortex. J. Neurosci. 28, 2539–2550 (2008).

    Article  Google Scholar 

  53. Zacks, J. M. et al. Human brain activity time-locked to perceptual event boundaries. Nat. Neurosci. 4, 651–655 (2001).

    Article  Google Scholar 

  54. Haxby, J. V., Connolly, A. C. & Guntupalli, J. S. Decoding neural representational spaces using multivariate pattern analysis. Ann. Rev. Neurosci. 37, 435–456 (2014). PMID: 25002277.

    Article  Google Scholar 

  55. Rodosthenous, T., Shahrezaei, V. & Evangelou, M. S-multi-SNE: semi-supervised classification and visualisation of multi-view data. Preprint at https://arxiv.org/abs/2111.03519 (2021).

  56. Rodosthenous, T., Shahrezaei, V. & Evangelou, M. Multi-view data visualisation via manifold learning. Preprint at https://arxiv.org/abs/2101.06763 (2021).

  57. Kuchroo, M. et al. Multiscale phate identifies multimodal signatures of covid-19. Nat. Biotechnol. 40, 681–691 (2022).

    Article  Google Scholar 

  58. Moon, K. R. et al. Manifold learning-based methods for analyzing single-cell RNA-sequencing data. Curr. Opin. Syst. Biol. 7, 36–46 (2018).

    Article  Google Scholar 

  59. Himberger, K. D., Chien, H.-Y. & Honey, C. J. Principles of temporal processing across the cortical hierarchy. Neuroscience 389, 161–174 (2018). Sensory Sequence Processing in the Brain.

    Article  Google Scholar 

  60. Brockwell, P. J. & Davis, R. A. Introduction to Time Series and Forecasting (Springer, 2002).

  61. Shin, Y. S. & DuBrow, S. Structuring memory through inference-based event segmentation. Topics Cogn. Sci. 13, 106–127 (2021).

    Article  Google Scholar 

  62. Kumar, M. et al. Brainiak: The Brain Imaging Analysis Kit (BrainIAK, 2020).

  63. Vodrahalli, K. et al. Mapping between fMRI responses to movies and their natural language annotations. NeuroImage 180, 223–231 (2018).

    Article  Google Scholar 

  64. Hanke, M. et al. A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation. Sci. Data 3, 160092 (2016)..

  65. Sengupta, A. et al. A studyforrest extension, retinotopic mapping and localization of higher visual areas. Sci. Data 3, 160093 (2016).

    Article  Google Scholar 

  66. Haxby, J. V., Guntupalli, J. S., Nastase, S. A. & Feilong, M. Hyperalignment: modeling shared information encoded in idiosyncratic cortical topographies. eLife 9, e56601 (2021).

    Article  Google Scholar 

  67. Shirer, W. R., Ryali, S., Rykhlevskaia, E., Menon, V. & Greicius, M. D. Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cereb Cortex 22, 158–165 (2012).

    Article  Google Scholar 

  68. Halchenko, Y. O. et al. Datalad: distributed system for joint management of code, data, and their relationship. J. Open Source Softw. 6, 3262 (2021).

    Article  Google Scholar 

  69. Busch, E. ericabusch/tphate_analysis_capsule Version 2 release (Zenoodo, 2023); https://doi.org/10.5281/zenodo.7626543

  70. Busch, E. Krishnaswamylab/tphate Initial release (Zenodo, 2023); https://doi.org/10.5281/zenodo.7637523

Download references

Acknowledgements

We thank Ariadne Letrou for reviewing the code and the helpful discussions. E.L.B. was supported by a NSF Graduate Research Fellowship (award no. 2139841). G.L. was supported by Canada CIFAR AI Chair and Canada Research Chair in Neural Computations and Interfacing. G.W. was supported by Canada CIFAR AI Chair and IVADO Professor research funds. S.K. was supported by the NIH (grant nos. R01GM135929 and R01GM130847), an NSF Career Grant (grant no. 2047856) and a Sloan Fellowship (grant no. FG-2021-15883). N.B.T-B. was supported by an NSF Grant CCF (grant no. 1839308) and CIFAR.

Author information

Author notes

  1. These authors contributed equally: Smita Krishnaswamy and Nicholas B. Turk-Browne.

Authors and Affiliations

  1. Department of Psychology, Yale University, New Haven, CT, USA

    Erica L. Busch & Nicholas B. Turk-Browne

  2. Department of Computer Science, Yale University, New Haven, CT, USA

    Jessie Huang, Tom Wallenstein & Smita Krishnaswamy

  3. Department of Mathematics, Yale University, New Haven, CT, USA

    Andrew Benz

  4. Department of Mathematics and Statistics, Université de Montréal, Montreal, Canada

    Guillaume Lajoie & Guy Wolf

  5. Mila—Quebec Artificial Intelligence Institute, Montreal, Canada

    Guillaume Lajoie & Guy Wolf

  6. Department of Genetics, Yale University, New Haven, CT, USA

    Smita Krishnaswamy

  7. Program in Applied Mathematics, Yale University, New Haven, CT, USA

    Smita Krishnaswamy

  8. Wu Tsai Institute, Yale University, New Haven, CT, USA

    Smita Krishnaswamy & Nicholas B. Turk-Browne

Authors
  1. Erica L. Busch
    View author publications

    Search author on:PubMed Google Scholar

  2. Jessie Huang
    View author publications

    Search author on:PubMed Google Scholar

  3. Andrew Benz
    View author publications

    Search author on:PubMed Google Scholar

  4. Tom Wallenstein
    View author publications

    Search author on:PubMed Google Scholar

  5. Guillaume Lajoie
    View author publications

    Search author on:PubMed Google Scholar

  6. Guy Wolf
    View author publications

    Search author on:PubMed Google Scholar

  7. Smita Krishnaswamy
    View author publications

    Search author on:PubMed Google Scholar

  8. Nicholas B. Turk-Browne
    View author publications

    Search author on:PubMed Google Scholar

Contributions

E.L.B., J.H., G.W., G.L., S.K. and N.B.T-B. conceived the idea. E.L.B., J.H., A.B. and S.K. designed the T-PHATE algorithm. E.L.B., J.H. and T.W. developed the T-PHATE software. E.L.B. curated the data, and designed and performed analyses. E.L.B., S.K. and N.B.T-B. wrote the initial paper draft. All authors revised the paper. S.K. and N.B.T-B. jointly supervised the work.

Corresponding author

Correspondence to Smita Krishnaswamy.

Ethics declarations

Competing interests

S.K. is a visiting professor at META Fundamental AI Research (FAIR). The other authors declare no competing interests.

Peer review

Peer review information

Nature Computational Science thanks the anonymous reviewers for their contribution to the peer review of this work. Handling editor: Ananya Rastogi, in collaboration with the Nature Computational Science team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods, additional analysis and Figs. 1–9.

Reporting Summary

Supplementary Data 1

Source Data for supplementary figures.

Source data

Source Data Fig. 1

Data to generate Fig. 1c.

Source Data Fig. 2

Data to generate Fig. 2b.

Source Data Fig. 3

Data to generate Fig. 3c.

Source Data Fig. 5

Data to generate Fig. 5a,b.

Source Data Fig. 6

Data to generate Fig. 6a,c.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Busch, E.L., Huang, J., Benz, A. et al. Multi-view manifold learning of human brain-state trajectories. Nat Comput Sci 3, 240–253 (2023). https://doi.org/10.1038/s43588-023-00419-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43588-023-00419-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing