Abstract
In-sensor computing, which integrates sensing, memory and processing functions, has shown substantial potential in artificial vision systems. However, large-scale monolithic integration of in-sensor computing based on emerging devices with complementary metal–oxide–semiconductor (CMOS) circuits remains challenging, lacking functional demonstrations at the hardware level. Here we report a fully integrated 1-kb array with 128 × 8 one-transistor one-optoelectronic memristor (OEM) cells and silicon CMOS circuits, which features configurable multi-mode functionality encompassing three different modes of electronic memristor, dynamic OEM and non-volatile OEM (NV-OEM). These modes are configured by modulating the charge density within the oxygen vacancies via synergistic optical and electrical operations, as confirmed by differential phase-contrast scanning transmission electron microscopy. Using this OEM system, three visual processing tasks are demonstrated: image sensory pre-processing with a recognition accuracy enhanced from 85.7% to 96.1% by the NV-OEM mode, more advanced object tracking with 96.1% accuracy using both dynamic OEM and NV-OEM modes and human motion recognition with a fully OEM-based in-sensor reservoir computing system achieving 91.2% accuracy. A system-level benchmark further shows that it consumes over 20 times less energy than graphics processing units. By monolithically integrating the multi-functional OEMs with Si CMOS, this work provides a cost-effective platform for diverse in-sensor computing applications.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
27,99 € / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
269,00 € per year
only 22,42 € per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding authors on request. More data are also presented in the Supplementary Information. Source data are provided with this paper.
Code availability
All the codes that support the findings of this study are available from the corresponding authors upon reasonable request.
References
Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
Wu, N. Neuromorphic vision chips. Sci. China Inf. Sci. 61, 060421 (2018).
Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).
Dai, S. et al. Emerging iontronic neural devices for neuromorphic sensory computing. Adv. Mater. 35, 2300329 (2023).
Du, J. et al. A robust neuromorphic vision sensor with optical control of ferroelectric switching. Nano Energy 89, 106439 (2021).
Li, G. et al. Photo-induced non-volatile VO2 phase transition for neuromorphic ultraviolet sensors. Nat. Commun. 13, 1729 (2022).
Jang, H. et al. An atomically thin optoelectronic machine vision processor. Adv. Mater. 32, 2002431 (2020).
Zhang, Z. et al. In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat. Commun. 13, 6590 (2022).
Zhao, R. et al. A framework for the general design and computation of hybrid neural networks. Nat. Commun. 13, 3427 (2022).
Wang, Y. et al. Optoelectronic synaptic devices for neuromorphic computing. Adv. Intell. Syst 3, 2000099 (2021).
Zhang, J., Dai, S., Zhao, Y., Zhang, J. & Huang, J. Recent progress in photonic synapses for neuromorphic systems. Adv. Intell. Syst. 2, 1900136 (2020).
Song, S. et al. Recent progress of optoelectronic and all-optical neuromorphic devices: a comprehensive review of device structures, materials, and applications. Adv. Intell. Syst. 3, 2000119 (2021).
John, R. A. et al. Reconfigurable halide perovskite nanocrystal memristors for neuromorphic computing. Nat. Commun. 13, 2074 (2022).
Wang, T. et al. Reconfigurable neuromorphic memristor network for ultralow-power smart textile electronics. Nat. Commun. 13, 7432 (2022).
Pi, L. et al. Broadband convolutional processing using band-alignment-tunable heterostructures. Nat. Electron. 5, 248–254 (2022).
Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).
Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022).
Sun, L. et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv. 7, eabg1455 (2021).
Wu, X. et al. Wearable in-sensor reservoir computing using optoelectronic polymers with through-space charge-transport characteristics for multi-task learning. Nat. Commun. 14, 468 (2023).
Lao, J. et al. Ultralow-power machine vision with self-powered sensor reservoir. Adv. Sci. 9, 2106092 (2022).
Zhong, Y. et al. A memristor-based analogue reservoir computing system for real-time and power-efficient signal processing. Nat. Electron. 5, 672–681 (2022).
Liang, X. et al. Physical reservoir computing with emerging electronics. Nat. Electron. 7, 193–206 (2024).
Moon, J. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat. Electron. 2, 480–487 (2019).
Portner, K. et al. Analog nanoscale electro-optical synapses for neuromorphic computing applications. ACS. Nano. 15, 14776–14785 (2021).
Hu, L. et al. All-optically controlled memristor for optoelectronic neuromorphic computing. Adv. Funct. Mater. 31, 2005582 (2021).
Tan, H. et al. An optoelectronic resistive switching memory with integrated demodulating and arithmetic functions. Adv. Mater. 27, 2797–2803 (2015).
Chen, J. Y. et al. Dynamic evolution of conducting nanofilament in resistive switching memories. Nano Lett. 13, 3671–3677 (2013).
Simanjuntak, F. M., Panda, D., Wei, K. H. & Tseng, T. Y. Status and prospects of ZnO-based resistive switching memory devices. Nanoscale Res. Lett. 11, 368 (2016).
Xu, N. et al. Characteristics and mechanism of conduction/set process in TiN/ZnO/Pt resistance switching random-access memories. Appl. Phys. Lett. 92, 232112 (2008).
Zhou, Z., Pei, Y., Zhao, J., Fu, G. & Yan, X. Visible light responsive optoelectronic memristor device based on CeOx/ZnO structure for artificial vision system. Appl. Phys. Lett. 118, 191103 (2021).
Wang, T.-Y. et al. Reconfigurable optoelectronic memristor for in-sensor computing applications. Nano Energy 89, 106291 (2021).
Wang, W. et al. CMOS backend-of-line compatible memory array and logic circuitries enabled by high performance atomic layer deposited ZnO thin-film transistor. Nat. Commun. 14, 6079 (2023).
Wang, Z. et al. Vacancy driven surface disorder catalyzes anisotropic evaporation of ZnO (0001) polar surface. Nat. Commun. 13, 5616 (2022).
Lanza, M. et al. Recommended methods to study resistive switching devices. Adv. Electron. Mater. 5, 1800143 (2019).
Kuzum, D., Yu, S. & Wong, H. S. Synaptic electronics: materials, devices and applications. Nanotechnology 24, 382001 (2013).
Russo, P., Xiao, M., Liang, R. & Zhou, N. Y. UV-induced multilevel current amplification memory effect in zinc oxide rods resistive switching devices. Adv. Funct. Mater. 28, 1706230 (2018).
Oh, I., Pyo, J. & Kim, S. Resistive switching and synaptic characteristics in ZnO/TaON-based RRAM for neuromorphic system. Nanomaterial 12, 2185 (2022).
Seo, S. et al. Artificial optic-neural synapse for colored and color-mixed pattern recognition. Nat. Commun. 9, 5106 (2018).
Yilmaz, A., Javed, O. & Shah, M. Object tracking. ACM Comput. Surv. 38, 1–45 (2006).
Wang, S. et al. Networking retinomorphic sensor with memristive crossbar for brain-inspired visual perception. Natl Sci. Rev. 8, nwaa172 (2020).
Acknowledgements
This work was in part supported by National Natural Science Foundation of China 92264201 (J.T.), 62025111 (H.W.) and 62174095 (Y.W.), National Key R&D Programme of China 2021ZD0109901 (L.F.), China Postdoctoral Science Foundation 2021M701845 (H.H.), the XPLORER Prize (H.W.), Tsinghua University Initiative Scientific Research Programme and the Centre of Nanofabrication, Tsinghua University. We are also grateful to J. Chen from the University of Zurich and J. Tang from Boston College for their valuable suggestions on the manuscript.
Author information
Authors and Affiliations
Contributions
H.H., Y.W. and J.T. conceived and designed the experiments. H.H. contributed to the OEM system fabrication. W.S., H.H., J.W. and X.B. contributed to the TEM analysis. H.H., X.M., Z.J. and R.H. participated in the measurements. Y. Li performed a simulation of image noise reduction under the supervision of Q.Z.; Jianing Zhang and Jinzhi Zhang performed a simulation of object tracking under the supervision of L.F. and Q.D.; and X.L., Y. Li and H.H. performed experiments on RC. F.X. and Y. Lu provided theoretical support. Y.D., P.Y., Z.L., Z.W., B.G., H.Y. and H.Q. analysed the data and discussed the results. H.H., X.L., Y.W. and J.T. wrote the manuscript. All authors discussed the results and commented on the manuscript. Y.W., J.T. and H.W. supervised the project.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Nanotechnology thanks Jang-Sik Lee and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Information
Supplementary Figs. 1–30, Notes 1–7 and Tables 1–3.
Supplementary Video 1
Real-time object tracking for the moving Tsinghua University school bus, based on the OEM system, achieving an accuracy of 96.1%.
Supplementary Video 2
The targets are the school buses (B1, B2) and motorcycle (M). Both buses and motorcycle can be well tracked during the driving process, and the motorcycle can also be well re-identified after being obscured by the school bus.
Source data
Source Data Fig. 2
Raw datasets for the multi-functional OEM mode. Figure 2b–d shows the D-OEM mode dataset. Figure 2f–h shows the EM mode dataset. Figure 2j–l shows the NV-OEM mode dataset.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Huang, H., Liang, X., Wang, Y. et al. Fully integrated multi-mode optoelectronic memristor array for diversified in-sensor computing. Nat. Nanotechnol. 20, 93–103 (2025). https://doi.org/10.1038/s41565-024-01794-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41565-024-01794-z
This article is cited by
-
Capacitive in-sensor tactile computing
Nature Communications (2025)
-
Chelated tin halide perovskite for near-infrared neuromorphic imaging array enabling object recognition and motion perception
Nature Communications (2025)
