Direct light-to-token conversion with integrated 2D photosensitive memory

Researchers introduced a physical tokenizer that performs analogue light-to-token conversion directly at the sensor level for vision transformers, integrating a 32 × 32 photosensitive memory array of monolayer MoS2 floating-gate phototransistors with peripheral circuitry. In a standard ViT pipeline, the prototype achieved software-comparable accuracy on CIFAR-10 while reducing energy consumption by 14.3-fold relative to a digital tokenizer, establishing physical tokenization as an energy-efficient hardware foundation for edge-intelligent vision systems.

Abstract Vision transformers ViTs process images as sequences of embedded tokens, yet in existing architectures, tokenization is performed entirely in the digital domain, downstream of image capture. This separation increases energy cost and prevents token formation from occurring where visual information is physically generated. Here we introduce a physical tokenizer that performs analogue light-to-token conversion directly at the sensor level for ViTs. The prototype integrates a 32 × 32 photosensitive memory array of monolayer MoS2 floating-gate phototransistors with peripheral addressing circuitry, enabling optical images to be stored as non-volatile states and selectively combined into patch embeddings in situ, thereby eliminating separate sensing, patch division and patch embedding stages. When deployed in a standard ViT pipeline, the physical tokenizer achieves software-comparable accuracy on CIFAR-10 while reducing energy consumption by 14.3-fold relative to a digital tokenizer. These results establish physical tokenization as an energy-efficient and scalable hardware foundation for edge-intelligent, data-intensive vision systems. This is a preview of subscription content, access via your institution https://wayf.springernature.com?redirect uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs44460-026-00122-3 Access options Subscribe to this journal Receive 12 digital issues and online access to articles 118,99 € per year only 9,92 € per issue Buy this article - Purchase on SpringerLink - Instant access to the full article PDF. 39,95 € Prices may be subject to local taxes which are calculated during checkout Similar content being viewed by others Subjects Data availability The data supporting the findings of this study are available from the corresponding authors upon reasonable request. Source data /articles/s44460-026-00122-3 MOESM1 are provided with this paper. Code availability The code supporting the findings of this study is available from the corresponding authors upon reasonable request. References Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. In Proc. International Conference on Learning Representations ICLR, 2021 .Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE/CVF International Conference on Computer Vision 10012–10022 IEEE, 2021 .Han, K. et al. A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. 45 , 87–110 2023 .Wu, B. et al. Visual transformers: token-based image representation and processing for computer vision. Preprint at https://arxiv.org/abs/2006.03677 https://arxiv.org/abs/2006.03677 2020 .Khan, S. et al. Transformers in vision: a survey. ACM Comput. Surv. 54 , 200 2022 .Sun, M. et al. VAQF: fully automatic software–hardware co-design framework for low-bit vision transformer. Preprint at https://arxiv.org/abs/2201.06618 https://arxiv.org/abs/2201.06618 2022 .Wang, T. et al. ViA: a novel vision-transformer accelerator based on FPGA. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 41 , 4088–4099 2022 .Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat. Electron. 1 , 22–29 2018 .Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3 , 664–671 2020 .Shi, W., Cao, J., Zhang, Q., Li, Y. & Xu, L. Edge computing: vision and challenges. IEEE Internet Things J. 3 , 637–646 2016 .Ren, Q. et al. Optoelectronic devices for in-sensor computing. Adv. Mater. 37 , 2407476 2025 .Wang, S. et al. Nonvolatile van der Waals heterostructure phototransistor for encrypted optoelectronic logic circuit. ACS Nano 16 , 4528–4535 2022 .Wang, J. et al. Technology and integration roadmap for optoelectronic memristor. Adv. Mater. 36 , e2307393 2024 .Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14 , 776–782 2019 .Zhou, G. et al. Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing. Nat. Commun. 14 , 8489 2023 .Jang, H. et al. An atomically thin optoelectronic machine vision processor. Adv. Mater. 32 , e2002431 2020 .Huang, H. et al. Fully integrated multi-mode optoelectronic memristor array for diversified in-sensor computing. Nat. Nanotechnol. 20 , 93–103 2025 .Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17 , 27–32 2022 .Chen, J. et al. Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nat. Nanotechnol. 18 , 882–888 2023 .Tan, H. & van Dijken, S. Dynamic machine vision with retinomorphic photomemristor-reservoir computing. Nat. Commun. 14 , 2169 2023 .Pan, X. et al. Parallel perception of visual motion using light-tunable memory matrix. Sci. Adv. 9 , eadi4083 2023 .Pang, X. et al. Non-volatile rippled-assisted optoelectronic array for all-day motion detection and recognition. Nat. Commun. 15 , 1613 2024 .Xu, J. et al. High-order dynamics in an ultra-adaptive neuromorphic vision device. Nat. Nanotechnol. 20 , 1419–1430 2025 .Wang, S. J. et al. An organic electrochemical transistor for multi-modal sensing, memory and processing. Nat. Electron. 6 , 281–291 2023 .Liao, F. et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron. 5 , 84–91 2022 .Lee, D. et al. In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing. Nat. Commun. 13 , 5223 2022 .Zhu, Q.-B. et al. A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems. Nat. Commun. 12 , 1798 2021 .Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577 , 641–646 2020 .Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nat. Electron. 1 , 52–59 2017 .Cai, F. et al. A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations. Nat. Electron. 2 , 290–299 2019 .Wang, S. et al. Networking retinomorphic sensor with memristive crossbar for brain-inspired visual perception. Nat. Sci. Rev. 8 , nwaa172 2020 .Li, C. et al. Revisiting the epitaxial growth mechanism of 2D TMDC single crystals. Adv. Mater. 36 , 2404923 2024 .Ma, S. et al. A 619-pixel machine vision enhancement chip based on two-dimensional semiconductors. Sci. Adv. 8 , eabn9328 2022 .Dodda, A. et al. Active pixel sensor matrix based on monolayer MoS 2phototransistor array. Nat. Mater. 21 , 1379–1387 2022 .Migliato Marega, G. et al. A large-scale integrated vector-matrix multiplication processor based on monolayer molybdenum disulfide memories. Nat. Electron. 6 , 991–998 2023 .Wang, H. et al. Recent advances in two-dimensional photovoltaic devices. J. Semicond. 45 , 051701 2024 .Yang, X., Yan, B., Li, H. & Chen, Y. ReTransformer: ReRAM-based processing-in-memory architecture for transformer acceleration. In Proc. 39th International Conference on Computer-Aided Design 1–9 ACM, 2020 .Kachris, C. A survey on hardware accelerators for large language models. Appl. Sci. 15 , 586 2025 .Nag, S., Datta, G., Kundu, S., Chandrachoodan, N. & Beerel, P. A. ViTA: a vision transformer inference accelerator for edge applications. In 2023 IEEE International Symposium on Circuits and Systems ISCAS 1–5 IEEE, 2023 .Leroux, N. et al. Analog in-memory computing attention mechanism for fast and energy-efficient large language models. Nat. Comput. Sci. 5 , 813–824 2025 .Fu, J.-H. et al. Oriented lateral growth of two-dimensional materials on c-plane sapphire. Nat. Nanotechnol. 18 , 1289–1294 2023 .Zou, X. et al. Robust epitaxy of single-crystal transition-metal dichalcogenides on lanthanum-passivated sapphire. Science 390 , eaea0849 2025 .Hinton, H. et al. A 200 x 256 image sensor heterogeneously integrating a 2D nanomaterial-based photo-FET array and CMOS time-to-digital converters. In 2022 IEEE International Solid- State Circuits Conference ISSCC 1–3 IEEE, 2022 .Zhu, K. et al. Hybrid 2D-CMOS microchips for memristive applications. Nature 618 , 57–62 2023 .Liu, C. et al. A full-featured 2D flash chip enabled by system integration. Nature 646 , 1081–1088 2025 .Liu, Y., Huang, Y. & Duan, X. Van der Waals integration before and beyond two-dimensional materials. Nature 567 , 323–333 2019 .Jayachandran, D. et al. Three-dimensional integration of two-dimensional field-effect transistors. Nature 625 , 276–281 2024 . Acknowledgements We thank Nexstrom Pte. Ltd., Singapore, for providing single-crystal monolayer MoS2 films. Funding This work was supported in part by the National Key R&D Program of China under grant 2023YFF1203600 S.-J.L. , the National Natural Science Foundation of China 62034004 F.M. , the Leading-edge Technology Program of Jiangsu Natural Science Foundation BK20232004 F.M. , the Natural Science Foundation of Jiangsu Province BK20233001 F.M. , the AI & AI for Science Project of Nanjing University 14380240 S.-J.L. and 14380242 F.M. . F.M. and S.-J.L. would like to acknowledge support from AIQ Foundation and the e-Science Center of Collaborative Innovation Center of Advanced Microstructures. L.-J.L. acknowledges support from the National University of Singapore and NRF Professorship NRF-P2025-002 . Author information Authors and Affiliations Contributions S.W., S.-J.L. and F.M. conceived the idea and designed the experiments. F.M. and S.-J.L. supervised the whole project. S.W., N.Y., X.S. and X.-J.Y. contributed to the device and array fabrication. S.W., X.-J.Y., J.S. and D.W. performed all the measurements and analysed the experimental data. X.-J.Y. contributed to the FPGA-controlled peripheral circuitry. J.S. carried out the simulation of the ViT model. L.-J.L., C.L. and H.L. contributed to the material growth. X.X. provided help in the array fabrication. W.Y., Y.Z., L.Y., Y.Y. and P.W. assisted in the design of the optoelectronic measurement set-up. C.P., Y.C. and Y.W. provided help in the data analysis. S.W., S.-J.L. and F.M. co-wrote the paper with input from all other authors. Corresponding authors Ethics declarations Competing interests The authors declare no competing interests. Peer review Peer review information Nature Sensors thanks Junseok Heo, Ming Liang Jin and Dmitry Polyushkin for their contribution to the peer review of this work. Additional information Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary information Supplementary Information download PDF https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs44460-026-00122-3/MediaObjects/44460 2026 122 MOESM1 ESM.pdf Supplementary Figs. 1–25 and Notes 1–6. Source data Source Data Fig. 2 download ZIP https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs44460-026-00122-3/MediaObjects/44460 2026 122 MOESM3 ESM.zip Source data for Fig. 2e–g. Source Data Fig. 3 download ZIP https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs44460-026-00122-3/MediaObjects/44460 2026 122 MOESM4 ESM.zip Source data for Fig. 3g,j. Source Data Fig. 4 download ZIP https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs44460-026-00122-3/MediaObjects/44460 2026 122 MOESM5 ESM.zip Source data for Fig. 4b–d,f. Rights and permissions Springer Nature or its licensor e.g. a society or other partner holds exclusive rights to this article under a publishing agreement with the author s or other rightsholder s ; author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. About this article Cite this article Wang, S., Yang, N., Yangdong, XJ. et al. Direct light-to-token conversion with integrated 2D photosensitive memory. Nat. Sens. 2026 . https://doi.org/10.1038/s44460-026-00122-3 Received: Accepted: Published: Version of record: DOI: https://doi.org/10.1038/s44460-026-00122-3