{"slug": "time-series-language-models-for-reasoning-over-multivariate-data-at-scale-icml", "title": "Time-Series Language Models for Reasoning over Multivariate Data at Scale (ICML)", "summary": "Researchers from Stanford University, ETH Zurich, and other institutions introduced OpenTSLM, a multimodal large language model that processes raw multivariate time series data alongside text for medical reasoning tasks. The model outperformed GPT-4o and other baselines in time series question answering, activity recognition, sleep staging, and ECG analysis, with clinician review finding its reasoning correct or partially correct 97% of the time.", "body_md": "OpenTSLM\n\nOpenTSLM: Time Series Language Models for Reasoning over Multivariate Medical Text- and Time Series Data\n\nPatrick Langer1, 2, 3, Thomas Kaar1, 3, Max Rosenblattl1, 3, Maxwell A. Xu4, 5, Winnie Chow6, Martin Maritsch7, Robert Jakob3, Ning Wang3, Juncheng Liu8, 9, Aradhana Verma10, Brian Han11, Daniel Seung Kim12, Henry Chubb11, Scott Ceresnak11, Aydin Zahedivash1, Alexander Tarlochan Singh Sandhu10, Fatima Rodriguez10, Daniel McDuff5, 13, Elgar Fleisch2, 3, 14, Oliver Aalami1, Filipe Barata2, *, Paul Schmiedmayer1, *\n\n1STANFORD MUSSALLEM CENTER FOR BIODESIGN, STANFORD UNIVERSITY, USA, 2CENTRE FOR DIGITAL HEALTH INTERVENTIONS, ETH ZURICH, SWITZERLAND, 3AGENTIC SYSTEMS LAB, ETH ZURICH, SWITZERLAND, 4UNIVERSITY OF ILLINOIS URBANA-CHAMPAIGN, USA, 5GOOGLE RESEARCH, USA, 6STANFORD UNIVERSITY, USA, 7AMAZON, USA, 8MICROSOFT, 9NATIONAL UNIVERSITY OF SINGAPORE, SINGAPORE, 10DIVISION OF CARDIOVASCULAR MEDICINE, STANFORD UNIVERSITY, USA, 11PEDIATRIC CARDIOLOGY, STANFORD UNIVERSITY, USA, 12DIVISION OF CARDIOLOGY, UNIVERSITY OF WASHINGTON, USA, 13UNIVERSITY OF WASHINGTON, USA, 14CENTRE FOR DIGITAL HEALTH INTERVENTIONS, UNIVERSITY OF ST. GALLEN, SWITZERLAND, *EQUAL CONTRIBUTION\n\nACCEPTED AT\n\nTime Series Language Models (TSLMs) are language models that can take raw time series as input alongside text. Instead of converting signals into plots, summaries, or long strings of numbers, a TSLM reads the signal directly and answers in natural language.\n\nThis makes them flexible in the same way multimodal chat models are flexible for images: you can attach one or many signals, combine signals of different lengths and sampling rates, ask open-ended questions, request explanations, compare windows, or ask for a forecast without training a separate model for each task.\n\nWith OpenTSLM, we present a multimodal LLM that treats time series as a native modality. It can reason over raw multivariate signals alongside text, and it is designed to scale to multiple very long time series without forcing the signals through the language-model token stream.\n\nThe paper compares two ways to fuse text and time series, **OpenTSLM-SoftPrompt** and **OpenTSLM-Flamingo**, on Gemma3 and Llama3.2 backbones at 270M, 1B and 3B scale. SoftPrompt places time series tokens in the language-model input. Flamingo uses cross-attention, allowing the language model to query compact time series representations instead of processing every signal point as a text token.\n\nOPENTSLM-FLAMINGO ARCHITECTURE\n\nWe compare OpenTSLM against the main ways people try to make language models handle time series today: serialize the values as text, fine tune on serialized values, show the model a plot, or use GPT-4o with image inputs.\n\nOpenTSLM outperforms every baseline across time series question answering, activity recognition, sleep staging, and ECG question answering. Even 1B-parameter OpenTSLM models surpass GPT-4o, showing that time series capability comes from the model interface, not just from making a general model larger.\n\nBEST MACRO-F1 PER METHOD FAMILY · HIGHER IS BETTER ↑\n\nOpenTSLM\n\nImage-based\n\nTokenized\n\nGPT-4o\n\nTSQATIME SERIES QUESTION ANSWERING\n\nOpenTSLM\n\n99.71\n\nImage-based\n\n94.94\n\nTokenized FT\n\n84.54\n\nGPT-4o\n\n59.24\n\nTokenized\n\n16.24\n\nHAR-CoTACTIVITY RECOGNITION\n\nOpenTSLM\n\n69.31\n\nTokenized FT\n\n60.44\n\nImage-based\n\n44.01\n\nGPT-4o\n\n10.83\n\nTokenized\n\n0.00\n\nSleep-CoTSLEEP STAGING\n\nOpenTSLM\n\n69.88\n\nImage-based\n\n18.56\n\nGPT-4o\n\n15.47\n\nTokenized FT\n\n9.05\n\nTokenized\n\n5.66\n\nECG-QA-CoTECG QUESTION ANSWERING\n\nOpenTSLM\n\n41.92\n\nImage-based\n\n26.17\n\nGPT-4o\n\n24.95\n\nTokenized FT\n\n—\n\nTokenized\n\n0.00\n\nBARS SHOW BEST MACRO-F1 WITHIN EACH METHOD FAMILY · OOM = OUT OF MEMORY | LANGER ET AL., 2025\n\nWe evaluated 84 rationales of our best model, OpenTSLM/Llama3.2-3B, through expert review by 7 Stanford Hospital clinicians, finding the reasoning correct or partially correct 97% of the time. They used a rubric based on ACC/AHA ECG competence guidelines and the RIME framework, focusing on ECG feature identification, reasoning quality, and use of patient context.\n\nECG REASONING VALIDATED BY SEVEN CARDIOLOGISTS | STANFORD HOSPITAL\n\nPOSITIVE\n\nPARTIAL\n\nNEGATIVE\n\n97%\n\nof 84 reviewed rationales from OpenTSLM/Llama3.2-3B rated correct or partially correct, using a rubric based on ACC/AHA ECG competence guidelines and the RIME framework.\n\nPATTERN RECOGNITION\n\nCLINICAL REASONING\n\nCONTEXT INTEGRATION\n\nWe, for the first time, present an architecture for TSLMs that enables scaling to multiple multivariate and long time series. While prior methods typically scale quadratically in memory with longer time series, our implementation achieves comparable performance with near constant memory.\n\nSIMULATION DATASETS | LANGER ET AL., 2025\n\nOPENTSLM-FLAMINGO\n\nPRIOR METHODS / SOFTPROMPT\n\nOpenTSLM-Flamingo keeps VRAM usage nearly constant when inputting more or longer time series.\n\nAll code, datasets, and models from the paper are released open-source.", "url": "https://wpnews.pro/news/time-series-language-models-for-reasoning-over-multivariate-data-at-scale-icml", "canonical_source": "https://www.aioniclabs.ai/research/opentslm", "published_at": "2026-07-17 10:15:03+00:00", "updated_at": "2026-07-17 10:20:58.895885+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research", "machine-learning"], "entities": ["Stanford University", "ETH Zurich", "Google Research", "Amazon", "Microsoft", "National University of Singapore", "University of Washington", "OpenTSLM"], "alternates": {"html": "https://wpnews.pro/news/time-series-language-models-for-reasoning-over-multivariate-data-at-scale-icml", "markdown": "https://wpnews.pro/news/time-series-language-models-for-reasoning-over-multivariate-data-at-scale-icml.md", "text": "https://wpnews.pro/news/time-series-language-models-for-reasoning-over-multivariate-data-at-scale-icml.txt", "jsonld": "https://wpnews.pro/news/time-series-language-models-for-reasoning-over-multivariate-data-at-scale-icml.jsonld"}}