Time-Series Language Models for Reasoning over Multivariate Data at Scale (ICML)

Researchers from Stanford University, ETH Zurich, and other institutions introduced OpenTSLM, a multimodal large language model that processes raw multivariate time series data alongside text for medical reasoning tasks. The model outperformed GPT-4o and other baselines in time series question answering, activity recognition, sleep staging, and ECG analysis, with clinician review finding its reasoning correct or partially correct 97% of the time.

OpenTSLM OpenTSLM: Time Series Language Models for Reasoning over Multivariate Medical Text- and Time Series Data Patrick Langer1, 2, 3, Thomas Kaar1, 3, Max Rosenblattl1, 3, Maxwell A. Xu4, 5, Winnie Chow6, Martin Maritsch7, Robert Jakob3, Ning Wang3, Juncheng Liu8, 9, Aradhana Verma10, Brian Han11, Daniel Seung Kim12, Henry Chubb11, Scott Ceresnak11, Aydin Zahedivash1, Alexander Tarlochan Singh Sandhu10, Fatima Rodriguez10, Daniel McDuff5, 13, Elgar Fleisch2, 3, 14, Oliver Aalami1, Filipe Barata2, , Paul Schmiedmayer1, 1STANFORD MUSSALLEM CENTER FOR BIODESIGN, STANFORD UNIVERSITY, USA, 2CENTRE FOR DIGITAL HEALTH INTERVENTIONS, ETH ZURICH, SWITZERLAND, 3AGENTIC SYSTEMS LAB, ETH ZURICH, SWITZERLAND, 4UNIVERSITY OF ILLINOIS URBANA-CHAMPAIGN, USA, 5GOOGLE RESEARCH, USA, 6STANFORD UNIVERSITY, USA, 7AMAZON, USA, 8MICROSOFT, 9NATIONAL UNIVERSITY OF SINGAPORE, SINGAPORE, 10DIVISION OF CARDIOVASCULAR MEDICINE, STANFORD UNIVERSITY, USA, 11PEDIATRIC CARDIOLOGY, STANFORD UNIVERSITY, USA, 12DIVISION OF CARDIOLOGY, UNIVERSITY OF WASHINGTON, USA, 13UNIVERSITY OF WASHINGTON, USA, 14CENTRE FOR DIGITAL HEALTH INTERVENTIONS, UNIVERSITY OF ST. GALLEN, SWITZERLAND, EQUAL CONTRIBUTION ACCEPTED AT Time Series Language Models TSLMs are language models that can take raw time series as input alongside text. Instead of converting signals into plots, summaries, or long strings of numbers, a TSLM reads the signal directly and answers in natural language. This makes them flexible in the same way multimodal chat models are flexible for images: you can attach one or many signals, combine signals of different lengths and sampling rates, ask open-ended questions, request explanations, compare windows, or ask for a forecast without training a separate model for each task. With OpenTSLM, we present a multimodal LLM that treats time series as a native modality. It can reason over raw multivariate signals alongside text, and it is designed to scale to multiple very long time series without forcing the signals through the language-model token stream. The paper compares two ways to fuse text and time series, OpenTSLM-SoftPrompt and OpenTSLM-Flamingo , on Gemma3 and Llama3.2 backbones at 270M, 1B and 3B scale. SoftPrompt places time series tokens in the language-model input. Flamingo uses cross-attention, allowing the language model to query compact time series representations instead of processing every signal point as a text token. OPENTSLM-FLAMINGO ARCHITECTURE We compare OpenTSLM against the main ways people try to make language models handle time series today: serialize the values as text, fine tune on serialized values, show the model a plot, or use GPT-4o with image inputs. OpenTSLM outperforms every baseline across time series question answering, activity recognition, sleep staging, and ECG question answering. Even 1B-parameter OpenTSLM models surpass GPT-4o, showing that time series capability comes from the model interface, not just from making a general model larger. BEST MACRO-F1 PER METHOD FAMILY · HIGHER IS BETTER ↑ OpenTSLM Image-based Tokenized GPT-4o TSQATIME SERIES QUESTION ANSWERING OpenTSLM 99.71 Image-based 94.94 Tokenized FT 84.54 GPT-4o 59.24 Tokenized 16.24 HAR-CoTACTIVITY RECOGNITION OpenTSLM 69.31 Tokenized FT 60.44 Image-based 44.01 GPT-4o 10.83 Tokenized 0.00 Sleep-CoTSLEEP STAGING OpenTSLM 69.88 Image-based 18.56 GPT-4o 15.47 Tokenized FT 9.05 Tokenized 5.66 ECG-QA-CoTECG QUESTION ANSWERING OpenTSLM 41.92 Image-based 26.17 GPT-4o 24.95 Tokenized FT — Tokenized 0.00 BARS SHOW BEST MACRO-F1 WITHIN EACH METHOD FAMILY · OOM = OUT OF MEMORY | LANGER ET AL., 2025 We evaluated 84 rationales of our best model, OpenTSLM/Llama3.2-3B, through expert review by 7 Stanford Hospital clinicians, finding the reasoning correct or partially correct 97% of the time. They used a rubric based on ACC/AHA ECG competence guidelines and the RIME framework, focusing on ECG feature identification, reasoning quality, and use of patient context. ECG REASONING VALIDATED BY SEVEN CARDIOLOGISTS | STANFORD HOSPITAL POSITIVE PARTIAL NEGATIVE 97% of 84 reviewed rationales from OpenTSLM/Llama3.2-3B rated correct or partially correct, using a rubric based on ACC/AHA ECG competence guidelines and the RIME framework. PATTERN RECOGNITION CLINICAL REASONING CONTEXT INTEGRATION We, for the first time, present an architecture for TSLMs that enables scaling to multiple multivariate and long time series. While prior methods typically scale quadratically in memory with longer time series, our implementation achieves comparable performance with near constant memory. SIMULATION DATASETS | LANGER ET AL., 2025 OPENTSLM-FLAMINGO PRIOR METHODS / SOFTPROMPT OpenTSLM-Flamingo keeps VRAM usage nearly constant when inputting more or longer time series. All code, datasets, and models from the paper are released open-source.