BMLL Technologies was founded in the machine learning laboratories of the University of Cambridge. We provide harmonised Level 3, 2, and 1 historical market data and analytics covering global equities, ETFs, futures, and US equity options, serving quantitative researchers, execution analytics teams, and trading desks at asset managers, hedge funds, and banks worldwide.
Our data is normalised to nanosecond precision across 100+ global venues and delivered through BMLL Data Lab, giving clients direct access to analytics-ready data without the overhead of sourcing, cleaning, or harmonising raw exchange data themselves.
The scale of this data is substantial. Analysing even a single day of whole-market tick data means working with tens to hundreds of gigabytes per session. That puts real pressure on the tools used to process it, which is why we actively look for high-performance data processing libraries for our clients to use, such as Polars.
The challenge: pandas stalls at tick-data scale #
Market data comes in three levels. Level 1 is top-of-book: best bid, best ask, last price. Level 2 adds the full stack of orders behind those prices. Level 3 captures every individual order message sent to an exchange, each submission, modification, and cancellation, at nanosecond granularity. It is the most complete picture of market intent available, and it is what most of our clients work with.
The challenge is tooling. Most quantitative teams use pandas on high-memory machines. Pandas is familiar, but it struggles at tick data scale. ** a single day of US equity trade data takes close to 3.5 minutes, before any joins or aggregations.** To keep runtimes manageable, teams end up filtering their analysis to a subset of ticks rather than working with the full dataset. That means incomplete coverage, missed signals, and analysis built on samples rather than reality.
Polars loads the same data 48x faster #
On the same US equity trade data (all trades for one day for all 11k+ REG NMS securities), the Polars load takes 4.3 seconds. That is a roughly 48x improvement over pandas, on the complete dataset, with no sampling required.
Polars handles the scale of market microstructure data well for a few reasons that matter in practice. Its lazy evaluation model lets analysts plan complex multi-step queries before any computation runs. Its as-of join, used to match each trade to the prevailing order book state at execution time, is natively supported and highly efficient. And it uses all available CPU cores without any configuration overhead.
What an as-of join does
An as-of join matches each row in one frame to the nearest row in another, keyed on a sorted column such as a timestamp, rather than requiring an exact match. For trade and quote data, that means matching each trade to the most recent quote at or before its timestamp, the prevailing order book state at the moment of execution, without a manual merge-and-filter step.
We now recommend Polars to clients working with our data programmatically, and we use it ourselves when benchmarking what is achievable on our data feeds.
The benchmarks #
How these benchmarks were run
All benchmarks below ran on BMLL Data Lab using a single 192-core, 1.5 TB RAM machine. They span two markets and three analyses, chosen to show what Polars makes possible at different scales and across different data types.
US equities: auction imbalance
A full auction imbalance analysis across US stock exchanges (XNYS, XNAS, BATS): all equities, nanosecond precision, across a 10-year period.
Single day: Shanghai Stock Exchange (2025-02-17)
To demonstrate performance beyond US markets, we ran the same trade and quote analysis on the Shanghai Stock Exchange (XSHG): all equities, nanosecond precision, for a single trading day.
16-day range: Shanghai Stock Exchange (March 1-16, 2026)
Scaling the same analysis across sixteen trading days: approximately 268 GB of trade data and 1.2 TB of L2 order book data.
What this means in practice #
Take a representative example: an asset manager with $100 billion in assets under management.
Most institutional teams we talk with running tick data analysis carry $300,000–$600,000 per year in dedicated data infrastructure, plus another $200,000–$400,000 in cloud compute for overnight batch processing. With Polars, the same workloads run in minutes, on demand, on a single machine. That entire infrastructure layer becomes optional.
The execution impact is larger. At 20% annual turnover, a $100B manager executes roughly $20 billion in trades per year. Every basis point of improvement in implementation shortfall returns $2 million to investors. Teams that iterate daily instead of weekly consistently capture more basis points, across more venues and instruments. That compounds fast.
The infrastructure savings largely pay for the data. The execution alpha is where the real return is.
BMLL does the heavy lifting of collection, cleaning, and normalisation, so users spend their time analysing rather than preparing data. With Polars, that analysis now runs on the full dataset in seconds instead of minutes, so users can ask more of the data without waiting on it.
Thomas Jardine, Head of Data Science - Americas @ BMLL
Conclusion #
Working with nanosecond-precision, petabyte-scale market data used to mean accepting slow feedback loops or investing heavily in infrastructure. Polars removes that trade-off.
Combined with BMLL’s harmonised market data, it makes daily iteration on whole-market microstructure workloads practical from a single notebook. Teams that previously worked around their tooling can now let their research questions set the pace.
If you are working with market data at scale, exploring what BMLL and Polars can do together is a natural starting point.