Work

ML research on market data (ClickHouse)

A ClickHouse analytics store feeding ML research on large-scale market data.

Result

<1 squeries over billions of rows

Billions of rows of tick data, queried in under a second.

Capital-markets research work

ML research on years of multi-venue market data needs fast queries. Slow queries mean fewer experiments.

In production

clickhouse-client · research@ch-mdcluster md_cluster · 6 shards

SELECT venue, toStartOfMonth(ts) AS month, count() AS ticks,
       round(avg(spread_bps), 3) AS avg_spread,
       round(quantile(0.99)(spread_bps), 3) AS p99_spread
FROM md.quotes_dist
WHERE symbol = 'BTC-USD' AND ts BETWEEN '2017-01-01' AND '2026-08-31 23:59:59'
GROUP BY venue, month ORDER BY month DESC, venue

venue	month	ticks	avg_spread	p99_spread
V1	2026-08-01	41,208,114	0.184	1.412
V2	2026-08-01	58,960,327	0.121	0.968
V3	2026-08-01	17,442,905	0.337	2.705
V4	2026-08-01	29,113,480	0.226	1.873
V5	2026-08-01	12,875,661	0.402	3.118

585 rows in set (showing 5). Elapsed: 0.841 sec. Processed 6.84 billion rows, 95.78 GB (8.13 billion rows/s., 113.88 GB/s.) · peak memory 3.41 GiB

A research query over years of tick data.

research · feature runsmd.quotes_dist · 6.84 bn rows

run	features	rows scanned	query	train	val MAE (bps)
r-2291	spread, depth L1-L5	6.84 bn	0.84 s	11m 40s	0.041
r-2290	spread, depth L1-L5	6.84 bn	0.86 s	11m 52s	0.043
r-2288	spread, trade imbalance	4.12 bn	0.57 s	8m 05s	0.047
r-2285	spread, volatility 1m	6.84 bn	0.79 s	10m 31s	0.052
r-2281	spread only (baseline)	6.84 bn	0.61 s	4m 12s	0.068

Tick data6.84 bnrows, 2017-01 to 2026-08

Feature query0.84 smedian over last 50 runs

Best runr-2291val MAE 0.041 bps

Research runs. Every feature query returns in under a second.

Representative views of production tooling. Data anonymized.

What we built

Market data into ClickHouse
Analytics store for tick data
Feature queries called from research code

ClickHousePythonKafkaKubernetes

Next case

Multi-venue execution

Next step

Have a hard system to build?

Tell us what it has to do. We will tell you how we would build it.

Start a project See what we build