QuantLake
GitHub

A self-hosted data lake for market data

QuantLake downloads historical market data from Tardis.dev and Databento to your Linux server. It saves one Parquet file per symbol, data type and UTC day, and adds each new day after the provider publishes it. You read the files directly with DuckDB, Polars or pandas.

Each square is one day of one data type for one symbol, saved as one Parquet file.

  • Saved
  • Downloading
  • Waiting to download
  • Skipped: over the Databento budget
  • Not published yet

How it works

A background server does the downloading. You tell it what to collect with the quantlake command.

  1. 1

    Install the server

    Copies the two binaries, registers the quantlake-server systemd service and creates the lake in ~/.local/share/quantlake. Then set your API keys with quantlake config set.

    sudo ./artifacts/linux-x64/quantlake-server install
  2. 2

    Register what to collect

    The server starts downloading within a few seconds, oldest day first. Without --to, it keeps adding each new day.

    quantlake targets add --exchange binance-futures \
      --data-type trades --symbol BTCUSDT --from 2026-10-01
  3. 3

    Get the file paths

    Prints the path of each saved file and a wildcard pattern that you can pass to DuckDB or Polars.

    quantlake files --exchange binance-futures \
      --data-type trades --symbol BTCUSDT

One Parquet file per symbol per day

The folder names are Hive partitions, so your query tool reads them as columns, including date. Data from both providers uses the same data type names and columns, based on the Databento schemas.

  1. source=tardis/Data provider
  2. exchange=binance-futures/Exchange or dataset
  3. data_type=trades/Data type
  4. symbol=BTCUSDT/Symbol
  5. date=2026-10-01/UTC day
  6. data.parquetOne file
SELECT date, count(*) AS trades, avg(price) AS avg_price
FROM read_parquet(
  '~/.local/share/quantlake/data/source=*/exchange=binance-futures/data_type=trades/symbol=BTCUSDT/date=*/data.parquet',
  hive_partitioning = true)
WHERE date BETWEEN '2026-10-01' AND '2026-10-07'
GROUP BY date ORDER BY date;

QuantLake converts the columns to a common format but keeps the rows as the provider sends them. Timestamps are UTC with nanosecond precision; prices and sizes are 64-bit floating point numbers. See the columns of each data type.

Data from Tardis.dev and Databento

Your plan with each provider decides which exchanges, symbols and dates you can collect. QuantLake reads what your API key allows and collects only that.

Tardis.dev

Crypto exchanges, such as binance-futures and deribit: trades, order book updates and snapshots, best bid and ask, funding rates and liquidations. Without an API key, only the first day of each month.

Databento

Futures, equities and options, such as CME Globex (glbx.mdp3) and Nasdaq (xnas.itch): the full order book (L3), top of book, trades, OHLCV bars, instrument definitions and statistics. Continuous contracts such as ES.c.0 are supported.

Data typeTardis.devDatabentoWhat it holds
tradestradesYesEvery trade
mbp-1quotesYesBest bid and ask after each change
mbp-10YesTop 10 price levels after each change
mbp-5, mbp-25book_snapshot_5, _25Top 5 or 25 price levels
book_tickerbook_tickerThe exchange's own best bid and ask feed
incremental_book_L2incremental_book_L2Updates to every price level
derivative_tickerderivative_tickerFunding rate, open interest, index and mark price
liquidationsliquidationsLiquidations
mboYesEvery order book event, order by order (L3)
tbboYesEach trade with the best bid and ask before it
bbo-1s, bbo-1mYesBest bid and ask every second or minute
ohlcv-1s … ohlcv-1dYesOHLCV bars
definitionYesInstrument definitions: tick size, expiry, legs
statisticsYesSettlement price, open interest and other official statistics

The Tardis.dev column shows the name that Tardis.dev uses. Databento also offers status, imbalance and consolidated best bid and ask. All data types

What the server takes care of

quantlake-server runs as a systemd service under your user account. It is the only process that writes to the lake.

Rate limits and retries
Stays within each provider's rate limits. Network and server errors are retried with backoff, waiting at most 6 hours.
Disk space
Stops starting new downloads when free disk space falls below a threshold, 20 GB by default.
Re-published days
Checks every month for days that the provider has published again, and downloads them again.
File integrity
Locks saved files with the Linux immutable attribute and checks every file against the catalog each week. Changed files are downloaded again.
Databento prices
Asks Databento for the price before each download. By default it downloads only data that costs nothing; you can set a monthly budget for paid data.
API keys
Stores your API keys encrypted with a public key. Only the server can decrypt them, and the CLI never prints them.

Usable by coding agents

QuantLake includes an Agent Skill that teaches Claude Code, Codex and other agents to find saved data, get file paths and register new collection targets through the CLI.

Every command prints JSON with --json. The output of quantlake files on the right is what an agent uses to write a DuckDB query.

Agents cannot edit the data files. Writes to the lake fail with Operation not permitted, and a note in the lake folder tells the agent which command to use instead.

npx skills@latest add supermomonga/quantlake --global
quantlake files … --json
{
  "root": "/home/you/.local/share/quantlake/data",
  "glob": "/home/you/.local/share/quantlake/data/source=*/exchange=binance-futures/data_type=trades/symbol=BTCUSDT/date=*/data.parquet",
  "files": [
    {
      "source": "tardis",
      "exchange": "binance-futures",
      "data_type": "trades",
      "symbol": "BTCUSDT",
      "date": "2026-10-01",
      "path": "/home/you/.local/share/quantlake/data/source=tardis/exchange=binance-futures/data_type=trades/symbol=BTCUSDT/date=2026-10-01/data.parquet",
      "rows": 3246752,
      "size_bytes": 16865029
    }
  ]
}

Requirements

  • Linux x64 with systemd and glibc 2.34 or later
  • A filesystem with the immutable attribute: ext4, xfs or btrfs
  • The .NET 10 SDK to build the two native binaries
  • An API key for Tardis.dev, Databento or both