QuantLake
GitHub
Documentation

Introduction

QuantLake is a self-hosted data lake that downloads market data from Tardis.dev and Databento and stores it as daily Parquet files on your Linux server.

You tell QuantLake which market data you need: the exchange, the data type, the symbol and the start date. A background server downloads the data, converts it to Parquet and adds each new day after the provider publishes it. You then read the files directly from DuckDB, Polars, pandas or your own backtests.

Data providers

QuantLake downloads historical market data from two providers:

  • Tardis.dev for crypto: trades, order book updates and snapshots, best bid and ask, funding rates and liquidations.
  • Databento for futures, equities and options, such as CME Globex and Nasdaq: the full order book (L3), top of book, trades, OHLCV bars, instrument definitions, statistics and more.

Data from both providers uses the same data type names and columns, based on the Databento schemas. You can read Tardis.dev and Databento trades with the same query.

How the parts fit together

QuantLake has two programs:

ProgramWhat it does
quantlake-serverA systemd service that downloads the data, writes the Parquet files and keeps the catalog of saved data. It is the only process that writes to the lake.
quantlakeThe CLI. It shows what is saved and where, and registers what to collect, delete or download again. The server carries out the changes.

Your analysis code reads the Parquet files directly, without going through QuantLake.

Tardis.dev ──┐                         ┌── quantlake CLI (what to collect, what is saved)
             ├── quantlake-server ──> data lake (Parquet + catalog)
Databento ───┘                         └── DuckDB, Polars, pandas (read-only)

Features

  • Collects what you ask for, and keeps collecting. Register a symbol once, and QuantLake downloads the history you asked for. It then adds each new day.
  • Parquet that you can query directly. Timestamps are UTC with nanosecond precision, prices and sizes are numbers, and the folders use Hive-style partitions.
  • No unexpected Databento charges. Before each Databento download, QuantLake asks Databento for the price. By default, it downloads only data that costs nothing. You can set a monthly budget to allow paid data.
  • Protected from accidental edits. Saved files are locked with the Linux immutable attribute, so scripts and AI agents cannot change or delete them by mistake. QuantLake also checks the files regularly and downloads again any that were changed.
  • Usable by AI agents. Every command can print JSON. An included Agent Skill teaches agents such as Claude Code and Codex to find, read and request data.
  • Careful with your resources. It stays within each provider's rate limits, pauses downloads when free disk space runs low, retries failed downloads with backoff and checks every month for days that the provider has published again.
  • Two native binaries. You don't need a .NET runtime on the server.

Next steps

  • Installation: build the binaries, install the server and set your API keys.
  • Collect data: check what your plan includes and register what to collect.
  • Read data: find the files and query them.