QuantLake
GitHub
Documentation

Installation

Build the two binaries, install the server as a systemd service and set your API keys.

Requirements

  • Linux x64 with systemd and glibc 2.34 or later, for example Ubuntu 22.04 or later.
  • A filesystem that supports the immutable attribute for the data lake, such as ext4, xfs or btrfs.
  • sudo access, for the one-time installation.
  • The .NET 10 SDK to build the binaries. There are no prebuilt releases yet.
  • An API key for at least one provider:
    • Tardis.dev. Without a key, QuantLake can only download the first day of each month, which Tardis.dev provides for free.
    • Databento. Databento charges for most downloads, but some data may be included in your plan at no extra cost. QuantLake only downloads data that costs nothing unless you set a budget (see Paying for Databento data).

Plan your disk space

Order book data is large. The table shows rough sizes for one symbol and one day, in Parquet:

Data typeExampleSize per day
incremental_book_L2Tardis.dev, binance-futures BTCUSDT0.4–1 GB
book_tickerTardis.dev, binance-futures BTCUSDT0.1–0.3 GB
mbp-25Tardis.dev, binance-futures BTCUSDTabout 0.1 GB
trades, mbp-1, mbp-5Tardis.dev, binance-futures BTCUSDT20–40 MB
tradesDatabento, glbx.mdp3 ES.c.0about 8 MB
derivative_tickerTardis.dev, binance-futures BTCUSDTa few MB
liquidationsTardis.dev, binance-futures BTCUSDTtens of KB
ohlcv-1mDatabento, glbx.mdp3 ES.c.0tens of KB

Build the binaries

git clone https://github.com/supermomonga/quantlake.git
cd quantlake
dotnet publish src/QuantLake.Server/QuantLake.Server.csproj -c Release -r linux-x64 -o artifacts/linux-x64
dotnet publish src/QuantLake.CLI/QuantLake.CLI.csproj -c Release -r linux-x64 -o artifacts/linux-x64

If you use mise, mise run build does the same thing.

Install and start the server

sudo ./artifacts/linux-x64/quantlake-server install

This command does the following:

  • Copies both binaries to /usr/local/lib/quantlake/ and links them from /usr/local/bin/.
  • Creates a key pair for encrypting secrets. The private key goes to /etc/quantlake/secrets-key, which only root can read. The public key goes to /etc/quantlake/secrets-key.pub.
  • Registers a systemd service named quantlake-server. The service runs as your user, and its only extra privilege is the right to lock and unlock files.
  • Creates the data lake at ~/.local/share/quantlake.

To keep the data on another disk, add --lake-root /path/on/that/disk. The disk must support the immutable attribute.

If you use mise, mise run deploy builds the binaries and runs install in one go. Arguments after -- are passed to install, for example mise run deploy -- --lake-root /data/quantlake.

Set your API keys

Each key is read from standard input, so it does not end up in your shell history. QuantLake encrypts it and stores it in your configuration file (see Configuration).

# Paste your Tardis.dev key, then press Enter
read -rs KEY && printf %s "$KEY" | quantlake config set secrets.tardis_api_key --stdin

# Paste your Databento key, then press Enter
read -rs KEY && printf %s "$KEY" | quantlake config set secrets.databento_api_key --stdin

The server picks up the keys within a few seconds. You can also pass the Tardis.dev key during installation with install --api-key-stdin.

Check that the server is running

quantlake status

The first line shows the lake folder, and the second shows whether the server is running, its free disk space and whether file protection is on.

Next, collect your first data.