Documentation
Installation
Build the two binaries, install the server as a systemd service and set your API keys.
Requirements
- Linux x64 with systemd and glibc 2.34 or later, for example Ubuntu 22.04 or later.
- A filesystem that supports the immutable attribute for the data lake, such as ext4, xfs or btrfs.
sudoaccess, for the one-time installation.- The .NET 10 SDK to build the binaries. There are no prebuilt releases yet.
- An API key for at least one provider:
- Tardis.dev. Without a key, QuantLake can only download the first day of each month, which Tardis.dev provides for free.
- Databento. Databento charges for most downloads, but some data may be included in your plan at no extra cost. QuantLake only downloads data that costs nothing unless you set a budget (see Paying for Databento data).
Plan your disk space
Order book data is large. The table shows rough sizes for one symbol and one day, in Parquet:
| Data type | Example | Size per day |
|---|---|---|
incremental_book_L2 | Tardis.dev, binance-futures BTCUSDT | 0.4–1 GB |
book_ticker | Tardis.dev, binance-futures BTCUSDT | 0.1–0.3 GB |
mbp-25 | Tardis.dev, binance-futures BTCUSDT | about 0.1 GB |
trades, mbp-1, mbp-5 | Tardis.dev, binance-futures BTCUSDT | 20–40 MB |
trades | Databento, glbx.mdp3 ES.c.0 | about 8 MB |
derivative_ticker | Tardis.dev, binance-futures BTCUSDT | a few MB |
liquidations | Tardis.dev, binance-futures BTCUSDT | tens of KB |
ohlcv-1m | Databento, glbx.mdp3 ES.c.0 | tens of KB |
Build the binaries
git clone https://github.com/supermomonga/quantlake.git
cd quantlake
dotnet publish src/QuantLake.Server/QuantLake.Server.csproj -c Release -r linux-x64 -o artifacts/linux-x64
dotnet publish src/QuantLake.CLI/QuantLake.CLI.csproj -c Release -r linux-x64 -o artifacts/linux-x64If you use mise, mise run build does the same thing.
Install and start the server
sudo ./artifacts/linux-x64/quantlake-server installThis command does the following:
- Copies both binaries to
/usr/local/lib/quantlake/and links them from/usr/local/bin/. - Creates a key pair for encrypting secrets. The private key goes to
/etc/quantlake/secrets-key, which only root can read. The public key goes to/etc/quantlake/secrets-key.pub. - Registers a systemd service named
quantlake-server. The service runs as your user, and its only extra privilege is the right to lock and unlock files. - Creates the data lake at
~/.local/share/quantlake.
To keep the data on another disk, add --lake-root /path/on/that/disk. The disk must support the immutable attribute.
If you use mise, mise run deploy builds the binaries and runs install in one go. Arguments after -- are passed to install, for example mise run deploy -- --lake-root /data/quantlake.
Set your API keys
Each key is read from standard input, so it does not end up in your shell history. QuantLake encrypts it and stores it in your configuration file (see Configuration).
# Paste your Tardis.dev key, then press Enter
read -rs KEY && printf %s "$KEY" | quantlake config set secrets.tardis_api_key --stdin
# Paste your Databento key, then press Enter
read -rs KEY && printf %s "$KEY" | quantlake config set secrets.databento_api_key --stdinThe server picks up the keys within a few seconds. You can also pass the Tardis.dev key during installation with install --api-key-stdin.
Check that the server is running
quantlake statusThe first line shows the lake folder, and the second shows whether the server is running, its free disk space and whether file protection is on.
Next, collect your first data.