Quant Infrastructure Series #1

An independent practicioner's introduction to building Quant Infrastructure.

Hello reader. This is part 1 of a series on quant infrastructure. Before I get into the meat I want to talk briefly about what this series is, what it hopes to achieve for the reader, and also to introduce myself as I’m likely unknown to you.

This series will be a no-nonsense guide/walkaround to building your own quant infrastructure a.k.a. a trading bot for cryptocurrencies. It will be from a perspective of an independent practitioner. Independent, meaning our workflow is that of a person working alone. We like simplicity, robustness and flexibility. We dislike friction and spending time on maintenance preferring to use it on market research and exploration instead.

The format will be that of a comprehensive discussion on the overall structure and then particular components. If the reader thinks he would prefer a step-by-step guide instead, he should not be overly discouraged as that may come in the future if it does not result directly from this discussion.

The goal is to become quickly useful to a motivated, intelligent person wishing to trade cryptocurrency (or other) markets without getting excessively bogged down in clerical programming work. Such a person should be able to use these articles to build robust quant infrastructure, and also to run and maintain it with relatively sparse time investment (thereby allowing more time to alpha (edge) research).

The infrastructure should be robust enough to allow for any mid-frequency (read: non-HFT) strategy regardless of the strategy’s particularities.

I also hope to supply the reader with insight on why certain components are the way they are and to save them from novice errors should they try to write a system without guidance or prior experience — something I’m guilty of and which has lost me both time and money in the past.

A prerequisite is that the reader is already competent in at least one programming language. Most code in these articles will be given in straightforward Rust with care taken to make it easily adaptable to other languages. Snippets related to research may also use Python.

I do not plan to discuss any currently profitable alphas (edges) in this series though we may discuss strategies for harnessing alphas and related topics.

In summary this will be:

1. A series of in-depth discussions on building quant infrastructure (a trading bot) from scratch.

2. From the perspective of an independent practicioner (see above).

3. We want a robust system that will allow us to trade any mid-frequency strategy without much additional effort.

Lastly to introduce myself: I’m an independent quant with a self-taught background, originally coming from software engineering. I’m active in cryptocurrency markets since mid-2021 and trading multiple alphas. Everything described here is a result of my learning and experience in this time and prior software engineering practice.

This is a good time to mention

HangukQuant

and

Quant’s Substack

from whom I’ve learnt a lot and who are also the main inspiration behind these and hopefully future articles. If you somehow found me and did not find them, you should go look at them right now (or soon after finishing this article). They are top of their class and the best ones I know of when it comes to all things Quant.

Disclaimer: Trading is a risky endeavour and I am not responsible for any losses you may incur implementing ideas learnt through these articles.

Disclaimer: This is not financial advice.

Bird’s Eye View of a Quant Infrastructure

With the introductions behind us let’s look at what we’re getting into.

Note from the author: we’re at the very first article and there’s a lot of room to experiment in terms of content, particularly based around reader interest. So if you, dear reader, have any requests or questions, you’re welcome to write them in a comment and I give you a reasonable chance it will be included.

Today’s article will be a very broad one. I expect future articles should be much more specific.

As Quants, our work pipeline (unapologetically simplified) is generally speaking like so:

Research → Backtest → Trade

Here we’ll focus on the latter two steps and make appropriate references to Research when relevant. A good infrastructure should support our research and quickly allow us to test ideas and then trade them without any code changes.

To give an idea of what we’re aiming for at the end, below is a simplified (though still reasonably complete) structure for our bot. This is a bird’s eye view, we will go down into particular component-level details in future articles one by one.

To avoid overwhelming the reader at the very beginning, I stripped this down to basic components and assumed only one thread. In a production system, we do not want this so that exchange requests or other slow/fallible things don’t delay the processing of market events. For now, though, the simplified structure will suffice.

// driver.rs

// Backtest and live trading is the same.

let instrument_store = ...;
let inventory = ...;
let order_executor = ...;
let statistics = ...;

// Notifications and commands for remote control. Here via Telegram.
let telegram = ...;

let strategy = PrintingFasterThanTheFed::new();

let feed = Feed::new();
let connector = ExchangeConnector::new(...);

connector.start_feed(feed);
connector.sync(instrument_store, inventory);

// An internal safety mechanism that will warn us if the system crashes.
let _on_quit = OnQuit(|| ...);

for event in feed {
  // Update state.
  instrument_store.on_event(event);
  inventory.on_event(event);
  order_executor.on_event(event);
  statistics.on_event(event);
  telegram.on_event(event);

  // Update strategy, potentially orders.
  strategy.on_event(event, instrument_store, inventory, ...);

  // Resync if we become out of sync with the exchange.
  if desync_detected {
    connector.sync(instrument_store, inventory);
  }

  // Do other updates such as fetching new instruments etc.
  ...
}

So we’ll have the following components:

• Instrument store - this is what keeps track of traded instruments (coins)

• Inventory - this is what keeps track of what positions and assets we currently have

• Order executor - no explanation needed I think

• Statistics - collects statistics

• Telegram - handles notifications and commands on Telegram (you may use any other platform though Telegram is quite nice)

• Connector - a connector to connect and translate our bot’s requests to and from the exchange

Beyond that, we will need to write code to download and store historical data for our traded cryptocurrencies and exchanges. Then we’ll need to use it to simulate historical trading (aka. backtest). For this, we’ll also need two more components:

• Scraper

• Storage

Other common components include:

• Dashboard - for live visual information about our trading

• Database - to record trades or historical event feed (.csv files tend to suffice)

• Visualizer - for live visualization and exploration of data (an optional, standalone research tool)

You’ve likely noticed we’re using an abstracted away Connector instead of a particular one for say Binance, Bybit or OKX. We want to start with this design in mind for three reasons:

1. To allow us to trade any exchange we write a connector for (duh! obviously).

2. To avoid introducing idiosyncracies of any particular exchange throughout our infrastructure, which would make it difficult to add other exchanges.

3. Because of (2) we get to write a lot simpler code throughout pretty much all components as we can control the assumptions that we can make about our Connector.

Okay, this will conclude today’s brief bird’s eye view of our Quant Trading System. It was short and frankly barely a sample.

In the next few articles, we will start building the foundation for our infrastructure. We will look at connecting to an exchange and scraping historical data from it. Binance is the obvious candidate. It also has a few quirks that we’ll cover.

After that, we will look at a Storage system that will allow for instant loading of historical data to Python or R for research and in Rust for backtesting. The code for this is simple, short and universally useful so I’ll likely write it as a package and release it for subscribers.

I invite the reader to leave questions, comments and/or requests for future articles.

Announcement

As I’m starting this Substack I do not know what to expect. Though important, infrastructure is rarely covered by existing online materials, possibly because it seems too base-level to write about for most quants/devs or perhaps because writing about research is more interesting.

Nevertheless, had I had access to good infrastructure information and tutorials when I started, it would’ve saved me years of experimentation and allowed me to commit more time to research (and do it more efficiently) ultimately leading to better alphas, greater profitability, fewer losses, etc. (or not, luck is a bitch, but you get the point).

As the Substack is new, my current aims are 1) to continue the series, 2) to understand if and how it is valuable to readers, and 3) to reach more readers.

With this in mind, future articles will likely rotate between paid-free every week and may include extras for subscribers such as implementation code. The specifics are very much up in the air.

That’s it for today!

Thank you for reading. I hope it was worth your time — or if it wasn’t yet, that it will be in the future!

Cheers!