Read more

(Part 2)

In Part 1, we wrote a basic event-driven simulator/backtester using candle data. In this Part 2, we extend it to make the simulator realistic by adding costs and funding. We also add support for simulating an arbitrary number of markets simultaneously.

Table of Contents

Part 1:

• Introduction

• Simulator/backtester architecture

• Preparing the data

• Simulating a single market

• Simulating market orders

Part 2:

• Simulating trading costs

• Simulating funding

• Simulating many markets

• Finish

• Subscriber materials (source code)

Simulating Trading Costs

Read more

Table of Contents

Part 1:

• Introduction

• Simulator/backtester architecture

• Preparing the data

• Simulating a single market

• Simulating market orders

Part 2:

• Simulating trading costs

• Simulating funding

• Simulating many markets

• Finish

• Subscriber materials (source code)

Introduction

In the last article, we looked at how markets work and at simulating them in theory.

Today we w…

Read more

First, I want to thank all 417 of you for reading and the support to date. It’s been almost 3 months since we started this publication on Quant Infrastructure.

In that time we have covered many topics including about 70%-ish of the trading part that I had in mind when I began writing the series from a cafe in Tokyo. Some highlights that you thought were particularly valuable (I’m basing this on views and likes) include:

• An extremely fast Time-Series Storage System

• An Inventory component for tracking inventory

• An Order Executor component for managing limit orders

Most importantly, the publication is no longer the vaporware that it was when we had one article back then. In addition to the already covered topics, we continue to regularly publish new articles and tutorials for Quant Infrastructure.

To reflect the new value of the publication, the price will increase for new subscribers.

The price will not change for existing subscribers, however. Everyone currently subscribed will keep their current, lower rate. This will continue to remain true in the indefinite future for everybody. Thank you, guys!

In addition, anyone who got left out and wishes to subscribe can still get a 20%-off within the next 7 days:

Get 20% off for 1 year

Okay, I also wanted to write a bit about what’s coming in the future. On the surface, we will continue with the series. Those of you who read the last article will know that we will write a market simulator/backtester next.

However, you may have noticed I experimented a bit with the format recently. I’m gradually learning and figuring out how to improve the publication. Some of you have DMd me questions and ideas on Twitter and I’ve been keeping a list of ideas for articles that occurred to me in the course of the daily grind.

I’m aware this is very vague, but, what I’m trying to say is to expect more high-quality (and improving!) content in the future.

Thank you all for your reading and support and wishing you successful trading.

Cheers!

-TaiwanQuant

Share TaiwanQuant’s Substack

• Introduction

• An Event-Driven Market Simulator

• Market Internals & The Order Book

• Simulating MFT

  • Market Orders

  • Limit Orders

• Simulating HFT

• Conclusion

Introduction

In the last article, we built an OrderExecutor component and looked at robustly managing limit orders on the exchange. In this article, we introduce the reader to market simulation. Future articles will cover writing a complete, event-driven backtester from candle data.

The article will introduce the concepts relevant to market simulation and lay the groundwork for writing our own simulator in a future article.

This is a broad topic so we will limit ourselves to the event-driven simulator we intend to write.

We will begin with a brief overview to look at what simulators are useful for (in addition to backtesting), and how we will use ours with the infrastructure covered so far in the series.

We will then look at the market internals of CEX Crypto exchanges, crucially the order book, to inform our understanding and intuition of how markets work. We will do this with the goal of simulating markets well and in a way that corresponds to reality — in addition to gaining a better understanding generally.

In the main part of the article, we will look at simulating a market using candlestick data. We will look at the challenges posed by candles, which lack much desirable information, and how to overcome them. We will focus on simulating realistic fill prices and a cost model for achieving this.

The last section briefly addresses concepts relevant to simulating HFT such as latency and the order book's order queue.

Crucially, our end goals are:

1. To produce a reliable simulator which we understand and can therefore trust and which will be useful in practice.

2. To improve our understanding of markets.

By proxy, I also wish to save the reader from dealing with any headaches stemming from an unreliable one.

As usual, our focus is on simplicity, robustness and correctness.

This article is an introduction and contains no code, which will come in the future.

Let's begin! 12

Read more

In this article, we look at tracking and managing orders and build an OrderExecutor for this purpose. Orders require a different approach from the Inventory because we actively control them as opposed to passively taking information from the exchange.

The technique we will show is commonly known as orders-in-flight. The implementation we will build will be simple and robust to errors, both theoretical and those encountered in practice during real trading.

We will address common problems such as duplicate orders and ghost orders and ensure that our order tracking is consistent with our inventory — which we covered in the previous article from the Inventory component’s side.

For consistency, I assume the Inventory component is similar to what we described in the previous part of the series.

Today’s article is free to read. As per usual, the complete source code is attached at the end of the article for paying subscribers.

We will use straightforward Rust to allow readers using other languages to follow and adapt the code to their language of choice relatively easily if they wish.

The article is exchange-agnostic and the techniques shown work on most if not all CEX exchanges such as Binance, Bybit, etc. I have not tested them with any DEXes, however — though I suspect they work there just as well.

Let’s begin.12

Read more

In this article, we show how to implement an efficient fixed-point decimal on top of a native integer. This implementation will be suitable for handling and storage of market data, particularly high-frequency order book data.

The technique is attractive to us for the following reasons:

1. It is just as performant as regular integers for many operations.

2. It uses the same space and representation as a regular integer, usually 64 or 128 bits, which makes it possible to store it as such.

3. It is portable across languages, in particular, we can use it directly from Python and R.

4. To the best of my knowledge, it is the simplest, fastest and most memory-efficient representation of precise decimals.

Its caveat is that it is constrained by range:

1. The 64-bit variant with 8 decimal digits is suitable for Crypto futures markets.

2. The 128-bit variant with 16 decimal digits is preferred to cover spot markets due extreme cheapness of certain shitcoins.

Here we implement the 64-bit variant. The 128-bit variant can be implemented in an analogous way or at the same time via generics/codegen.

The complete source code was released early with Subscriber Materials for Part 1. We attach it to the end of this article again for the reader’s convenience.

Let’s dive in.12

Read more

A lot of new readers subscribed this week. Welcome, guys! Happy to have you here and I hope to make this publication worth your time.

To give some context for new readers. We’re currently doing component-level discussions and guides for Quant Infrastructure. The most recent articles have gone and will continue like this:

1. Inventory — robust inventory tracking (completed).

2. OrderExecutor — limit order submission and management via orders-in-flight.

3. InstrumentStore — keeping track of instruments traded by the system.

There are also articles on a (very) performant time series data store and an exchange client. I have things planned for after and may do a poll to see what readers would like to see in the future.

This article is an interlude to the main series and deals with different kinds of precise decimals — we postponed the topic in previous articles. It is the first time we write an interlude in this format.

I hope that you will find this publication useful and thank you for your reading!

-TaiwanQuant

Get 10% off for 1 year

Introduction

In today's article, we discuss the topic of precise decimal numbers in quantitative trading systems and implement a fast precise-decimal type suitable for handling and storage of high-frequency data.

We will mention where, why and how precise decimals are used and discuss tradeoffs between three different implementations from the perspective of Quant Infrastructure: two big-decimal types and a fast fixed-point type.

We will implement our own type that is suitable for simulating, trading, and storage of high-frequency order book data and is widely used in HFT. It is likewise useful in MFT and generally for Quant and trading systems.

This article is an interlude to the main infrastructure series. It is written, aside from the implementation part, as a high-level overview rather than a low-level guide, which is a departure from the main series. We also break character by addressing a topic that is closer to HFT rather than MFT.

The article is written mainly from the perspective of MFT as that is where most of my work and experience is in.

I expect this to be most useful to those readers handling or wishing to handle high-frequency data such as order book updates or ticker updates, or other kinds of market data for which preserving discreteness is required and thus floats are unsuitable. We will discuss when this occurs and how to approach these cases.

It will also be relevant to those concerned with the performance and robustness of their infrastructures in both HFT and MFT, especially wrt. simulation times and data store performance.

The article’s purpose is (1) to serve as an introductory and high-level overview of handling precise decimal quantities in Quant trading systems, and (2) to add a useful tool to our Quant engineering toolbox.

It is split into two parts:

• In Part 1 (this one) we make address the topic at a high level. The majority of our discussion is here.

• In Part 2 we implement a performant decimal type for high-frequency order book data for MFT and HFT for this article’s Subscriber Materials.

Part 2 will release in a few days up to a week. After that, we will resume the main line with the OrderExecutor article.

Let’s dive in.12

Read more

Today we write a robust Inventory component for our Quant Infrastructure to correctly and reliably track the traded inventory on the exchange. We will maintain a local copy of all wallet assets and open positions and make it available to other components. Our Inventory will be exchange-agnostic and suitable for trading live as well as backtesting without any code changes and within the same infrastructure.

We will also address reliability problems encountered in the course of trading live on Crypto exchanges, which, to my knowledge, has not been done anywhere previously.

This is the first article out of three that cover the three core components of the trading part of the infrastructure: InstrumentStore, Inventory and OrderExecutor. We will cover these in the next few articles and make an interlude for fixed-point numbers. The next article will introduce the idea of orders-in-flight and show how to neatly handle limit orders in our Quant Infrastructure.

A typical trading loop on a Crypto CEX will see us passively ingest events from the exchange — usually through a WebSocket connection— which will carry updates about the state of our account, orders, traded instruments, prices, the order book, etc., whilst we occasionally make requests to the exchange via an HTTP API.

Robust tracking of state on the exchange carries some nuance:

1. The data feed itself can malfunction. Events can be skipped, delayed, or arrive out of order.

2. When related entities are updated separately — as is typically the case with inventory and orders — the tracked local state can become inconsistent and cause erroneous behaviour by our infrastructure.

An example of erroneus trading caused by orders-inventory inconsistency:

Suppose we want to buy 1 BTC. We post an order and obtain a fill. The event with the filled order arrives so naively we think we no longer have an order posted; but the event about the new inventory hasn’t arrived yet. Between those two events, we may mistakenly think we don’t have the desired amount of BTC and don’t have an order posted and may mistakenly post another order and so on.

We will address these issues in the article.

We will begin by defining the exchange events which we will ingest, followed by the implementation of a simple inventory component as a base case. Then, we will make our inventory component robust to errors. Last, we will show how we can handle the orders-inventory inconsistency.

Today’s article is free to read. The complete source code is available for paying subscribers in the Subscriber Materials section at the end of the article as per usual.

Let’s dive in.12

Read more

We show a data store implementation suitable for loading arbitrary amounts of time-series data instantly (at hardware performance limit) to Python and Rust. The technique can be adapted to other languages. The basic implementation is very short at around 150 lines of code and is suitable for independent practitioners and Quant teams alike.

Broadly, the data store has three roles:

1. Destination/warehouse for historical data.

2. Data source for research/exploration.

3. Data source for simulations/backtests.

A good implementation is valuable because performance and capacity gains here will tend to translate into (usually non-linear) gains everywhere else — most importantly in research and backtesting.

For example, ideally, we would like data access to have as little friction as possible — ideally, we would like it to be instant and not have a size constraint of how much data we can handle at once. We would also like our backtests to not be bottlenecked by the data feed or for the data feed to take computing resources from the backtest.

This can be hard because computing resources are finite and easily exhausted by market data — to the point where some ideas are impractical or impossible to research even on the most expensive AWS machines. To make them possible we have to be smart about how we manage data.

Most Quants will start by doing this wrong — typically by reaching for a third-party database or a columnar data file format. In practice, a much better solution requires barely 150 lines of code and no third-party system.

The databases are inadequate for the data sizes we need to work with. For example, a naive select query on a SQL database with 1e6 records (small by Quant standards — e.g. 1-minute data for a single perp contract) can easily take minutes to complete and parse rows into usable data types. We expect to commonly deal with 1e9 and higher at any one time.

Columnar data storage file formats such as Apache Parquet are much better here though still complicated and very slow vis-a-vis theoretical limits and the solution we describe here.

Instead, we show a technique that addresses all of the earlier points. Loading of most data is instant and hardware resource usage minimal. Our system is able to run within a few percent of the maximum disk throughput. It is also much simpler and orders of magnitude more performant than most (all?) third-party alternatives.

The method is portable across programming languages, here Rust and Python.

To give a demonstration of what is possible, the following Python code:

%%time
klines = read_klines('btc/usdt', 'futures', 'binance')
klines.shape

takes this long to complete with the following result:

CPU times: total: 0 ns
Wall time: 0 ns
(1926509,)

Yes, that is 0 nanoseconds. The above code loads 1,926,509 1-minute candlesticks for Binance USD-stablecoin margined futures into a Numpy array in Python.

To show that our technique is fully usable, we compute the mean price and see again how long this takes:

%%time
np.mean(klines['close'])

results in:

CPU times: total: 0 ns
Wall time: 10.3 ms
26704.60773270721

This test was run on a mid-range computer from 3 years ago with an M.2 SSD.

We provide one more benchmark from Rust’s side, which is more relevant for backtesting. This time we load all BTC-quoted pairs from Binance Spot and compute an EWMA for each of them:

EWMA all BTC pairs
---------------------------
Time       : 18.42 seconds
Klines     : 614098628
Throughput : 2798.56 MB/s
Klines/s   : 33346791.62 klines/s

I had redone the benchmark since the last article announcement after reminding myself that I had an Ethereum node syncing in the background and eating my I/O. The result is 2x the speed now compared to the last article’s announcement.

As before, the complete source code is attached to the article for the reader’s benefit. The implementation is simple and takes around 150 lines of code.

The article itself is a commentary on architecture and a how-to guide. The reader should be able to easily extend the data store to support more data types based on his need, such as aggregated trades and order book updates. He should also be able to adapt the source code for his language of choice.

Let’s dive in.12

Read more

This topic is a natural place to begin the series. As Quants, we need historical data to research and backtest ideas and exchanges expose large amounts of it via their APIs. Third-party clients often either do not exist or are subpar in one way or another.

The article is envisioned as a template/how-to guide for building an in-house client for exchanges that expose an HTTP API. This includes all common CEX and hybrid exchanges: Binance, Bybit, OKX, Kucoin, dYdX, etc.

For practical reasons, we seek to convey the idea rather than include every single line of code of the completed client.

The reader will be happy to learn that exchange APIs are very similar to each other such that being able to integrate Binance Futures means we are also able to integrate Bybit Spot and so on.

The article demonstrates how to write a client that is simple, understandable, robust to errors, and which we are able to trust, update, and fix when necessary. We think these properties are necessary if we want to use the client to trade real money, and if we value our time and want to stay maximally headache-free as Quants — at least on the infrastructure side.

We also lay the groundwork for the next article, which will show a performant storage system, which is the natural progression after a client used to download data.

In the long term, we plan to use the client to allow us to trade.

The complete source code in Rust will be available at the end of the article for paying subscribers.

For completeness, other ways to acquire data include:

1. Buy it from a vendor.

2. Record exchange’s live feeds. This method is rarely useful outside of HFT as the data needed usually spans years and so recording it is impractical for us.

3. Some exchanges will also make their data available for download outside the API. For example, here’s Binance’s public data catalogue released recently.

Disclaimer: Trading is a risky endeavour and I am not responsible for any losses you may incur implementing ideas learnt through these articles.

Disclaimer: This is not financial advice.

Read more

This series will be a no-nonsense guide/walkaround to building your own quant infrastructure a.k.a. a trading bot for cryptocurrencies. It will be from a perspective of an independent practitioner. Independent, meaning our workflow is that of a person working alone. We like simplicity, robustness and flexibility. We dislike friction and spending time on maintenance preferring to use it on market research and exploration instead.

The format will be that of a comprehensive discussion on the overall structure and then particular components. If the reader thinks he would prefer a step-by-step guide instead, he should not be overly discouraged as that may come in the future if it does not result directly from this discussion.

The goal is to become quickly useful to a motivated, intelligent person wishing to trade cryptocurrency (or other) markets without getting excessively bogged down in clerical programming work. Such a person should be able to use these articles to build robust quant infrastructure, and also to run and maintain it with relatively sparse time investment (thereby allowing more time to alpha (edge) research).

The infrastructure should be robust enough to allow for any mid-frequency (read: non-HFT) strategy regardless of the strategy’s particularities.

I also hope to supply the reader with insight on why certain components are the way they are and to save them from novice errors should they try to write a system without guidance or prior experience — something I’m guilty of and which has lost me both time and money in the past.

A prerequisite is that the reader is already competent in at least one programming language. Most code in these articles will be given in straightforward Rust with care taken to make it easily adaptable to other languages. Snippets related to research may also use Python.

I do not plan to discuss any currently profitable alphas (edges) in this series though we may discuss strategies for harnessing alphas and related topics.

In summary this will be:

1. A series of in-depth discussions on building quant infrastructure (a trading bot) from scratch.

2. From the perspective of an independent practicioner (see above).

3. We want a robust system that will allow us to trade any mid-frequency strategy without much additional effort.

Lastly to introduce myself: I’m an independent quant with a self-taught background, originally coming from software engineering. I’m active in cryptocurrency markets since mid-2021 and trading multiple alphas. Everything described here is a result of my learning and experience in this time and prior software engineering practice.

This is a good time to mention and from whom I’ve learnt a lot and who are also the main inspiration behind these and hopefully future articles. If you somehow found me and did not find them, you should go look at them right now (or soon after finishing this article). They are top of their class and the best ones I know of when it comes to all things Quant.

Disclaimer: Trading is a risky endeavour and I am not responsible for any losses you may incur implementing ideas learnt through these articles.

Disclaimer: This is not financial advice.

Bird’s Eye View of a Quant Infrastructure

With the introductions behind us let’s look at what we’re getting into.

Note from the author: we’re at the very first article and there’s a lot of room to experiment in terms of content, particularly based around reader interest. So if you, dear reader, have any requests or questions, you’re welcome to write them in a comment and I give you a reasonable chance it will be included.

Today’s article will be a very broad one. I expect future articles should be much more specific.

As Quants, our work pipeline (unapologetically simplified) is generally speaking like so:

Research → Backtest → Trade

Here we’ll focus on the latter two steps and make appropriate references to Research when relevant. A good infrastructure should support our research and quickly allow us to test ideas and then trade them without any code changes.

To give an idea of what we’re aiming for at the end, below is a simplified (though still reasonably complete) structure for our bot. This is a bird’s eye view, we will go down into particular component-level details in future articles one by one.

To avoid overwhelming the reader at the very beginning, I stripped this down to basic components and assumed only one thread. In a production system, we do not want this so that exchange requests or other slow/fallible things don’t delay the processing of market events. For now, though, the simplified structure will suffice.

// driver.rs

// Backtest and live trading is the same.

let instrument_store = ...;
let inventory = ...;
let order_executor = ...;
let statistics = ...;

// Notifications and commands for remote control. Here via Telegram.
let telegram = ...;

let strategy = PrintingFasterThanTheFed::new();

let feed = Feed::new();
let connector = ExchangeConnector::new(...);

connector.start_feed(feed);
connector.sync(instrument_store, inventory);

// An internal safety mechanism that will warn us if the system crashes.
let _on_quit = OnQuit(|| ...);

for event in feed {
  // Update state.
  instrument_store.on_event(event);
  inventory.on_event(event);
  order_executor.on_event(event);
  statistics.on_event(event);
  telegram.on_event(event);

  // Update strategy, potentially orders.
  strategy.on_event(event, instrument_store, inventory, ...);

  // Resync if we become out of sync with the exchange.
  if desync_detected {
    connector.sync(instrument_store, inventory);
  }

  // Do other updates such as fetching new instruments etc.
  ...
}

So we’ll have the following components:

• Instrument store - this is what keeps track of traded instruments (coins)

• Inventory - this is what keeps track of what positions and assets we currently have

• Order executor - no explanation needed I think

• Statistics - collects statistics

• Telegram - handles notifications and commands on Telegram (you may use any other platform though Telegram is quite nice)

• Connector - a connector to connect and translate our bot’s requests to and from the exchange

Beyond that, we will need to write code to download and store historical data for our traded cryptocurrencies and exchanges. Then we’ll need to use it to simulate historical trading (aka. backtest). For this, we’ll also need two more components:

• Scraper

• Storage

Other common components include:

• Dashboard - for live visual information about our trading

• Database - to record trades or historical event feed (.csv files tend to suffice)

• Visualizer - for live visualization and exploration of data (an optional, standalone research tool)

You’ve likely noticed we’re using an abstracted away Connector instead of a particular one for say Binance, Bybit or OKX. We want to start with this design in mind for three reasons:

1. To allow us to trade any exchange we write a connector for (duh! obviously).

2. To avoid introducing idiosyncracies of any particular exchange throughout our infrastructure, which would make it difficult to add other exchanges.

3. Because of (2) we get to write a lot simpler code throughout pretty much all components as we can control the assumptions that we can make about our Connector.

Okay, this will conclude today’s brief bird’s eye view of our Quant Trading System. It was short and frankly barely a sample.

In the next few articles, we will start building the foundation for our infrastructure. We will look at connecting to an exchange and scraping historical data from it. Binance is the obvious candidate. It also has a few quirks that we’ll cover.

After that, we will look at a Storage system that will allow for instant loading of historical data to Python or R for research and in Rust for backtesting. The code for this is simple, short and universally useful so I’ll likely write it as a package and release it for subscribers.

I invite the reader to leave questions, comments and/or requests for future articles.

Announcement

As I’m starting this Substack I do not know what to expect. Though important, infrastructure is rarely covered by existing online materials, possibly because it seems too base-level to write about for most quants/devs or perhaps because writing about research is more interesting.

Nevertheless, had I had access to good infrastructure information and tutorials when I started, it would’ve saved me years of experimentation and allowed me to commit more time to research (and do it more efficiently) ultimately leading to better alphas, greater profitability, fewer losses, etc. (or not, luck is a bitch, but you get the point).

As the Substack is new, my current aims are 1) to continue the series, 2) to understand if and how it is valuable to readers, and 3) to reach more readers.

With this in mind, future articles will likely rotate between paid-free every week and may include extras for subscribers such as implementation code. The specifics are very much up in the air.

That’s it for today!

Thank you for reading. I hope it was worth your time — or if it wasn’t yet, that it will be in the future!

Cheers!