Quant Infrastructure #4 - Keeping Track of Inventory

How to stay in sync with an asynchronous exchange.

Trading on an exchange requires us to keep track of the state of our account and markets in close to real-time. Our trading infrastructure and the exchange run asynchronously with each other and communicate over the network, which is at times unreliable and always carries some latency.

Today we write a robust Inventory component for our Quant Infrastructure to correctly and reliably track the traded inventory on the exchange. We will maintain a local copy of all wallet assets and open positions and make it available to other components. Our Inventory will be exchange-agnostic and suitable for trading live as well as backtesting without any code changes and within the same infrastructure.

We will also address reliability problems encountered in the course of trading live on Crypto exchanges, which, to my knowledge, has not been done anywhere previously.

This is the first article out of three that cover the three core components of the trading part of the infrastructure: InstrumentStore, Inventory and OrderExecutor. We will cover these in the next few articles and make an interlude for fixed-point numbers. The next article will introduce the idea of orders-in-flight and show how to neatly handle limit orders in our Quant Infrastructure.

A typical trading loop on a Crypto CEX will see us passively ingest events from the exchange — usually through a WebSocket connection— which will carry updates about the state of our account, orders, traded instruments, prices, the order book, etc., whilst we occasionally make requests to the exchange via an HTTP API.

Robust tracking of state on the exchange carries some nuance:

1. The data feed itself can malfunction. Events can be skipped, delayed, or arrive out of order.

2. When related entities are updated separately — as is typically the case with inventory and orders — the tracked local state can become inconsistent and cause erroneous behaviour by our infrastructure.

An example of erroneus trading caused by orders-inventory inconsistency:

Suppose we want to buy 1 BTC. We post an order and obtain a fill. The event with the filled order arrives so naively we think we no longer have an order posted; but the event about the new inventory hasn’t arrived yet. Between those two events, we may mistakenly think we don’t have the desired amount of BTC and don’t have an order posted and may mistakenly post another order and so on.

We will address these issues in the article.

We will begin by defining the exchange events which we will ingest, followed by the implementation of a simple inventory component as a base case. Then, we will make our inventory component robust to errors. Last, we will show how we can handle the orders-inventory inconsistency.

Today’s article is free to read. The complete source code is available for paying subscribers in the Subscriber Materials section at the end of the article as per usual.

Let’s dive in.¹²

Exchange Interface

We begin by defining what it is we want to track.

In this article, we will use the example of perpetual futures with cross-collateral.

Since I suspect many readers will want to implement spot, the primary difference between spot and futures (as far as tracking is concerned) is that for futures there are two entities to track: the wallet assets, which are used as collateral, and positions. The spot market has only one: wallet assets. The reader should have no problems in converting the code to spot should he wish to.

In the first article of the series, we have shown a snippet similar to the one below. It will come in useful now to give us some context on how our Inventory component will be used within the trading loop:

// driver.rs

fn run() {
    let mut instrument_store = InstrumentStore::new();
    let mut inventory = Inventory::new();
    let mut order_executor = OrderExecutor::new();
    let mut strategy = Strategy::new();

    // Trading loop.
    loop {
        let event = next_event();

        instrument_store.on_event(&event);
        inventory.on_event(&event);
        order_executor.on_event(&event);
        strategy.on_event(&event);        
    }
}

I write some simple types below to represent events and the different entities we will need. These are based on Bybit Derivatives and are similar across exchanges.

It is a good idea to have standard types such as Position, WalletAsset, etc. defined in the core infrastructure independently of exchange-specific code. This will allow us to exclude particularities of any one exchange from the core infrastructure and make it easier to add more exchanges in the future.

The (simplified) types look as follows:

// types.rs

pub enum Event {
    Order(Order),
    Position(Position),
    Wallet(WalletAsset),
}

pub struct Order {
    pub symbol: String,
    pub side: OrderSide,
    pub last_exec_qty: Option<FixedPoint>,
}

pub struct Position {
    pub symbol: String,
    pub side: Option<PositionSide>,
    pub size: FixedPoint,
    pub entry_price: FixedPoint,
}

pub struct WalletAsset {
    pub asset: String,
    pub balance: FixedPoint,
}

We haven’t dealt with writing a generic exchange connector yet. To make this code useful first, the reader is welcome to use the event types for his exchange of choice. It is a good idea to convert them into distinct, standard types similar to the ones above in the Inventory component or earlier in the pipeline.

Basic Inventory

The purpose of an inventory component (as mentioned) is to track and expose to other components the state of our wallet and positions. For convenience, we may also expose some additional information via utility functions:

• Size of collateral;

• Positions;

• Unrealized PNL;

• and so on.

A strategy component may use this information to calculate position sizes, a statistics component may use this for statistics, etc.

We can write a basic Inventory that does not concern itself with robustness yet like this:

// inventory.rs

pub struct Inventory {
    wallet: HashMap<String, WalletAsset>,
    positions: HashMap<String, Position>,
}

impl Inventory {
    pub fn new() -> Self {
        Self {
            wallet: HashMap::new(),
            positions: HashMap::new(),
        }
    }

    pub fn on_event(&mut self, event: &Event) {
        match event {
            Event::Position(x) => {
                if !x.size.is_zero() {
                    self.positions.insert(x.symbol, x);
                } else {
                    self.positions.remove(&x.symbol);
                }
            }
            Event::Wallet(x) => {
                if !x.balance.is_zero() {
                    self.wallet.insert(x.asset, x);
                } else {
                    self.wallet.remove(&x.asset);
                }
            }
            _ => ()
        }
    }

    pub fn get_wallet(&self, asset: &str) -> Option<&WalletAsset> {
        self.wallet.get(asset)
    }

    pub fn get_position(&self, symbol: &str) -> Option<&Position> {
        self.positions.get(symbol)
    }
}

This is rather trivial. The reason we remove zero balances and positions instead of keeping them is that it will make a visible performance difference during backtests by making iteration shorter (not shown here).

The reader may add the aforementioned utility functions as he finds them useful, for example, to calculate unrealized PNL, or the total collateral size (both require passing in contract prices).

This naive version could be dangerous to use directly as we do not account for the possibility of an erroneous state occurring, which is likely to eventually happen in real trading. We want to address this and make our inventory robust to errors.

We also want to solve the orders-inventory consistency problem mentioned in the introduction.

Robust Inventory

The exchange’s event feed is the only way to get reliable, sequential information in close to real-time on most exchanges. The exchange’s HTTP API is typically unsuitable because it is rate-limited and can return old information due to caching and/or an eventually-consistent API contract.

It seems then that we should always trust the event feed. In practice, weird things happen and we would like to build a degree of redundancy into our system. Especially as a bad inventory can be very dangerous to our capital.

Most of what I show in this section is the result of my own experience and/or comes from competent acquaintances. To my knowledge, it is not discussed at length anywhere else in published materials.

An example of things that could go wrong are:

• Latency — events have historically been delayed even by a minute during high volatility/exchange load;

• Inventory-orders inconsistency;

• Missing or out-of-order events;

• General junkiness of various exchanges — included as a means of protecting against anything else that could be there.

We want to do three things:

1. Detect when the inventory state is likely bad.

2. Resynchronize it.

3. Warn about desyncs and allow for manual intervention.

It is my belief that most problems have simple solutions that are very close to optimal so here too we will use a simple approach.

First, we recognize that errors don’t happen often and that in most cases we can group them as simply bad inventory states and handle them generally rather than one by one.

Our job is thus reduced to ensuring that any erroneous state S’ is found and naturally transitions to a correct state S. In a somewhat unrigorous but visually helpful mathematical notation:

\(S' \rightarrow S\)

We first add a bad flag so we can mark an inventory as bad and in need of resynchronization:

// inventory.rs

pub struct Inventory {
    ...
    is_bad: bool,
}

impl Inventory {
    pub fn mark_bad(&mut self) {
        self.is_bad = true;
    }

    pub fn set(&mut self, account: Account) {
        self.wallet = account.wallet;
        self.positions = account.positions;
        self.is_bad = false;
    }
}

Then in our trading loop we check it and resynchronize the inventory if needed:

// driver.rs

fn run() {
    // Trading loop.
    loop {
        ...
        if inventory.is_bad() {
            // Reload inventory from the API.
            if exchange.use_api_to_reload_inventory() {
                let account = exchange.get_account();
                inventory.set(account);
            }

            // Reload inventory from the event feed.
            else if exchange.use_data_feed_to_reload_inventory() {            
                inventory.clear();
                data_feed.reconnect();
            }
        }
    }
}

This approach has caveats. As mentioned before, the exchange API could return old information. There is also no sequential consistency guarantee between when we set the inventory and continue listening to events.

The idea is that most of the time, this will be correct — when it isn’t, we will simply detect a bad inventory state again and repeat the process until it is.

We can use the fact that some (if not most) exchanges send the complete inventory state every time we connect to the event feed. I tend to use the API method out of inertia (it is what I have been using historically) but include the second method for the reader’s benefit as it is likely better.

What is left is to figure out when to mark an inventory as bad. The bad news is a perfect answer may not exist as we can’t atomically compare our local state to the one on the exchange.

However, below I list some approaches that should catch all problems eventually. We set the bad flag when:

1. A request fails with an error related to the inventory such as an insufficient funds error.

2. A comparison with information fetched from the API has not matched for longer than a few minutes.

3. The infrastructure reconnects to the exchange’s data feed.

These checks have some caveats. The first one assumes that our code is correct lest we may end up with a request-resync loop (as opposed to just a request loop — either is undesirable). The second should be done in a separate thread where it won’t block any other meaningful work. It should also be fine to resync on any failed requests if the reader wishes to.

A notification and a way to reload inventory manually are desirable. I mention this for completeness since we haven’t covered monitoring or remote control yet.

The last remaining problem is solving inventory-orders consistency.

Inventory-Orders Insistency

A common problem we will bump into is the inventory-orders consistency issue mentioned in the introduction. This occurs when order and position updates occur in separate events after one is handled but before the other is.

At the time of ‘?’ we may mistakenly think we have neither the order nor the position:

\(Order\space Event \rightarrow\space ? \rightarrow Position\space Event\)

For the time being, the reader may assume order tracking is similar to inventory tracking with the last known state updated via events as a mental model.

The way I like to handle this is to reconstruct positions from order fills. We will be able to use order events and thus maintain consistency with order tracking, which will also use them.

The same method will also double as another redundancy check: if the inventories tracked via position events and order events diverge for longer than two minutes, either must be wrong, and we should resync.

We will see in the next article that orders are subject to their own distinct constraints, which will further help us ensure that the inventory is correct.

We can do the parallel inventory technique in this way:

// inventory.rs

const CHECK_FREQUENCY: i64 = 1000; // 1 second
const BAD_AFTER: i64 = 60000; // 1 minute

pub struct Inventory {
    ...
    positions_from_fills: HashMap<String, FixedPoint>,
    last_check: i64,
    last_fail: Option<i64>,
}

impl Inventory {
    pub fn get_position_orders_consistent(
        &self,
        symbol: &str
    ) -> Option<FixedPoint> {
        self.positions_from_fills.get(symbol).copied()
    }

    pub fn on_event(&mut self, event: &Event, local_time: i64) {
        match event {
            ...
            Event::Order(x) => {
                if let Some(q) = x.last_exec_qty {
                    let p = self
                        .positions_from_fills
                        .entry(x.symbol)
                        .or_default();
                    *p += q;
                    if p.is_zero() {
                        self.positions_from_fills.remove(&x.symbol);
                    }
                }
            }
        }

        // Check every second (specified by CHECK_FREQUENCY) or so 
        // rather than every event.
        let time_since_last_check = local_time - self.last_check;
        if !self.is_bad && time_since_last_check >= CHECK_FREQUENCY {
            self.last_check = local_time;
            let positions: HashMap<_, _> = self
                .positions
                .iter()
                .map(|(s, p)| (s, p.signed_size()))
                .collect();
            if positions != self.positions_from_fills {
                if self.last_fail.is_none() {
                    self.last_fail = Some(self.last_check);
                }
            } else {
                self.last_fail = None;
            }
            if let Some(last_fail) = self.last_fail {
                if local_time - last_fail > BAD_AFTER {
                    self.is_bad = true;
                }
            }
        }
    }
}

I’ll summarize the important points:

• Any component concerned with orders-positions consistency can now use get_position_orders_consistent() to get position sizes consistent with orders.

• We perform the consistency check every second rather than every event to avoid pessimizing performance for backtests (we could also disable the check). We pass in time rather than get it from the OS to support simulated time.

With this, we have written an Inventory component that can track our inventory on the exchange and made it robust to errors. We’re finished.

Finishing Thoughts

With this, we conclude our discussion on the Inventory component. We have discussed its role, created a basic implementation, and made it robust to errors. We have also shown how to make inventory and order tracking agree with each other.

The next article will introduce an OrderExecutor to handle limit orders through orders-in-flight.

There may be an article on fixed-point numbers before that. They are relevant because orders need to comply with filters and we don’t have a way to do this yet with what we have written so far (we have been using a stub FixedPoint type so far).

That’s it for today!

I hope the article was useful to you and thank you for your reading and support!

Cheers!