Jul 15, 20226 min read

Infrastructure of algorithmic trading systems

Build the infrastructure of your algorithmic trading system

A development process’s infrastructure can be understood as a step-by-step guide when working on a trading project. Every developer has a bit different approaches, but the skeleton of the process is usually the same. This article is an introduction to building your trading system from scratch. Each topic is crucial and contains steps you should not forget to do.

Depending on your previous experience, it can be a great collection of tools you already know with some new insights, or also everything can be new for you.

Infrastructure of Algo-Trading system

The development process can be divided into four sections. Each one is different and crucial for the final system. I used this process to build many trading strategies, from simple ones using simple price momentum on future markets to advanced systems trading hundreds of shares weekly using cutting-edge AI technology. With some simple ideas, all the process can finish within one day; however, developing an advanced system may take months/years to be fully functional. I have experience with both. Once you are in, it is a never-ending process of creating new strategies, upgrading existing ones, playing with new tools, and expanding your mind with thousands of exciting ideas.

1. The idea / concept / design

Every project, algorithm, or scientific theory, has an idea in the beginning. The concept in algorithmic trading begins with a thought which could be gained as inspiration from reading some blog or new scientific paper. Thanks to the exciting age in which we live you can get inspiration anywhere. I usually get trading inspiration from new white papers (mostly machine learning, respectively its applications), live observations from the market while programming another system, new data sources, and different blogs. Specifically one of my favorites is quantocracy. I visit it regularly for new ideas or interesting articles from different authors.

Testing the idea

The thought has to be developed and tested into the idea of the whole system. You have to grow it into a practical trading design. This growth is supported by systematic testing if the idea has a sense.

If there is an edge, and most importantly, why this idea should have an edge. This systematic testing is based on prices and statistically testing if the observed signal has significant results. These statistical tests compare the results when the signal occurred, vs. the set where signal didn’t occur, vs. the opposite signal has occurred (if possible to construct it). You can simply reject the idea during the testing or find out that the idea does not work as you first thought.

Looking for data

Next to systematic testing, you have to think about what kind of data you need to support the idea, respectively, to find a given signal. Since most of us are not hedge funds that pay hundreds of thousands of US dollars annually for high-quality data, we have to keep our feet on the ground and think if we can afford necessary data and sufficient quality.

Both, data and testing can stop your idea in the beginning or reshape your first thoughts. Every new data input has to be tested if it is a significant addition. When you successfully tested the idea and found good data providers, you can re-transform the concept into practical code. It is the final stage of the design process, and it can be a challenging part for a beginning programmer.

2. The system

Now when the concept is ready, tested, and you have data that can support it and help you find signals. The system is about transforming the concept into the code and, most importantly, creating the backtest on historical data.

Depending on your trading idea, this can be very quick or exhausting work for a few weeks or months. If your idea is a simple one, working with price data or just simple data input, you can use some existing packages or your previous code to create the backtests and move to the next part.

When your idea is more complex or creates thousands of signals (some other decision algorithm is also necessary), most of the time, none of the existing backtesting engines will help you, and you have to do this process according to the concept. In this case we need special data preparation according to the model, so in the diagram we used Machine Learning “Black Box”. In reality it is just another complex process, not necessarily black box.

When working with a lot of data, you need to think about speed (you don’t want to wait one month for some results obtained in two nights when you code it properly). Depending on needs you can create 3 types of backtests: very fast vector based (everything by using pandas DataFrames), event based (for-loop for each DateTime point), or special one according to your advanced model. By creating a new backtester for each different system will help you understand it better. In-sample sets would possibly give you insights that can better-up your idea or test some new inputs. It doesn’t only better-up the previous step but also gives you valuable time to think about live trading infrastructure.

The backtesting engine is different for complex designs and not applicable to live trading (mostly because of computational time). The most important result of this part is the out-of-sample backtest. Out-of-sample results were unseen when you developed the idea, created the model, tuned the model, upgraded your idea, etc.

3. Testing

This part can be very harmful to your ego and physical health when your strategy does not pass through it!

When the backtest on out-of-sample is ready. We have the final model (algorithm or price signal), and we are not touching it anymore. We should do a few tests to confirm that the final model is ready for trading.

The most crucial part is the randomization. You cannot read about this in any other sources. We have built it with our best know-how and experience. Each of our trading models has to go through this process before going live. The expected result of randomization is proof that the model is beating the randomness – we trade random strategies which have similar properties as the final system, similar logic but random, number of trades, average trade length, and so on. Randomization and robustness testing are explained in-depth in our another article.

When our system’s performance is not statistically beating the random strategies, that means if you trade random, you can have better results. So there is no edge. If you worked on the system for a few days, continue. If you worked a few months, give yourself one night of crying and then analyze what went wrong, think about making the changes that are not over-optimization, or just take a deep breath and let it go.

4. Trading

Finally, the strategy has an edge, and it was tested and is functional on historical data. Now it is time to create a trading infrastructure that consists of code that prepares signals daily/hourly/minute, an execution system, and a reporting system.

If you can use your older code, it is welcomed and helps with your work. It is good to code this online while doing simulation trading because you can see many problems that occur during the executions.

This part of the process is not only about coding. Some research is still necessary and consists of creating position management and testing different scenarios when trading. Intraday monitoring of your positions and making earlier exits depending on the market behavior, in other words, adaptive stop-losses and profit-gains. Testing different scenarios or simply saying cutting the edge of your strategy while watching the real market.

This kind of monitoring and in-depth execution development can give you valuable insights for later system upgrades. When the execution system is fully ready and functional on paper trading, create a reporting system that reports trades and returns daily. You will use that in live trading to compare if the model behaves the same as in the backtest.

When the simulation is making great, just turn it live with some small capital first. This part is essential because some errors can never be found by simulations. Enjoy the profits, watch it consistently, upgrade it for more markets, higher capitalization, and so on. Then start again with a new idea.

Closure

Don’t get me wrong; each section is full of exciting stuff and different approaches and sections are more interconnected than it looks. Working on your own backtester can give you excellent thoughts on upgrading the idea or on using additional data. The same when working on executions.

If you are just starting your trading journey and some parts look unexplained or difficult, I recommend reading our ebook about futures trading.

When you’re more experienced, stay tuned! We are working on new exciting articles. In a meantime you can look at our advanced ebook. Even when we showed the steps to create an independent trading system, the ebook’s insights will give you a much broader overview of the development process.

Search

Topics

Advanced (6)Algorithmic Trading (17)Basics (24)Futures (1)Investing (6)Libraries (1)Python (10)Statistics (1)Stock Market (9)Stock Pairs (2)Strategy Development (16)Time Series (2)Trading (8)volatility (2)