How Obsidian avoids bias in its Automated Trading System
Developing, testing and implementing automated Trading Strategies is a complex undertaking which draws on multiple disciplines. Its ultimate expression is the Holy Grail of trading-a multi-layered system which succeeds in all market conditions.
A (non-exhaustive) list of the requirements to build such a system include
Having developed the system there are essentially four stages of testing to go from coding to actual trading implementation:
Back-testing is both critical and forms the core part of a comprehensive testing framework. It covers the first two elements above and uses historical market data. The aim is to check whether the trading strategies will perform well with real capital at stake. Mastering back-testing can make the difference between having Trading Strategies that look good theoretically and those that deliver a meaningful statistical bias i.e. make money more often than they lose.
At Obsidian we use a time series of closing price data for each of the assets in any of our portfolios. Typically, though not exclusively, we use two-thirds of this data, as In-Sample data, to determine the parameter values we will use within the Strategies. These are then tested against the remaining third of the data, the Out-of-Sample data, in order to select the most appropriate setup for Paper and Live Trading.
Overfitting and Underfitting:
Overfitting is a very common problem whilst creating automatic Trading Strategies. When trying to find the best set of values for the parameters one way to assess potential candidates is to calculate the error between the values calculated using the proposed parameter values and the actual market values. Logically the parameter values are chosen to minimise this error. However, the danger here is that is in trying to map the underlying Signal the regime picks up the Noise as well. The Noise being elements of the signal which have nothing to do with the underlying process and so are either random or unrelated. Given that it is very unlikely that this noise will be repeated in the same way in the future means that the chosen parameter values will probably mis-forecast the future and lead to large tracking errors and ultimately losses. Therefore, any set of parameter values which too accurately match the In-Sample data will often perform terribly at the Out-of-Sample test and even worse in Live Trading due to overfitting of the data.
At Obsidian, we avoid this by firstly keeping the In-Sample and Out-of-Sample data separate i.e. ensuring that this is no overlap. Secondly, we accept that there will and can be a tracking error in the In-Sample test and a similar, though typically slightly higher, error in the Out-of-Sample Test. At the heart of our process we aim to capture the key drivers through our proprietary parameters and calibration process, rather than the ‘perfect’ formula, which may anyway be impossible to describe mathematically. In addition, as would be expected with Machine Learning, our system has built up a repository of knowledge through tens if not hundreds of thousands of previous iterations so is well equipped to quickly establish the optimal parameter values.
Underfitting is the opposite of overfitting and though less common can have equally undesirable effects on future performance. As its name suggests it occurs when a parameter set inadequately recreates an In-Sample data set. It is often a result of the use of an insufficient number of parameters, inadequate data set or a flaw in the fitting method. At Obsidian we avoid this by consistently using a well-defined process which optimally captures the important elements of the time-series and no more.
Survivorship Bias:
Survivorship Bias is a common problem when modelling asset performance particularly with stocks. If they deliver a large alpha; during a period of out-performance against their peers and the market, then most algorithms will weight them accordingly on the assumption that this is somehow inherent to them and will continue. However, this excess return could be because of some unique competitive advantage or the newness of a sector so will not necessarily persist indefinitely. The risk here is that future trading may over-commit to this stock and when it loses its edge the portfolio suffers alongside. An excellent example is Enron, see below:
Many conventional systems in the early 2000s would have included Enron in any portfolio. However, its implosion in 2001 would have wiped out not just the profit but also the initial capital invested.
Survivorship Bias works both ways inasmuch as an underperforming stock would be largely ignored by most modelling routines as its inclusion was detrimental. As a result, if it then proceeded to outperform due to restructuring, innovation or market shifts the portfolio would miss out as it was underrepresented. This form of bias is becoming increasingly common with the advent of advanced machine learning tools that allow the oversimplification of back testing.
Obsidian addresses Survivorship Bias by automatically introducing bias hurdles at the core strategy creation level and by making it a centre pillar of our technology. We therefore do not avoid it but on the contrary, embrace it. Because of the proprietary nature of the modelling routines the system works objectively and ‘does the best it can’ with what it is presented with. In other words, an underperforming stock is not wholly discarded, it is represented but through the Allocator and Rotator action it may have little, or no capital allocated. However, because of the dynamic nature of these sub-systems, if the stock starts to recover then it can be promoted into the active portfolio and then have capital allocated to it. Similarly, any stock that has had an excellent recent performance will naturally be present in a live portfolio at inception. If it starts to under-perform then again through the Allocator and Rotator its capital allocation will fall and eventually it could be rotated out. The fact that Obsidian does not delete any strategies but simply moves them into the background, means that stocks can reintroduced later if necessary. There are no short cuts to avoiding this pitfall but there are ways to deal with it profitably.
Data Snooping:
Data Snooping in a common problem when modelling financial data and essentially involves finding patterns in data that do not really exist or at least are not actionable. They are due to randomness and not some underlying, this far, hidden connections. Relying on these relationships can be dangerous once live trading commences as they do not persist beyond the original data set.
This can occur when the analysis relies on a large data set. On the one hand this is necessary as any results have less value if extracted from a short time series. But conversely the larger the data set the more likely pseudo patterns will be found which are due to randomness. For example, a typical portfolio of say 50 stocks and ten years’ worth of closing price data will involve 125,000 individual data points so it is not surprising if many patterns can be established which seem plausible. This happens when all possible combinations of relationships are tested, and further when little thought is given to whether they have any tangible meaning in the real world. Ultimately eliminating Data Snooping completely is impossible as for instance we do not have a control closing price time series for each stock, which would be necessary as part of a proper scientific analysis. Obsidian minimises the impact of Data Snooping by limiting the search criteria to a historically trusted set, using them consistently across analyses and assessing the results critically with market and trading knowledge.
Look-ahead Bias:
Related to Survivorship Bias is Look-ahead Bias which occurs when known future events are effectively incorporated into the past data used to establish Trading Strategies. A good example would be including Netflix in a potential Nasdaq portfolio prior to its inclusion in late 2010 whilst excluding Logitech at the same time. More generally only analysing bull market periods can lead to disastrous results when faced with the next sell-off as the Trading Strategies are biased to upside moves and inherently insensitive to downtrends. This is commonly referred to as having a Time-Period bias. Obsidian overcomes these problems by being careful with selection criteria and using a proprietary framework that ensures statistically true positive results which is independent of market regimes.
Human Bias:
One often overlooked failing is the introduction of Human Bias, less so at the development and testing stage; although this can be fatal, but more at the implementation and live trading stage. It occurs when quant-traders are essentially unfaithful to their own systems. Following news or major moves in the market they decide to override the system’s trading decisions and manually intervene at their own discretion. Whilst they may be fortunate and make a correct call in the short term it is probable that they will not be able to replicate this day after day leading to the actual performance below that expected. Humans are emotional so any biases due to personal issues or environmental conditions will translate into trading decisions. Making money with this additional noise will be more due to luck than judgement. Machines follow objective rules and have no emotional baggage so if the instructions are robust, they will not make an error or overreact. Obsidian avoids this problem altogether by ‘Locking’ Portfolios once they are created and shielding all parts of the decision-making process and trading from human intervention.
Obsidian’s Trading Systems is a successful amalgamation of human judgement and instinct combined with systematic and objective machine learning which results in a dynamic, evolving environment which ultimately produces superior returns and improved risk metrics, and is applicable across markets, asset classes and most importantly, market conditions.