Three Backtesting Mistakes You Must Know as a Trader!

Have you ever spent ages creating a super backtested trading plan, only to see your profits disappear when you actually started trading?

Situations where your backtest results looked amazing, with almost every trade winning. But then real-life trading surprised you with unexpected price movements.

Or maybe you forgot to consider trading fees in your backtest and later found them eating into your profits.

Sound familiar?

See, these are common mistakes that even experienced traders make when backtesting.

But, By understanding these mistakes, you can use backtesting to improve your win-loss ratio in the real market.

So, In this blog post, we'll break down these backtesting blunders. We'll show you what can go wrong, and more importantly, how to avoid those mistakes.

So,

The three Most Common mistakes that traders generally commit are

1)    Look Ahead Bias

2)    Overfitting

3)    Ignoring Transaction Cost

Now, Let's have a look at each of these mistakes in detail

Mistake #1: Look Ahead Bias

You all know that Backtesting is a common technique used to evaluate trading strategies by applying them to historical data. Ideally, this should give you a realistic idea of how the strategy would have performed in the past.

We may also know how long in the past we should go to backtest the data

But,

What about the future?

See, Look-ahead bias occurs when the backtest uses information that wouldn't have been available at the time the trade decisions were being made. This can be information about future prices, but also other data points that weren't known at the specific time.

Let's take an example to understand this;

Consider you're developing a trading strategy for buying and selling a stock. The strategy you come up with is to:

• Buy the stock if the price increases by 5% in a single day.
• Sell the stock if the price decreases by 3% in a single day.

In an ideal backtest, the strategy would only consider the historical closing price data up to the point of the decision. This means:

• To decide whether to buy today, the strategy would only analyze the price data up to yesterday's close. It wouldn't peek ahead and see what tomorrow's price is.
• Similarly, to decide whether to sell today, the strategy would only consider the price data up to today's close. It wouldn't consider any price information from after the market closes.

Now,

Let’s say that the backtesting tool accidentally uses tomorrow's closing price (which the strategy wouldn't have access to in real time) to decide whether to buy today.

If tomorrow's price is higher than today's closing price by more than 5%, the strategy will incorrectly appear to signal a "buy" even if today's price movement wasn't necessarily indicative of a good buying opportunity.

Conversely, if tomorrow's price is lower than today's closing price by more than 3%, the strategy will be flagged to "sell" today, even though the stock price might rebound later in the day (information the strategy wouldn't have had access to).

Why is this a problem?

By using future information (tomorrow's price), the strategy seems much more effective than it would be in reality. This can lead to:

Misleading Results. The backtest paints an overly optimistic picture of the strategy's performance. You might become overly confident in the strategy's ability to predict future price movements, potentially leading to risky trades in real time.

The strategy might not work as well when applied to live trading because it relied on information that wouldn't be available in real time.

So,

How to Avoid Look-ahead Bias?

Well,

The fundamental rule is to Use Only Past Data. Your trading strategy should only rely on indicators and data that were available before the potential trade signal. Avoid using hindsight or future events (like economic reports or news) to adjust your backtesting parameters.

Professional traders often use Separate Data Sets for Development and Testing. They split their historical data into two sets:

One set is used to develop and refine their trading strategy.

The other unseen set is used solely for testing the strategy's performance. This helps prevent the strategy from being overly optimized to the specific data used for development.

Another Strategy is Walk-Forward Optimization. This technique involves testing your strategy in stages.

First, You backtest on a limited historical period.

Then you add another period of data and test again.

This process is repeated to simulate real-time trading where you're constantly encountering new market conditions.

Mistake # 2: OverFitting

Overfitting in backtesting is like fitting a custom suit that looks amazing on a mannequin but might burst at the seams when you actually wear it.

But,

What Exactly is Overfitting?

Overfitting occurs when a trading strategy is tweaked and optimized too much to fit the specific historical data used in the backtest. The problem occurs more in AI backtesting as you can change the parameters more frequently and easily. This can make the strategy appear very successful on past data, but it might not perform well when applied to new, unseen market conditions.

To understand the concept, let's take an example

Let's say you're backtesting a simple Relative Strength Index (RSI) mean reversion strategy for a stock. Here's the initial approach to the trading signals using backtesting:

• Buy Signal: You buy the stock when the RSI falls below 20 (indicating oversold territory).
• Sell Signal: You sell the stock when the RSI climbs above 60 (indicating overbought territory).

By this strategy, you may get the backtest results that look promising, showing good overall profit.

However, when you delve deeper,

You notice your biggest losses occurred in 2018 when you bought during a significant market downturn. The stock's RSI dipped well below 20 for an extended period.

Overfitting Attempt 1 - Adjusting Entry Point:

Believing a lower RSI buy threshold might capture better entry points, you test a revised strategy:

New Buy Signal: Buy the stock when the RSI falls below 15.

Sell Signal: Maintain the sell signal at an RSI above 60.

This revised strategy shows improved backtesting results, particularly for 2018. However, upon further analysis:

You discover that some winning trades in 2020 involved the stock price continuing to rise after you sold at RSI 60 (missing out on potential profits).

Overfitting Attempt 2 - Adjusting Exit Point:

Thinking a longer holding period might capture these extended runs, you test another revision:

Buy Signal: Maintain the buy signal at RSI below 15.

New Sell Signal: Sell the stock when the RSI climbs above 70.

This new iteration might show even better overall backtested performance. But there's a catch!

You've essentially "cherry-picked" parameters (changing buy and sell points based on specific years' performance) to fit the historical data. This doesn't guarantee future success.

Did you notice the Problem with Overfitting?

See,

If you keep tweaking the strategy's parameters (like entry and exit points, indicators) to squeeze out every bit of past performance, you might end up with a strategy that's "over-tailored" to that specific data set.

As a result,

The backtest results might not be realistic. The strategy might appear to perform flawlessly on past data, but it might struggle with new market conditions.

Why is Overfitting a problem?

Overfitting can lead to unrealistic expectations about the strategy's future performance. When applied to live trading, the overfitted strategy might not perform as well as expected, potentially leading to losses.

So,

How to Avoid Overfitting?

Overfitting can be avoided, But, to do this, you need to make sure,

Not to Overfit to a Small Data Set. The more historical data you use to backtest your strategy, the less likely it is that your strategy is simply capitalizing on random fluctuations ("noise") in the data, rather than identifying true market patterns. Aim for a larger data set to ensure your strategy can adapt to various market conditions.

Similar to avoiding look-ahead bias, experienced traders often Separate Data Sets for Development and Testing. Use one set to develop and refine your strategy. Use a completely separate set of unseen data to test the strategy's performance on unfamiliar market conditions. This helps ensure your strategy is more generalizable and can adapt to different scenarios.

We know, It's tempting to endlessly adjust your strategy's parameters to achieve perfect backtesting results. However, this can lead to overfitting. So, Choose a reasonable set of parameters based on market logic and trading principles. Focus on evaluating the overall performance of your strategy across the data set, rather than getting hung up on achieving a specific win rate or other singular metric.

And as with Look Ahead Bias,

Use Walk-Forward Optimization

This technique is particularly helpful for experienced traders. It involves testing your strategy in stages.

In the first stage, you Backtest on a limited historical period.

Then add another period of data and test again.

Repeat this process, progressively adding new data. This simulates real-world trading where you're constantly encountering new market conditions. By observing how your strategy performs on evolving data sets, you can gain a better understanding of its ability to adapt to changing market dynamics.

Mistake # 3: Neglecting Transaction costs

Transaction costs, which include commissions, fees, and spreads, eat into your profits on every trade. It doesn't matter whether you are doing manual backtest or Automated Backtesting, Backtests that neglect these costs will present a rosier picture of a strategy's performance than what you'd experience in real-time trading.

We can try to understand this with the help of an example,

Consider Avis Budget Group for this example. Avis Budget Group, trading under the ticker symbol CAR, is a publicly-traded company in the car rental industry.

Broker Fees/Commissions

Let's assume, you're backtesting a day trading strategy for CAR and the backtest results look promising. But,

For each trade you make incur a commission fee. While the specific amount varies depending on your broker, let's assume a hypothetical commission of $0.05 per share (common for many online brokers).

So,

For a backtested strategy that generates a profit of $0.20 per share, a $0.05 commission fee eats away at 25% of that profit. This can significantly impact your overall returns, especially with frequent day trading.

If your backtest shows your strategy generates an average profit of $0.20 per share on 20-day trades for CAR, this can translate to a total backtest profit of $4 (20 trades * $0.20 profit/trade).

However, if you neglect the $0.05 commission fee per trade, the backtest doesn't account for a total cost of $1 (20 trades * $0.05 commission/trade).

Taking commissions into account, your actual profit from those 20 trades would be $3 (total backtested profit - commission cost = $4 - $1).

Higher Spreads

But Brokerage Fees and Commissions are not the only costs that can erode your profits in real-life

Compared to mega-cap stocks, CAR might also have a lower average daily trading volume. This can make it slightly less liquid than very actively traded stocks.

Moreover, there can be a noticeable difference between the bid and ask price, impacting potential profits on trades.

Now, Let's see How Backtesting on this stock goes.

Let's say your backtest for CAR is showing a swing trading strategy that generates a profit of $1 per share. However, As CAR is a slightly illiquid stock, the backtest might make some unrealistic assumptions:

One is the Instantaneous Fills. The backtest might assume you can enter and exit trades at the exact price points identified by the strategy.

The second is the Negligible Costs. The backtest might not factor in any transaction costs associated with trading CAR.

Now,

In real-world trading with CAR, these assumptions can lead to a different outcome:

Due to lower liquidity, it might be challenging to get filled at the exact price points identified by the strategy. You might have to;

Either, wait for another trader willing to buy or sell at your desired price, potentially missing the opportunity.

OR, Cross the Spread. Buy at the ask price (higher than your target) or sell at the bid price (lower than your target), reducing your profit or even turning it into a loss.

Did you notice the problem?

 If the average bid-ask spread for CAR is $0.20, and you have to cross the spread on every trade, your $1 profit shrinks to $0.80.

And this is not the only cost!

Remember, transaction costs like commissions also eat into profits. Factoring in a $0.05 commission per share further reduces your profit to $0.75.

So,

Why Ignoring Transaction Costs can be a problem?

Backtests without transaction costs might make a marginally profitable strategy appear more attractive. The strategy might seem to generate small but consistent profits. However, when you factor in transaction costs, those small profits could vanish, or even turn into losses.

How to Address this Problem?

For the Transaction cost,  

First of all, you should understand the commission structure of your chosen broker, including any per-share or tiered commission fees.

Next, many backtesting platforms allow you to specify transaction costs. Include the estimated commission fees to get a more realistic picture of your strategy's performance. When evaluating a strategy, prioritize its profitability after accounting for all transaction costs, not just the gross profits shown in the backtest.

To cater to the Problem of Large Spreads and Illiquid Stocks,

You can look for historical data on the average bid-ask spread for CAR and incorporate it into your backtesting platform.

Similarly,

You can also use backtesting tools that allow simulating limit orders. This lets you test how your strategy performs when you set buy and sell orders at specific price points within the spread.

The Problem of lower liquidity can also be addressed,

By aiming for trades that capture larger price swings to compensate for potential spread costs. You can also focus on longer-term trades to reduce the impact of short-term bid-ask spreads.

In Conclusion,

Common mistakes like look-ahead bias and overfitting can paint a misleading picture of your strategy's true potential. By incorporating the techniques discussed here, you can ensure your backtesting results are grounded in reality and provide valuable insights to guide your trading decisions. This will help you to

Trade Smarter!