Why your backtest and live trading disagree, and why they shouldn't

The first time I ran a new strategy live alongside its simulation, the two disagreed within two days. The simulation had bought and the live system had not. The cause turned out to be a small error in how the simulation looked up its data, which let it rank instruments using a figure it could not yet have known. In effect it was peeking one step into the future, and the backtest had looked excellent for exactly that reason.

That is what this paper is about. When a simulation and live trading disagree, one of them is wrong, and the useful thing is to find out which.

To be able to answer that at all, three things have to hold. Running the same simulation twice must give the same answer. The simulation and the live system, given the same market data and starting from the same positions and cash, must make the same decisions. And the price history the simulation uses must not change underneath it without anyone knowing. When all three hold, a process is what engineers call deterministic: the same inputs always produce the same result. When any of them fails, a change in results could have come from the market, the code or the data, and there is no way of telling which.

None of this tells you whether a strategy is any good. Whether it makes money after costs, and keeps doing so, is a separate question. What it tells you is whether the numbers in front of you can be trusted, and in my experience they often cannot, for reasons that are ordinary and easy to miss. These are the seven I have run into.

Seven things that quietly break the agreement

Each of these has happened on my own platform. None of them announced itself; each one simply made the simulation and the live system disagree, or made the same simulation give a different answer from one day to the next.

1. The order of a sum

Computers store most decimal numbers very slightly inaccurately, and one surprising result is that the order in which you add numbers can change the answer. Adding 0.1 and 0.2 and then 0.3 gives 0.6000000000000001; adding the same three numbers the other way round gives exactly 0.6. The difference is tiny, but it is real.

My portfolio simulation runs many strategies at the same time and adds up their profits each day. The strategies finish in whatever order the computer happens to run them, so the daily totals were being added in a slightly different order on every run. Two runs of exactly the same code on exactly the same data therefore gave annual returns that differed in the second decimal place.

The clue was that every position was identical and only the totals moved; if the strategies had really behaved differently, the positions would have differed too. The fix was to add the results up in the same fixed order every time. An existing test had missed it, because on a quiet computer the strategies happened to finish in the same order on each run, so the test now checks the order itself.

2. History that changes

Data vendors sometimes revise past prices. I once saw a vendor supply two different versions of the same daily price history eleven hours apart, differing on more than a thousand days. A nightly job that downloaded the full history simply took whichever version it was given, and one of my environments quietly started working from different history than another.

A platform can do the same thing to itself. Futures contracts expire, so a long price history has to be stitched together from many contracts. A common way of doing that recalculates the whole history every time one contract expires and the next takes over, which means every past price changes slightly. Run the same simulation afterwards and it can give a different answer, because the rounding can tip a close decision the other way.

The rule I follow is that nothing which fed a recorded decision is ever overwritten without a record. Corrections do happen, and they are legitimate, but each one is logged and every simulation knows which version of the data it used. My futures histories are built so that new contracts are only ever added to the end, and the past is never recalculated.

3. A closing price you cannot know yet

Many backtesting engines let a strategy place what is usually called a “this bar on close” order; in MultiCharts, for example, it is a MarketThisBar order. The strategy evaluates its rules using the bar’s closing price and is then filled at that same close. In a simulation that is easy, because the closing price is already in the data. In real trading it is impossible. An order to trade at the close has to be sent before the market closes, when the closing price does not yet exist, so the decision has to be made on a price taken a few minutes earlier.

If the simulation decides on the real close while the live system decides on the earlier price, the two will sometimes reach different decisions from what looks like the same day of data. My fix is to record one price for every instrument shortly before the close. The live system decides on that recorded price, and the simulation uses the same recorded price wherever one exists.

4. A stale copy of the data

A simulation run early one morning gave a different result from the same simulation run four hours later, against the same database. The newest day’s prices had arrived in the database eleven minutes before the first run, but the simulation did not read the database directly. It read through a separate service that keeps a copy of each price history in memory to save time, and that copy had not yet been refreshed. The first run was simply working from yesterday’s data.

I could prove it with arithmetic: the size of the difference matched, to six significant figures, what swapping one day’s figure for the other would produce. In that case the live system was right and the simulation was wrong, which is a reminder that the backtest is not automatically the side to trust. Anything that keeps a copy of data between runs can serve an old version, and a result that depends on it is no longer a product of the data and the code alone.

5. Settings nobody stored

Every strategy has settings, such as how far back to look or how large a move to wait for. When a setting is not saved with the strategy, the software fills it in with a default value. I once changed some defaults to match the values used in live trading. It seemed harmless, but several stored tests had saved only some of their settings and relied on the defaults for the rest. They all silently picked up the new values, and their results changed.

A default value is not a promise that the value will stay the same. Every setting a strategy uses is now stored with it, so that changing the software can never change a result that nobody meant to change.

6. Rounding at the edge of a tick

Exchanges only accept prices in fixed steps, called ticks; for many shares the tick is one cent. Rounding a price to the nearest valid tick looks like a one-line calculation, but the same floating-point rounding described above means a price that is already valid can be pushed down by a whole tick, so that 20.04 becomes 20.03. The live order then goes out one cent away from the price the strategy chose, and the simulation and the live system disagree for no reason the strategy can see.

We found this by testing the calculation directly, before it had affected a real order. Prices are now rounded with exact decimal arithmetic, and the direction of rounding depends on the kind of order, so a buy limit can never be rounded above its limit and a sell limit never below it.

7. “Unknown” shown as “fine”

Over two days I found five problems in five unrelated parts of the platform that all turned out to be the same mistake. A health check with no data reported a fault that was not happening. A comparison that could not run reported zero differences. A list that had never been loaded reported that nothing needed attention. A screen whose request had failed showed a confident zero. A software build that had not actually rebuilt anything reported success.

In each case the system did not know the answer and said “all clear” instead. Zero, green and OK are perfectly normal values, so nothing looked wrong.

A check that did not run has not passed.

The platform now treats “unknown” as a separate state, and shows it as such all the way to the screen.

Finding out: run them side by side

The best way to catch these problems is not to study the backtest harder. It is to run a new strategy live on a paper account, which trades real market prices with simulated money, alongside its simulation, and to insist that every signal, position and profit figure matches. The ranking error at the start of this paper was found that way within two days.

Another case came to light in the same way. Several of my strategies trade different markets but share one pool of margin, the money a broker requires to be set aside against open positions. The simulation was calculating that margin as if all of those markets were closed at the same moment. That moment never actually happens: bitcoin futures now close for only a few minutes a day, and other markets keep their own hours. Nothing in the backtest gave any sign of it, but comparing it with live trading did. Fixing it meant building a more deterministic way of managing margin across the portfolio, one based on moments that actually occur in each market and on figures that the simulation and the live system share, so that the two always reach the same answer.

Most differences can be engineered away

The encouraging part is that almost every difference I have found between simulation and live trading could be solved with engineering. It mostly takes a different mindset: instead of asking what the simulator allows, ask what a live order could actually have done, and build the strategy so that the simulator is held to the same standard.

Two examples show what that looks like in practice.

• Work out the price in advance. Rather than waiting for a bar to close and then deciding, a strategy can calculate ahead of time the price at which its signal would trigger, and place that price as a limit or stop order for the next bar. The live system and the simulation then work from the same price, set before the market moves, and neither depends on a number it could not have known.
• Use finer data. A daily bar records the open, the high, the low and the close, but not the order in which they happened, so a simulator has to guess whether a limit or a stop would have been reached first. Using intraday data to replay what actually happened during the day removes most of that guesswork, and the simulated fills come much closer to the real ones.

Neither idea is exotic. Both come from treating the simulator as something that has to earn its results, rather than something to be trusted by default.

Why a difference means stop

It is tempting to wave away a small difference between simulation and live trading as noise. The trouble is that you do not know what it represents, and a small difference can be the first visible sign of a much larger fault. The only differences I accept are the ones I can explain and put a number on in advance, such as slippage, where an order fills at a slightly worse price than planned. Anything else is treated as a fault.

A single extra trade shows why. In a portfolio where capital is shared out among strategies according to how each is doing, one extra position changes that strategy’s results, which changes how much capital every other strategy receives, which changes the size of every trade that follows. The extra position will also be closed at some point, producing a trade the simulation never makes.

One extra trade

live has it, the simulation does not

→

Strategy results change

its profit and equity differ

→

Capital shifts

every other strategy gets a different share

→

Every later trade

is sized differently

So when the two sides disagree, the platform stops, and agreement is restored before anything else happens. If the positions, the cash and everything else that affects decisions already agree, the simulation can be used to reset the strategy’s internal records without touching the account. If they do not, the broker’s confirmed record of what actually filled is the fact everything else must match, and the simulation is replayed from those fills. Only then does the work of finding the cause begin. Patching the one row that differs and moving on would cause the least disturbance, but it is the wrong goal.

Why this isn’t theory

Every example in this paper happened on the trading platform I built and run, and each one now has a control behind it. Every night the platform compares the simulation, the live systems and the risk records for every strategy, and stops if they disagree. The price used for decisions at the close is recorded once and shared by both sides. Futures price histories are only ever added to, never recalculated. Every setting a strategy uses is stored with it. None of these controls is clever. Each exists because I found the problem the hard way and made sure it could not happen again.

I came to this after a career on both sides of the desk, as a quant, a portfolio manager and a trading desk developer, and later building, selling and operating banking software that institutions trusted with their books. That is why I hold a trading platform to the same standard as a bank’s systems: if it cannot explain a difference, its numbers cannot be trusted.

A process that reproduces exactly can still run a poor strategy; it simply runs it reliably. What it gives you is the knowledge that when results change, it was the market and not the machinery, and that the amount of risk you take is set by rules you chose.