My backtest showed a 92.8% win rate, and the strategy didn't work: how an insider detector introduced look-ahead bias

Translated from the Russian original with AI and reviewed by the author.

Last spring I had a hobby project around Polymarket, a prediction exchange. You trade contracts there on the outcome of an event: “will X happen by such-and-such date”. A contract costs between 0 and 1 dollar. When the outcome is determined (the market has resolved), the contract pays 1 if the event happened and 0 if it didn’t. So the price is roughly the probability the market assigns to the event: a contract at 0.10 is something the market gives a 10% chance.

Say you buy a contract at 0.10. If the event happens, you get 1 and make 0.90, which is 900% on what you put in. If it doesn’t, you lose 0.10. The share of winning trades is called the win rate (WR), and ROI shows what percentage you made or lost on everything you put in.

The insider detector

It all started with a question that had nothing to do with trading: can you find people who knew the outcome in advance just from public trades? The obvious tell: someone buys a cheap, unlikely outcome for a large sum, and that outcome comes true. So I wrote a scorer, a script that gives a wallet and market pair a suspiciousness score. Here is how it selected trades for scoring:

SELECT t.maker, t.condition_id, t.value_usd, t.timestamp
FROM trades_p t
JOIN markets m ON m.condition_id = t.condition_id
WHERE t.side = 'BUY'
  AND t.price < 0.25                      -- cheap outcome, less than a 25% chance
  AND m.resolved_outcome = t.outcome      -- and that outcome came true

For an investigation this is the right query: we look at the past, the result is known, and we search for traders whose unlikely bets paid off.

Here is what such a case looks like on a real market. On the evening of January 2, the contract “Maduro out by January 31” cost 6.5 cents, and by the morning of January 3 the outcome had happened and the price went to 1.

Say someone bought “yes” for a large amount on the evening of January 2. The detector looks at that trade later, when the outcome is already known, and honestly flags it as suspicious. The live system on the evening of January 2 sees only a purchase at 6.5 cents: whether the outcome will come true is unknown to it. The backtest replays the same evening of January 2, but takes data from a database where the outcome is already recorded. If it relies on that anywhere, it sees more than the live system would.

The strategy and the first leak

Then I wanted to turn the detector into a strategy: if the scorer flags a wallet, buy the same side. In the first backtest, the entry was counted at the moment of the wallet’s first cheap purchase and at its price. That is already an idealization: the scorer notices a trade only after it has happened. After that, profit is taken in parts when the price reaches 2x, 3x and 5x, a stop-loss closes the position if the price falls, and the remainder is held until the market resolves. The trades for the backtest were selected by the same scorer.

The problem is in the last line of the query. The scorer only picks trades that have already won. In a backtest over history the resolved_outcome field is filled in, and the strategy “copy the suspicious ones” really meant “buy whatever won”. In live trading the outcome is unknown at the time of the trade, and there is no such hint. In machine learning this is called target leakage: the target variable leaked into the features. Here it leaked straight into the WHERE.

Simplified, it looks like this. 100,000 purchases of cheap contracts priced from 0.02 to 0.25, as in the query, and no inside information: each trade wins with a probability exactly equal to its price. No weights, stop-losses or partial profit-taking, only selection:

import random

random.seed(1)
trades = []
for _ in range(100_000):
    price = random.uniform(0.02, 0.25)
    trades.append({"price": price, "won": random.random() < price})

def report(name, picked):
    staked = sum(t["price"] for t in picked)              # we pay the contract price
    returned = sum(1.0 for t in picked if t["won"])       # a win pays 1
    wins = sum(t["won"] for t in picked) / len(picked)
    print(f"{name:12} trades {len(picked):6}  wins {wins:6.1%}  "
          f"ROI {returned / staked - 1:+8.1%}")

# honest: the outcome is unknown at trade time, so we copy every such buy
report("honest", trades)
# hindsight detector: a "suspicious" trade = cheap AND winning
report("leaky", [t for t in trades if t["won"]])

honest       trades 100000  wins  13.6%  ROI    +0.3%
leaky        trades  13551  wins 100.0%  ROI  +493.9%

The honest version comes out around zero, as it should with fair prices. The leaky one gives 100%, because the selection simply lists the winning trades.

The check found only half

I noticed the leak in the scorer and checked it: I built a mode where the scorer sees only what is known at the time of the trade. That is the entry price, the bet size, how fresh the wallet is, how many markets it trades on, and how old it is on the blockchain. I removed both resolved_outcome and features like “how much the wallet has already won on cheap outcomes”. The run gave a 90.8% win rate on 5,646 positions, and 92.8% on 9,589 at the working threshold, and I wrote “edge is real” in the plan.

The second leak was not in the scorer but in the prices. The backtest looked at hourly candles, records of what the contract price was each hour: high and low. They were stored not by the “yes” and “no” outcomes, but by the “winning” and “losing” sides. The replay simulation took the winning side’s price for a position even when the copied side had lost.

The winning side’s price almost always rises toward the end, so the partial profit-taking fired for almost every position. Losses remained only where the stop-loss triggered earlier or the price never reached the first target. That is where the 92.8% comes from.

Third, a smaller one: the entry price was taken as the best of all the wallet’s purchases on that market, including future ones.

By then this scorer was already running in paper trading: positions opened on live signals, but without money. The first version closed 635 positions at a loss of 532 paper dollars. The backtest promised 92.8%, and paper trading was losing.

The next day I wrote a new backtester. Candles in it are stored by outcome, entry happens at the price at the moment of the signal, with execution delay, the scorer sees only the features as of the trade, and the report has a built-in leak check. Copying purchases by fresh wallets under 10 cents gave a 5.5% win rate in it. It is not the same sample as in the first run. The average price and the number of trades of this run were not kept in the project records, but at prices under 10 cents the fair win rate is no higher than 10%, and 5.5% fits within that bound: the strategy has no edge. Copying large bets on cheap outcomes is a losing move. This test says nothing about whether there were insiders among them.

Three more ways to lie

The new backtester needed fixing too, this time on other strategies:

1. Unclosed positions. One strategy took profit in parts. If the market didn’t resolve before the end of the backtest window, the position was counted at the profit already taken, and the remaining shares were ignored entirely. Almost every such position looked like a winner. After the fix, the remaining shares are counted as lost. This is a conservative estimate: the market hadn’t resolved by the end of the window, and the backtest doesn’t know how it will end. With it, a 98.1% win rate and +287% ROI turned into 62.9% and −9.9%.
2. Entry time. In the backtest the news strategy entered at the moment the news was published, while the live system opened a position once the news was processed. Only 8% of the news was processed within an hour, and 12% only after the market had already resolved. The backtest bought at prices the live system never had.
3. Date filter. The condition COALESCE(end_date, updated_at) <= today dropped markets whose planned end date was in the future, even though they had already resolved early. It hit the long-running events the news strategy traded on: after the fix it had 103 trades instead of 56, and ROI went negative, −5.9%.

A high WR doesn’t mean profit

The reverse problem: one of the strategies bought almost certain outcomes at 0.90 to 0.96. On resolved markets it won 179 trades out of 195, that is 91.8%, and still lost about 2%. Back to the trade from the beginning, only at 0.90. You win, you make 0.10. You lose, you lose 0.90. Without a fee you break even at 9 wins out of 10, and at its prices, with the fee included, the threshold was about 93.5%.

What I took away

1. Code that looks for patterns in the past can’t be moved into forecasting without checking. The future leaks in not only through features, but also through which rows get selected and through how the data itself is built: candles of the “winning side” already know the outcome.
2. A backtest has to see the world the way the live system sees it: with delays, fees and positions that haven’t closed yet.
3. Win rate tells you nothing without the size of wins and losses; look at ROI.

None of the strategies in this post turned out profitable on an honest backtest, so don’t copy them.

The draft was prepared with AI from the author’s notes and code; the figures were checked against the project repository, and the example on synthetic data was checked by running it.