A backtest is a claim about what you knew at 07:02

Every backtest on news is a claim about a clock. When a row in your research set says an item published at 07:02 and carried an actionability of 78, you're asserting two separate things: that the item existed at 07:02, and that 78 is what a system would have said at 07:02 rather than what it thinks about that text now. Desks check the first. Almost nobody checks the second.

I spent four years at a stat-arb shop watching good backtests die in production, and the autopsy was repetitive enough to be dull. The signal was usually fine. The history it had been fit on had been touched after the fact.

A number that moves after the fact isn't history

Silent revision is the failure I'd put first on the diligence list, ahead of coverage, ahead of latency, ahead of price. It works like this. A vendor improves its scoring, backfills the improvement across the archive, and ships it. Nothing errors. Your files have the same shape and the same row count. Every judgement in your sample was now made with information that did not exist on the morning it was timestamped, and your Sharpe goes up, because the archive has quietly learned what happened next.

Price data revises too, but price revisions are rare, announced, and mostly corporate actions. A judgement about a headline is a model output, and model outputs are cheap to recompute. That asymmetry is why I'll trust a tick database I have never audited and won't trust a scored news archive I haven't diffed.

So diff it. Pull the same window with the same since and until, three weeks apart, join on id, and compare the scores field by field. Anything that moved is a row you cannot fit on. That test takes an afternoon and it belongs before anyone writes a strategy, not after somebody senior asks why live is running at half of sim.

Our position is the boring one. An item is scored when it crosses, and the verdict on the row is the verdict from that moment. Ask for the same window in March and again in November and the numbers come back the same.

One trap you can still walk into on your own: handing archived text back to POST /v1/score gives you today's read, and novelty depends on what was already public at the time. Fine for calibrating weights on a stratified sample. Not a way to reconstruct a series.

Five numbers you can argue with beat one you can't

Actionability comes out 0 to 100, and the gate sits at 60. Underneath it are five sub-scores, each rated on its own merits and then weighted: novelty at 0.30, materiality 0.25, surprise 0.20, specificity 0.15, directness 0.10.

The reason to care isn't transparency as a virtue. It's that an opaque score is untestable. When a single number underperforms on a subset of your universe, your only lever is to stop using it. When a decomposed one underperforms, you can ask which component is wrong, and the answers are different problems. Specificity misfiring means the feature is noisy in that corner. Novelty misfiring means the model's sense of what was already public is thin there. The first is a re-weighting you can do in an afternoon, on your own view of what materiality is worth at your holding period. The second isn't, and knowing which you're looking at beats a slightly better composite.

One boundary worth stating plainly, because it changes what you should fit. The 60 gate is a single global number. It isn't set per sector or per market cap, and small caps carry thinner disclosure and fewer prior events to judge novelty against, so a 58 on a small name and a 58 on a mega-cap are not the same object. We say so in the docs. If your universe is skewed small, treat the gate as a starting point and set your own.

Same event, two horizons, two different trades

The sentiment head emits short_direction and long_direction, both signed integers from −5 to +5, and they're permitted to disagree. A buyback authorisation is the canonical case: something is being bid tomorrow, and whether that capital allocation is good for the equity in two quarters is a separate question with a separate answer. A going-concern paragraph runs the other way, mild in the short direction because it's buried deep in a filing and severe in the long one.

A vendor that publishes one sentiment number has made a holding-period assumption for you and not said which. Run one book intraday and another at multi-week and that assumption is wrong for at least one of them, and the missing horizon isn't recoverable from the number you got.

Two ways a news backtest lies to you

Survivorship first, because it's easy to fix and easy to skip. Assemble your research universe from today's index membership and you've selected on companies that are still here. In a news backtest that bites harder than in a price backtest, because the largest-scoring events are disproportionately the ones that preceded a company ceasing to be a member. Join against a point-in-time universe you control. Every item carries its own symbols with the exchange attached, so the join is yours to make rather than ours to assume.

Look-ahead is subtler and arrives through the extraction rather than the model. Walk a live archive with offset pagination while new rows land at the head of the sort order and your pages shift under you, leaving a sample with holes and duplicates that nobody notices until the results look interesting. /v1/feeds pages on an opaque cursor, and next_cursor describes a position rather than a row count, so appending at the head can't disturb a page you've already walked. Pin until at the start of a long run and reconcile on ids.

Pick the tier by the shape of the study rather than the size of the desk. A gate calibration on a single sector needs weeks of history and finishes on Pro. Anything with a seasonal term in it needs the 8,760 hours, and at 500 rows a request a hundred thousand items is two hundred calls.

Sign the position on direction, size it on conviction

Conviction is a separate number, 0 to 100, and this is the separation I'd defend hardest. Direction picks the side. Conviction goes into the sizing function, usually as a multiplier on whatever the risk model already permitted. A low-conviction +4 should be taken at a fraction of the notional of a high-conviction +2, and that's a sentence you cannot even write if the two are baked into one column.

Desks that collapse them end up sizing on the strength of the language, which is a measure of how a release was written and not of how much the engine will stand behind it. That distinction survives contact with production. Most things don't.

All of it reduces to one question that a compliance officer and a researcher ask in the same words and rarely realise they share: what did your system know at 07:02 that morning, and can you show me? A desk asks because a backtest that can't answer is fiction. A regulator asks because a trade that can't answer is exposure. Build for the first and the second comes free, which is the only argument I've found that gets a research team and a control function to agree on a vendor before lunch.