Where data comes from, by market
Lesson 4 · about 10 min
A backtest is only as good as the bars it runs on, and bars are not a fact of nature. They are built by someone, from some feed, with some rules about what counts as a trade. Before you trust a data set, you should know who made it, what it includes, and what it quietly leaves out.
Stocks
US stock data comes ultimately from the consolidated tape, which combines trades from every exchange and off-exchange venue. Free daily data (from broker platforms, charting sites, or public APIs) is usually derived from this tape and is fine for daily-bar tests on liquid names. The things to check:
- Adjustment. Is the series adjusted for splits and dividends? The next lesson covers why it matters.
- Delisted names. Does the data set include companies that no longer exist? Most free sources do not.
- Session. Do the bars include pre-market and after-hours trades, or only the regular 09:30 to 16:00 Eastern session? Intraday data from different vendors will differ here.
- Volume. Consolidated volume or a single exchange's volume? The numbers can differ by a factor of three or more.
Intraday stock data with reliable timestamps generally costs money. Broker feeds are often adequate for a first pass; dedicated vendors are better for anything under 5-minute bars.
Futures
Futures data comes from the exchange (CME, ICE, Eurex and so on), and each contract month is a separate instrument. To test a multi-year strategy you either stitch the months together into a "continuous" series, with the roll choices that Lesson 3 explains, or you test month by month and handle rolls explicitly.
Two session issues matter. Most futures trade nearly 23 hours a day, so a "daily bar" can mean the full session or only the pit hours, and the two give different opens, highs and lows. Settlement prices are set by the exchange and can differ from the last trade. Decide which one your rules use and be consistent.
Forex
There is no central forex exchange and no consolidated tape. Every broker or liquidity aggregator shows its own prices, so "EUR/USD 1-hour bars" from two brokers will not match, and neither is wrong. The differences are small for majors during liquid hours and large during rollover, holidays and news.
| Question | Why it matters |
|---|---|
| Bid, ask or mid? | Most retail feeds are bid; a long entry actually fills at the ask |
| Whose spread? | Spread varies by broker and time of day; it is a cost, not a fixed number |
| What time zone is "daily"? | A daily bar closing at 17:00 New York contains different trades from one closing at 00:00 UTC |
| Weekend gaps handled how? | Some feeds fabricate Sunday bars; some skip them |
Forex data is best treated as an approximation of the market, and tests should be robust to a pip or two of difference either way.
Crypto
Crypto is fragmented across exchanges, each with its own price. Data from one exchange describes that exchange, not "bitcoin". A strategy tested on one exchange's spot bars will behave differently on another exchange's perpetual futures. The key checks:
- Spot or perpetual? Perps trade at a basis to spot and carry funding (Module 4).
- Which exchange, and is it still operating? Exchange failures have taken data histories with them.
- Wash trading. Reported volume on some venues is unreliable; use volume from exchanges with a reputation to protect.
- Trading hours. 24/7, so a "daily bar" needs a stated cutoff, usually 00:00 UTC.
Exchange APIs typically give free historical bars, often with limits on how far back or how many you can pull per request.
Key idea: Every data set has an origin, a session definition, and a list of things it excludes. Write those down for the data you use, because your results are conditional on all three.
A data record
Keep a short file with your data set. It should answer: source, instrument, session and time zone, adjustment method, date range, known gaps, and the date you downloaded it. This takes five minutes and saves you from re-discovering the same quirk six months later.
| Field | Example |
|---|---|
| Source | Broker A historical API |
| Instrument | ES continuous, back-adjusted, roll on volume |
| Session | Full session, 18:00 to 17:00 Eastern |
| Bars | 15-minute |
| Range | 2016-01-04 to 2024-06-28 |
| Known gaps | 2020-03-09 partial day; three missing overnight sessions in 2019 |
| Downloaded | 2024-07-01 |
Try it: Pull one day of 5-minute bars for the same instrument from two different sources (two brokers, or a broker and a public site). Count how many bars differ in close by more than a tick. That number is your data uncertainty, and any edge smaller than it is not testable with this data.
Recap
- Stock data: check adjustment, delisted names, session hours and volume source.
- Futures data: separate contract months, continuous series need roll rules, session and settlement matter.
- Forex data: no central tape; bid vs mid, broker spread, and daily close time all vary.
- Crypto data: exchange-specific, spot vs perp, wash volume, 24/7 with a stated daily cutoff.
- Keep a data record with source, session, adjustment, range, gaps and download date.
See it drawn
Original diagrams for the ideas on this page. Illustrative, not real market data.