Validate and Clean.

From vendor delivery to a clean series in five steps

The Nexus overview shows validation and cleaning as one picture. Here is each step: what Nexus does, and whether it runs today.

1. 1

   Raw kept

   As delivered, tagged with its source

   Every vendor bar lands tagged with its source and feed. A unique key allows one row per bar per source, so a duplicate cannot be stored. Hourly and daily bars are insert-only: a bar delivered again never overwrites the stored one. Each live broker bar is written once.

   Runs

   Minute bars from vendors accept a re-delivered bar in place. Some raw repairs have set corrupt zero or missing fields to empty in place.

2. 2

   Validate

   25 checks per dataset, plus nightly futures checks

   The dataset battery runs 25 checks and records one result per dataset per check: pass, warn, fail, skip or too little history. A failed bar-level check, such as an outlier or a duplicate, can send the dataset to quarantine. A failed series-level check, such as a flat range or a crushed adjustment, puts the whole series under review for a person to confirm. The rest only warn. Separately, 11 futures checks run every night on the front and next contract pairs and the contract symbols.

   Stalled

   The battery last ran in full on 23 September 2026, across about 3,200 datasets. Putting it back on a schedule is the first open item. The nightly futures checks ran last night.

3. 3

   Clean

   One logged operation per fix

   Each fix is one operation in a log, with statistics before and after. A dropped bar moves to a quarantine table, from which it can be restored. The clean copy of a dataset is a pure function of its raw bars and its operation log, so it can be rebuilt from scratch at any time.

   Runs

   6,632 operations logged.

4. 4

   Promote

   Clean only when the checks pass

   A dataset moves from unclean to validating, then to cleaning, and then to clean, or else to quarantined. Today 6,432 datasets are clean, 45 are quarantined and 21,868 have not been through promotion.

   Runs

   Most datasets have not been promoted, so research does not require clean.

5. 5

   Serve

   One data API for research and trading

   Research and trading read through the data API. At the start of a research run, a series marked corrupt or misderived is dropped, with its reason recorded, and a run left with no series is refused. A stale series raises an alert and is noted on the run, which continues.

   Runs, with limits

   Reads return raw bars by default. The clean layer is read only when asked for.

What the checks look for

Each check is aimed at a failure that has happened somewhere: a flat range from a stale feed, a timestamp shifted by an hour, a roll that looks like a crash, a currency quoted upside down. A failure quarantines the dataset or puts the series under review, as in step 2; advisory checks only warn.

For your firmWhen a result looks too good, every bar behind it can be traced to the vendor's delivery, to the checks it passed or failed, and to every change made to it.

Every fix is an operation you can undo

Cleaning never edits raw data. It writes an operation to a log and rebuilds the clean copy from raw plus the log. Removing an operation and rebuilding restores the data as it was.

• Quarantine, not delete

  A bar removed by cleaning is moved to a quarantine table with the operation that removed it. Undoing the operation puts the bar back.

• Wrong contract

  When a continuous series picks up a bar from the wrong contract, the correction is its own operation, with the bars it touched, rather than a quiet edit.

• Restatements refused

  The simulation stores each nightly forecast once. If the data under a stored forecast later changes, the nightly job refuses to overwrite it and fails, so the change is investigated instead of quietly absorbed. In October 2026 this caught a restatement of two soft-commodity series.

Checks on the live feed, as it arrives

History can be checked at leisure, but a live price has to be judged before the book trades on it. These guards run on the live capture and on the series the live book reads.

• Capture rules Runs

  Ticks are stamped on arrival and each bar is written once. A late tick is dropped and a gap is never filled with a made-up bar. Watchdogs reconnect a silent feed. The rules are set out in Capture.

• Gap refetch Runs

  Each weekday night, gaps of 30 minutes or more are fetched again from the broker. The minutes that should exist are learned from each market's own last four weeks, not assumed from a timetable.

• Hourly gap sweep Runs

  Each morning, any hour that has minute bars but no hourly bar pages operations.

• Splice guards Runs

  The series the live book reads refuses a first broker return of more than about 5 %. It reports a market stale after six refusals in a row, and it times the broker tail in open-market hours. The join is described in History meets live.

• Market status Runs

  Open, closed and holiday status is recomputed every five minutes for 47 calendars, so a closed market is not mistaken for a dead feed.

• Freshness panel Age only

  The trader's console shows each market's data as fresh or stale by the age of its newest bar: up to 16 hours is fresh for exchange data and 48 hours is stale, while for the blended series the limits are 4 and 18 hours. Age alone misses thin data. One coffee series showed fresh for three months while it held about one bar a day instead of ten.

What was known, and when

A backtest should see only what was known on the day. For prices that means bars that had closed. For reference data it means the version that had been published.

• Completed bars only. Research and the live book use a bar only once it has closed by the decision time, through the same engine. There is no forward-looking mode.
• Contract observations. Settlement, volume and open interest are append-only, enforced in the database. Each row carries three timestamps, a content hash and a revision number.
• Rate vintages. One interest-rate reference series is kept with two clocks: when a value applies and when it became available. Each of its 73 revisions is chained to the one before, with a content hash.
• Broker contracts. Changes to the broker's contract details are kept as an append-only history.
• Forecast availability. A stored model forecast carries the time it could first have been known: the end of its last input bar, plus a publication lag and a compute allowance.
• Not yet for prices. Price bars have no vintages. Research reads up to a cut-off on bar time, not on arrival time, so a vendor's later revision or late delivery is not replayed.

Next: a data-quality panel on the trading screen

The freshness panel judges age alone, and the dataset battery has stalled. The planned panel covers every tradeable market in three tiers: downloaded history, the broker-blended series and the live stream. A single green, amber or red banner sits at the top of the trading screen.

1. 1

   Revive the inputs

   Put the dataset battery and the store's coverage summary back on a schedule, and show that both run.

2. 2

   Measure history and the blend

   Density against each market's own baseline over the last 5 and 20 sessions, gaps and overdue bars by the market's calendar, bad bars, roll agreement, and how far the broker's returns stray from the exchange's after the handoff.

3. 3

   Measure the stream

   Tick age, quote delay, 1-second coverage, and crossed or zero spreads, rolled up into the banner on the trading screen.

• Every measure follows the market's calendar, so a closed market is never red for age or gaps.
• A measure that cannot be computed shows as unknown, never as zero.
• The first version displays only. It does not stop the book trading.

StatusPlanned. The work is scoped and its design is under review. Nothing on this list runs yet.

What runs, stated plainly

A data-quality page that hides its own gaps would undercut its point. Each line here was checked against the database or the code on 15 October 2026.

• This page describes the data platform. It shows no strategy returns and makes no performance claim.
• Counts move as data lands and checks run; each is dated to the day it was read.

Sources, read 2026-10-15: the validation results, dataset states, dispositions and cleaning operation log in the market-data store; the check registry and the cleaning, splice and capture code on the services' main branches; the data gate at the start of a research run; and the work board for the stalled and planned items.

Reading