Nexus deep diveStep 5 · ProductionCompanion to the case study
The platform that tests a strategy runs it every day: each market on its own clock, a rehearsal before the first order, a reconciliation after the last, four layers of watching, and backups that are restored.
A backtest runs once. Production runs every day, including the day a feed stalls, a broker changes its dealing rules or a job fails at three in the morning. Nexus is run by one operator and a fleet of AI agents, so the system has to notice its own faults and put each one where someone will act on it. This page follows one day: what runs when, how the trading day is checked before and after, what watches the jobs, what happens when something goes red, and how the data is kept. Each part is marked with what runs today.
Read on 2026-10-22 from the main branch of each service: the timer and unit files, the checker and unit registries and the trader's check code, with the schedule checked against the production host's installed timers at 14:42 UTC. Restore figures are from the drill's record.
Every scheduled job is a systemd timer set to a fixed UTC time or to a local time, and the order scheduler is a resident service that fires each market at its exchange's time. There is no crontab and no workflow engine, as set out in Scheduled by systemd. Data is refreshed and backed up overnight, the checks report in the morning, and orders go out in the afternoon, UTC.
| UTC | Trading | Data | Checks and backups |
|---|---|---|---|
| 00:00–05:00 | — | Gaps in the live capture refilled from the broker's history, 03:37 on weekdays. Contract symbols and their check, Databento contract legs and continuous series, index constituents, futures carry and its check, the IG price blend recomputed in full, the crypto universe, Dukascopy | Daily dumps 00:10, weekly on Sundays 01:10, monthly on the 1st 02:10, offsite copy 04:00. Restore drills on the 2nd at 02:30 and on Mondays at 03:00 and 03:30 |
| 05:30–07:50 | Reconciliation and the day ledger 05:30, the broker's dealing rules 05:38, roll readiness 06:07, measured spreads 06:15 | The 25-check validation battery 05:50, a sweep for minute data with no hourly bar 07:50 | Morning reports from 06:20, the checker registry 07:07 |
| 13:30–19:07 | The pre-trade test 14:37, orders 16:03 to 19:03, a target-only snapshot 16:30, the live book run beside its point-in-time twin 18:25 | Databento 13:30, realised volatility and correlation 18:00 and 18:10 | The checker registry 19:07 |
| 22:00–23:45 | Fills into the book of record 22:00, the broker's transactions and funding 23:15, the portfolio snapshot 23:30 | The Cboe volatility complex 23:45 | — |
The trader runs as a resident loop that reads the schedule every minute. Each market trades at its own exchange's local time, so the order slots follow each exchange's daylight saving and holidays. Around the slots sit a rehearsal of the whole book before the first order and a reconciliation of the day after the last. Times below are UTC for this week.
00:30 Chicago time, 05:30 UTC this week
One job runs in a fixed order: the slippage bridge from model to live prices, the day ledger, a breakdown of any gap between the model's positions and those held, the bridge from simulation to live, the reconciliation of the trader's book against research's targets, a check that the ledger was written, and a proposed verdict on the session.
Can halt tradingOnly the book reconciliation, and only when the weight gap is more than a tenth of the book's gross weight or the two sides' configuration signatures disagree. A halt freezes new orders and keeps positions.
Every day, with a failure alert on the job.
05:38 to 06:15 UTC
The broker's dealing rules for each market, such as minimum size and margin, are read and kept as dated observations. Roll readiness is checked for every dated contract; it detects and never trades. Measured spreads are recorded for the cost model.
Every day.
14:37 UTC, Monday to Friday, before the first slot
The pre-trade test logs in to the broker for real and reconciles the trader's book against the broker's positions. Then it runs the live trading path over every active market with orders switched off: market status, prices, contract and roll-pair resolution, then the margin cap, the halt and the kill switch, and the position and order limits, price collars and self-match check on every order. Each would-be order is recorded with any limit it touched. When trading is halted or the kill switch is set, the test skips and records why.
Found a fault in itselfOn 27 August 2026 the rehearsal wrote a simulated position onto the working book, and the 18:03 slot then sent an opening order with no closing one, on the demo account. Since 28 August a dry run never touches holdings, and roll pairs are checked on every tick.
Since 24 August 2026. An unclean reconciliation, a halted path, an error or any would-be flip, open or close sends an alert; a broken path also fails the job. A book that has drifted is caught again by the first slot's own reconciliation, which stops that slot.
16:03, 17:03, 18:03 and 19:03 UTC this week
Slots are set by market group: metals at 11:03 Chicago time, sugar at 17:03 London, grains, FX, livestock and the other softs at 12:03 Chicago, rates at 13:03 Chicago, energy at 14:03 New York and equity indices at 14:03 Chicago. A slot missed by a restart is caught up within two hours, and an exchange holiday is skipped. Before each cycle the book is reconciled with the broker, and every order passes the staged limits, collars, margin cap, halts and kill switch set out in Execute.
On an IG demo account. The position and order limits and the price collars run at warn: they flag a breach without stopping the order.
22:00 to 23:30 UTC
The day's fills are copied into the book of record, the broker's transactions are pulled to reconcile overnight funding, and the portfolio is snapshotted.
Every day.
31 checks on the trading before it, each scored pass, warn, fail or skip. One fail makes the day fail and sends an alert.
Trading halts after three tick errors in a row, three failed orders in a row on open markets, or a fall in equity from its peak deeper than a threshold derived from the book. It also halts after a set number of days with no authenticated action from the operator. A kill switch outside the trading process stops it, and one that cannot be read counts as set.
Every halt keeps positions. Closing them is a decision for a person. Each brake in detail →
For your firmA rehearsal of the whole book before the first order and a scored ledger of the day after the last are two of the cheapest controls a systematic desk can add. How execution works →
The live feed is checked in the same rhythm as everything else, every fifteen minutes and every night. The checks themselves are set out in Capture and Validate and clean; here is when they run.
A failed job is only useful if someone hears about it. Nexus watches at four speeds, and every service and timer in the unit registry names the layer that watches it.
| Layer | How often | What it watches | When red | State |
|---|---|---|---|---|
| Failure handlers | On failure, and a sweep every 20 minutes | Every system service and timer on the production host. A unit that fails sends its last log lines to the alert bus; the sweep catches any unit left failed. | A post to the operations channel in Slack, at most once an hour for each unit | Runs |
| Fast checks | Every 15 minutes | Seven checks on the machinery of change: the release installer, the coordinator's turns, the merge backstop, and GitHub's quota and status | Opens or updates an S3 incident, and pages once when it opens and once when it recovers; delivery of those pages is not yet on record | New since 18 October 2026 |
| Checker registry | 07:07 and 19:07 UTC | 43 checks, 42 of them on the production host and one in CI: alert wiring, deploy and release drift, data and trading correctness, database grants and audit rules, capacity, CI, and the work board's own records | One work item per finding, reused while it stays open; no item closes itself | Runs |
| Host collectors | Every 15 minutes; staleness hourly | Disk, memory, swap, unit states, database sizes and logs on each host. A host that stops reporting fails the hourly check. | A post to Slack after two bad readings in a row, at most once a day for each check. Memory above 75% or swap above 50% for two readings counts as bad. | Runs |
Every service and timer on the production host, and those the collectors report on two others, each with its owner, the signal that matters, its threshold and what watches it.
A wired handler is not a delivered page. The alert bus made 8,934 deliveries to Slack in the 30 days to 27 August 2026, and none failed. Of the production host's 126 user-level units, 112 have a handler on a route added on 11 October 2026; delivery from them is not yet on record, and a checker opens a work item for any that fails.
Alerts reach Slack through Hermes, which commits each one to its database before a separate loop posts it.
An alert that only says something is wrong is forgotten by the next morning. In Nexus a red result becomes a record that someone has to close.
A checker's red finding opens one row on the work board, keyed by the check and the finding. The next sweep finds the same row open and adds nothing, so a fault that persists does not flood the board, and a finding about an open row is added to it as a note. No row closes itself: an agent or the operator closes it with an outcome.
The fast layer opens an S3 incident instead. The register grades every incident: S1: money at risk, or a control that keeps it safe not working. S2: a protective control failing with nothing saying so. S3: detection or diagnosis degraded, the fault contained. S4: latent, found by inspection before it could act. The register went into use in mid-October 2026; earlier incidents were entered afterwards.
Work items are picked up like any other work on the board: leased by one agent seat, fixed in its own worktree, reviewed by a different agent unless the fix is trivial, and deployed on the operator's word. See Agents.
From the checker's red to green, 4 September 2026, in UTC.
06:35 The job that renders the status board fails.
07:07 The morning checker sweep finds it and opens a work item.
13:03 The fix merges and is installed.
13:04 The job runs again and exits clean. The checker reads 0 failed of 83 user units.
A backup nobody has restored is a hope. Nexus dumps the trading database every fifteen minutes and every database each night, copies them offsite, and restores them on a schedule to prove the data comes back.
| Copy | When (UTC) | Kept on the host | Kept offsite |
|---|---|---|---|
| The trading database | Every 15 minutes | 24 hours | 24 hours |
| The live price capture | Every 15 minutes, the last hour, overlapping | 7 days, one per hour | 7 days, one per hour |
| Every database, daily | 00:10 | 6 days | 7 days |
| Weekly | Sunday 01:10 | 3 weeks | 4 weeks |
| Monthly | 1st, 02:10 | 3 months | 12 months |
Mondays, 03:00 UTC
The book of record is restored from its latest weekly dump into a scratch database. The drill passes when the restore completes.
Mondays, 03:30 UTC
Each archived file tree is checked against the SHA-256 recorded when it was written, then opened and read to its end.
The 2nd of each month, 02:30 UTC
The TimescaleDB market store is restored in full from that night's daily dump, and its hypertables and chunks are counted against the live database.
First run on 2 September 2026: 12 hypertables and 661 chunks in 8 minutes 59 seconds.
Most production faults arrive with a change. Nexus has one path for a change to reach production and one rule for who may write production data.
Thirty-four agent seats build Nexus, as set out in Agents. In production they read, report and fix through the same board as any other change. They do not trade, and nothing they write reaches production without the deploy tool and the operator's word.
An agent reads the 16-job overnight roster, the day ledger, the last batch sent to the broker and any drift between what is merged and what is deployed. It opens a work item for each finding and can post one digest to the operations channel on a morning with something to follow up. It never deploys.
A read-only review of the latest batch: orders, fills, the touch at the time of each order, round-trip time and the reconciliation. It never places, cancels or closes an order.
A checker's work item is ranked by the coordinator, an agent itself, and taken by one seat under a lease. Unless the fix is trivial, a different agent reviews the exact commit before it merges.
An agent runs the deploy tool only on the operator's word, and the tool's refusals apply to agents and the operator alike.
A shared layer for the agents: acknowledged messages between roles, and one live view of every seat for the coordinator. The role inboxes are live; the coordinator does not read the seat view yet.
Each line here was checked against the code, the timer files or the registries on 22 October 2026.
Sources, read 2026-10-22: the production host's installed system and user timers, read at 14:42 UTC; the timer and unit files on the main branch of the trader, data, backup, book-of-record, risk, research, console and host-monitoring services and of the tooling repository; the checker registry, the fast-check roster and the unit registry; the trader's pre-trade, reconciliation, day-ledger, limit and halt code; the backup service's retention settings and restore-drill code; the database write-authority standard; and the dated status records for the restore drill of 2 September 2026, the incident of 27 August 2026, the work item of 4 September 2026 and the alert deliveries to 27 August 2026.
I can put a pre-trade rehearsal, a daily reconciliation ledger and alerting that someone acts on around your trading, or review the ones you run now.