Inventory & rehearse
Build the target, rehearse restores into scratch databases and deploy every application with its timers off.
Nexus engineering · Deep dives
Move the writer.
Prove it writes.
A production migration moves authority as well as data: who may write each dataset, what depends on what, and how to get back. In August 2026 Nexus moved to a new production server in three cuts on one day. This is how the move was designed, and what the checks that looked green missed.
Decide before the windowOne writer per datasetFind every consumerProve the work
A retrospective of the August 2026 move, from its runbook, decision record and post-mortem. Reviewed 2 November 2026 and rechecked 3 November.
01 · The move
One day of cuts. Weeks of preparation.
The production platform moved from its old server to a new one in three cuts on Saturday 22 August 2026, after about two weeks of preparation. The record includes outages, missed dependencies and checks that gave false confidence, which is what makes it useful.
Market data
Stop the data writers, take the final copy, restore it and repoint the data path.
Trading record
Move the trader, its live capture and its state, handing over each service and timer explicitly.
Remaining services
Move the rest of the service tier, then find and verify every consumer. Here, cuts 2 and 3 at first missed their callers; see 05.
Keep, then retire
The old server stayed frozen as the rollback host for a month and was decommissioned on 23 September 2026.
The first cut took about an hour.
From 06:19:33 to about 07:20 UTC, including defects fixed in the window; the data legs took about 29 minutes. Measured history, not a recovery-time promise.
02 · Design decisions
Decide the hard things before the window.
Most of a migration's risk is settled before any writer stops. These were decided in advance, and each has its reason in the record.
Disk, not CPU.
The old server was 78% full, against a 58 GB market-data database growing by about 5.2 million one-second bars a day (measured 18–19 August 2026). That set the timetable.
One move, not a split.
Leaving trading on the old server and moving research was considered and rejected. The trader writes live capture into the same market-data database that research reads, so a split would have separated the processes while leaving them sharing one database across a network link.
Separate the irreplaceable from the re-fetchable.
Historical market data, 99.4% of all database bytes, could be fetched again; the live capture and the trade record could not. So they moved in separate cuts: market data first, then the trader and its 87 MB database in a short writer stop.
Timed around the markets.
Cut 1 ran on a Saturday, after Friday's US close. The first trading to depend on it was Sunday's weekend book, so the repoint was verified the same day rather than discovered on Sunday.
Only what needs writers stopped.
Tooling, restores, credentials and the deployed applications, with their timers off, were staged days ahead. The window held only the final copy, the streaming handover, the repoints and switching timers on.
Move the host, not the code.
The trader cut copied the revision already running rather than a newer checkout, so a host move was never mixed with a code or schema change.
03 · Make the hidden dependencies visible
A server inventory is also a consumer inventory.
Who reads and writes?
List APIs, long-running workers, capture processes, reverse proxies, user and system timers, cron, local files and databases without an obvious service. A process may keep an old connection after its environment file changes.
Can the state be restored?
Rehearse database and extension compatibility, roles, ownership, grants and application reads. This move pinned the TimescaleDB extension version and used the documented serial restore. The backups carry data, not ownership or grants, so roles and grants were generated from the old server on the day and applied after the restore.
What lives outside the database?
Configuration, service credentials, local execution state and filesystem paths all matter. Compare secret-file custody and configuration keys without exposing values. A value identical on both servers can still be wrong on the new one.
04 · Control the transition
The critical boundary is writer ownership.
Three moments in each cut, and what each one decides, distilled from the runbook.
Admit the migration window.
Prove restore compatibility, inventory consumers and establish the running revision. Record the stop list, expected data loss and outage tolerance, abort triggers, rollback owner and the state that must be preserved.
Keep target writers disabled. Test credential recovery and required paths before the window; a healthy process on the new host does not yet establish a working service.
Take a consistent final handover.
Stop scheduled triggers and long-running writers explicitly. Verify that writes have ceased, take the final copy and prove the restored objects, grants and counts. A timer name alone may miss a continuously running capture process.
Restart consumers after repointing so they release old connections. Declare which host now owns each writer, then verify new records and the route a real consumer uses.
Rollback now has a data problem.
Once the new host accepts writes, the frozen old copy is behind. Restarting it without reconciling new records can lose history or create conflicting authority.
Stop new writers first, identify the accepted delta and decide whether it can be replayed or whether repair forward is safer. For trading, reconcile broker orders, positions and cash before resuming. A retained server is a recovery option, not proof of a lossless rollback.
05 · Failures that changed the method
Green was sometimes the wrong answer.
| What looked reassuring | What actually happened | Stronger acceptance evidence |
|---|---|---|
| The service starts | Proxies still sent traffic to services on the old server. The whole console returned 502s for an hour, and the trader panel had been down for three. | Inventory every caller and proxy; exercise the route from the consumer. Count obsolete destinations across the complete inventory. |
| The backup job is green | The old host could keep backing up a frozen database while the new database lacked coverage. | A new-host backup artifact with meaningful size, a restore proof, and retirement of the obsolete backup source. |
| The unit is active | The trader scheduler ran with database authentication failures; another missing dependency left useful work skipped. | Read the journal and measure the output: rows, instruments, completed work or pushed bytes. |
| The second boot has no failed units | Absence of failures would also occur if catch-up jobs never fired. | The recorded check found 27 services ran within 101 seconds of boot and all returned success with exit code 0. |
So completion is defined by the capability the platform must deliver. A running process, a successful restore command and a configured timer are useful signals; each needs an observation that would fail if the work never happened.
06 · Close the operation
Retire the old host only after the evidence survives.
Recoverable data
Confirm backup coverage follows the authoritative copy. Prove restore results, grants and usable application reads. Smoke checks and a database restore exercise have different scope from full service recovery.
Working consumers
Exercise public and private routes, scheduled jobs, alert delivery and the first expected write after cutover. Reconcile trading state rather than inferring correctness from the new service’s health endpoint.
Durable ownership
Preserve the cutover record, running revisions, configuration decisions and unresolved exceptions. Remove obsolete writers, timers and routes deliberately; disabling a unit is not a complete decommission.
The record kept the three cuts separate from the later soak and decommission gates. At institutional scale, the change owner needs named acceptance evidence, a recovery decision and an exception record before declaring the operation closed.