D
DAAVILE Algo Trading
← Back to Research
Research Lab

ORCHESTRATION POSTMORTEM: SIX HOURS, ZERO ROWS

Four terminals burned 6.5 hours and returned zero rows. This orchestration postmortem exposes the retry loop, dead lane, and fixes that now stop the waste.

Orchestration Postmortem: Six Hours, Zero Rows
6.5 hoursone cell looping, zero progress
~250+requeue cycles, no backoff
0run_register_result calls
13raw runs banked, never registered

The orchestration postmortem starts with a bad surprise: four busy terminals, 6.5 hours, and zero registered rows. On 2026-10-31, we dispatched two background workers to a mutation round. One champion strategy fed a ladder of candidate variants across four terminals, with about six hours of wall time.

When the operator stopped the round at 14:03, the scorecard showed roughly 250 requeue cycles and 13 full backtests that were never registered. The campaign card hadn't moved a single row. Six and a half hours of fleet compute bought nothing.

This run failed without crashing, and that was the dangerous part. There was no error page and no red alert, only busy machines and an empty card. The failure became the spine of this article; the fixed rules it bought are the ending.

We read this failure against the lab's broader three-phase testing protocol. The protocol separates clean, delayed, and robustness runs; this postmortem focuses on the plumbing around them.

#The orchestration postmortem begins with an over-ambitious brief

The brief was ambitious on paper. Each worker had to find at least twelve qualifying candidates and climb from the first depth level to the second, using four terminals inside roughly six hours.

Two workers split the fleet. One took two terminals and an explicit instruction to hunt high-drawdown-group variants. The other took the remaining two and subtler clues.

Both started at 06:38 without a pinned model. Each worker chose its own reasoning model at spawn time, so nobody downstream could tell which model produced which behavior.

Communication was sparse from the first minute. Worker B sent a heartbeat at 06:42 saying "investigating." Worker A sent one at 07:18 saying "implementing."

Neither worker sent another message. There was no completion, no escalation, and no further heartbeat. In hindsight, that silence was the first failure signal, and nobody was watching for it.

A silent run is already a lost run; the logs only show it after the fact.

#The six-and-a-half-hour timeline shows one early win and a long retry storm

The driver log for the round runs 1,330 lines. Every marker below is a logged event in that log or in the run inbox; none is inferred.

TimeEventWhat it meant
06:38Both workers dispatchedFour terminals committed, ~6 h window opened
06:42Heartbeat from worker B"investigating" — the last message B ever sent
07:13–07:3013 runs banked raw on lane A's primary terminal12 unique candidates, one four-week window in early 2026
07:18Heartbeat from worker A"implementing" — the last message A ever sent
07:30:25Requue storm beginsOne unprovable cell starts looping to the front of the queue
07:30–14:02~250+ cycles, zero progressLaunch fails, full backtest runs, journal check fails, requeue — forever
14:02:21Storm endsNothing registered; both workers stopped by operator order
14:03AftermathThree of four terminals empty; the primary holds leftover static charts
Failure timeline of the 2026-10-31 orchestration round showing dispatch at 06:38, last heartbeat at 07:18, thirteen raw runs banked by 07:30, the requeue storm from 07:30:25 to 14:02:21, and the stop at 14:03
The round's whole life on one axis. All productive work finished by 07:30; everything after was one cell, requeued.

All productive work finished by 07:30. Thirteen raw runs were banked inside seventeen minutes on one terminal that did exactly what it was told.

Everything after 07:30:25 was the same cell requeued for 6.5 hours. The workers stopped by operator order at 14:03; both returned a stopped state with an outcome of failed and a liveness of exited. Nothing in the system raised a hand.

#A five-step requeue loop turned one unprovable cell into 6.5 hours of waste

The storm had one cell at its center: a deep-points variant of one of the first worker's candidates. It repeated a five-step loop with no exit condition.

Diagram of the requeue storm loop: launch fails with config path missing, cell requeued to the front, a full seventy-five second backtest runs on the primary terminal, the journal check finds no config-load resolve, and the cell is marked not banked and requeued again
The loop that ate the evening: five steps, roughly 250 repetitions, no backoff.
StepWhat happenedWhat should have happened
1 · Launch attemptFails instantly: the config path does not existRun the prepare chain first — inputs, then config
2 · RequeueCell goes back to the front of the queueCount the attempt; three and quarantine
3 · Full backtest~75 s on the primary terminal, every cycleSame — the compute was never the problem
4 · Journal checkNo config-load resolve for the setfileSame — this check was correct and is the law
5 · Verdict"Not banked" → requeue to frontMark journal-unproven, move on, escalate

The loop was a doctrine failure before it was a code failure. The lab's two-step staging law requires a candidate's setfile to exist in every agent sandbox directory under its exact configuration name.

The backtest ran on the primary terminal, but the journal recorded no config-load resolve for that setfile. The law held for the window that produced the thirteen banked runs and failed for the storm cell. A staging path and name mismatch inside the worker's own code was never isolated because the worker was stopped.

The correct response was to quarantine an unprovable cell after a fixed number of attempts. Don't bank it. Move on and escalate. Instead, the driver requeued it forever with no backoff, fail counter, or escalation path.

The terminals looked busy but were empty. One failed instantly; the other reran the same full backtest every cycle.

We call this failure a requeue storm. It belongs to the same family as the fleet-sweep "blocking read wedge" because it returns wherever a retry path exists without a budget. A retry without a budget is a while-loop that spends machines.

Compute cost makes the failure expensive. A full backtest on the primary terminal took roughly seventy-five seconds, and the storm consumed hours of it for a cell that could never be banked.

The second terminal burned its evening failing instantly on a config path that never existed. The two terminals on the dead lane idled at zero. Four machines produced one useful window of seventeen minutes, and an operator had to escalate at two in the afternoon to find out.

A counter that counted to three would have prevented those costs.

#One lane died silently while three others never produced a registered row

Lane scoreboard for the failed round: the primary terminal banked thirteen raw runs with none registered, the second terminal failed every launch with a missing config, one probe returned run id zero with zero open charts, and the fourth lane executed no runs
Four terminals, two workers — and the only number that mattered at 14:03 was zero.

The second worker's lane died without producing a storm. Its agent terminal exited early, with liveness marked "exited" before the stop order arrived.

Its only probe returned a run id of "0" instantly and opened zero charts. The lane's own ledger logged it as failed. The worker produced a tools directory, one candidate, and zero executed runs. A lane that cannot launch is absent; treat it as down and park it.

The prepare chain was the known cure: prepare inputs, prepare the configuration, then launch. A zero run id means the lane is down and must be parked. Neither worker applied that rule.

The orchestrator acknowledged two heartbeats, then stopped checking worker liveness. It didn't poll again until the operator escalated at about 14:00. Both workers already carried unverifiable liveness and a required-action flag that went unactioned.

No watchdog, autosync, or auditor daemons were running, and the health file had been stale for days. A worker can die silently. The coordinator cannot be allowed to.

The same failure shape later surfaced in the data-integrity audit. A health check that only reads a registry can still miss a lane that died outside it.

#The run produced 13 raw backtests but 0 registered results

The useful output was small, but it wasn't empty. Thirteen raw backtests existed; none had crossed the registration door.

FactValueStatus at 14:03
Raw runs banked13 (12 unique candidates)Exist as raw reports and log lines only
WindowFour weeks in early 2026Phase-one condition
Profit range1,355.88 – 2,195.98Unverified — not quotable yet
Trade counts3,192 – 3,203Band: green
Max equity drawdown8,598.40 on all 13Identical across all 13 — flagged for auditor eyes
Registration calls0The card still held only the two manual baseline cells

Thirteen full backtests existed only as raw reports and log lines. The 1,330-line driver log contained zero registration calls, and the campaign card still held only the two manual baseline cells.

The storm cell's number never arrived, and the thirteen runs that did arrive never moved. All 13 carried the same 8,598.40 maximum equity drawdown. That may fit a shared baseline floor, but it was unproven, so it stayed out of the record.

After the stop, three of the four terminals stood empty. The primary held leftover static charts from the killed loop: gold, fifteen-minute timeframe, one per chart. The next driver run had to close them one by one.

#Six root causes explain why busy terminals stayed empty

Six failures compounded. A bad cell met a broken staging path, three dead lanes, weak supervision, and a brief that allowed one stall to crowd out every other worker.

  1. Unbounded requeue loop, no circuit breaker. One unprovable cell locked a worker for 6.5 hours. No max-attempts counter, no escalation path, no completion message ever sent.
  2. Staging-law breach on the storm cell. The two-step setfile law held for one window and failed for another: a path and name mismatch inside the worker's own code, never isolated because the worker was stopped.
  3. Three of four lanes never launched a single successful run. One failed every attempt on a missing config path; one answered with run id "0"; one never executed anything. The known prepare-chain cure sat written down, unapplied.
  4. The second worker died silently. A structured background session with no pinned model went to "exited" liveness, and nothing on the coordinator side noticed for hours.
  5. No supervision on the orchestrator side. Two heartbeats were acknowledged and then nobody polled again until the operator escalated. Required actions sat unactioned; the watchdog daemons were not running.
  6. Scope was unrealistic for the window. Twelve qualifying candidates per worker, two ladder levels, four terminals in ~6 hours, with the launch protocol broken on three of four lanes, and a design that let one bad cell starve every other lane.

#Five circuit breakers now stop failed runs from burning the fleet

We adopted the recovery rules the same night, and they remain standing law:

  • Never requeue-front the same cell-and-candidate more than 3 times. Then mark it journal-unproven or launch-failed and move on. Escalate through the inbox if a lane is dead for more than 30 minutes.
  • No banked result without proof. Before any launch on a fragile lane, run the prepare chain and require a non-zero run id plus the config-load resolve in the journal. Run id "0" means the lane is down: park it.
  • The coordinator polls liveness at least every 15 minutes while a run is live, and acts on required-action flags instead of acknowledging heartbeats.
  • Register only what can be proven. Banked cells need journal proof and verified-report notes. The thirteen probe runs stay unregistered until the auditor proves them, including the identical-drawdown flag.
  • Isolate failures per lane. One bad cell must never be able to starve the other lanes.

The successor round launched the next day with prove-first evidence, a fifteen-minute heartbeat, a three-attempt circuit breaker per cell, and park-on-zero for dead lanes. Thirteen probe runs from that failed evening remain unregistered pending auditor proof, including the identical-drawdown flag. We kept that scar visible so nobody forgets what an unproven row costs. The evening's last useful product was a driver that can no longer spend six hours proving nothing.

The failure is now part of the operating record. You can compare the current campaign state with the published DAAVILEFX results.

Takeaways
  • Retry only within a hard budget: the storm cell requeued roughly 250 times; the breaker now allows 3 attempts.
  • Register only proven results: the 13 raw runs stay unregistered until the auditor proves them.
  • Park a dead lane before it consumes the round: run id "0" means the lane is down.

Open the quantum dashboard to see the live evidence and the gates that now prevent a silent six-hour waste.

#Questions we get asked

#What happened during the 6.5-hour mutation round?

It was a requeue loop. Four terminals and two workers kept the round busy, but one unprovable cell was requeued roughly 250 times. The run ended with 13 raw backtests and 0 registered results.

#Why did the campaign card stay empty?

The journal could not prove the configuration loaded. The backtests ran, but the lab's rules do not allow an unproven result to be registered.

#What stops the requeue loop now?

A three-attempt circuit breaker now stops it. The coordinator checks liveness every 15 minutes and parks a lane that returns run id "0".

#Did the 13 raw runs become published results?

No. All 13 remain unregistered until the auditor proves them, including the identical 8,598.40 drawdown across every run.

Copied
// Software lab RYIUK — local LLM inference, backtesting pipelines, agentic workflows. No signals. No courses.
RYIUK.pro →