D
DAAVILE Algo Trading
← Back to Research
Research Lab

BIT-IDENTICAL CONTROL: RUN IT FIRST, OR THE WAVE IS VOID

One bit-identical control runs before every mutation wave. Two grader defects showed why: a stale file and a midnight date can erase otherwise valid evidence.

Bit-Identical Control: Run It First, or the Wave Is Void
to the centcontrol reproduction tolerance
2 defectsstale-file wins + midnight drop
Raw rowsthe only source of truth to grade from
cell onethe cheapest cell in the wave

Every mutation wave starts with a bit-identical control. It isn't an experiment. It's a configuration we already know the answer to, run bit-identical to the known result before anything new is spent. Its only job is to fail loudly if the run has drifted.

Without the control, a wave's numbers are indistinguishable from a wave that measured nothing. With it, drift is detected at cell one instead of after the campaign is reported. We found 2 defects in the measuring layer: a control that looked like it had failed when it had not, and a wave that silently vanished after midnight. Both failures push toward the same discipline.

#A bit-identical control checks the instrument before the wave

A control cell is a frozen copy of a verified baseline. It runs with zero mutation. Its expected output isn't a number to discover; it's a number already certified. The arithmetic is unforgiving: the control must reproduce the baseline to the cent. Not close. The cent.

What the control checksWhat it protects
The test harness itself — data slice, broker contract, tester session, session windowEvery other cell in the wave, which is interpreted against this run
That the mutation is the only difference in the waveAttribution — the reason a mutation cell "improved" or "collapsed"
That the toolchain has not drifted since the baseline was certifiedComparability across weeks, across waves, across the whole campaign

The control is also an arena's admission ticket. An arena is approved only when its frozen-parent arm reproduces the control to the cent; when it does not, the arena is demoted to behavioural-only, and per-arm numbers from it are read as behaviour, never as a gate result. Nobody gets to argue a failed control into a pass. The control does not negotiate.

#The arena: one control, many arms

The control earns its keep inside a structure called an arena. Each wave is organized as one arena per family per cell: six mutation arms, one frozen-parent arm, and one sacrificial arm whose job is to try to blow the strategy up on purpose. The frozen-parent arm is the control, and the arena is approved only when that arm reproduces the known result to the cent.

ArmWhat it isWhy it exists
Mutation arms (6)One axis changed from the controlAttribution — each number answers "what did this one lever do?"
Frozen-parent arm (1)Zero changes, the known answerThe gate — the arena is void if it does not reproduce
Sacrificial arm (1)A deliberate destabilizationProves the wave can detect a failure, so a clean grade means something

The sacrificial arm is the piece outsiders skip. A wave in which every cell passes proves nothing unless the wave has demonstrated it can produce a fail. The blow-up arm is the lab's negative control: it establishes that the failure-detection path works, so the passes on the other arms are informative rather than merely consistent.

#What the control cannot catch

Honesty about limits: a control cell catches drift in the instrument and nothing else. It compares a run against a known result. It cannot tell you the known result was ever correct. If the baseline itself was mis-certified, a faithfully reproducing control locks the error in place and makes it permanent. It cannot catch a data slice that changed before the baseline was certified, because both runs share the same wrong tape. And it cannot substitute for the row-count check, because a control that reproduces perfectly inside a wave whose other cells were dropped by a selection bug is a clean measurement of nothing.

Those limits do not weaken the case for the control; they locate it. The control is a drift detector bolted to the front of a wave, and its whole value comes from being cheap enough to run before anything expensive happens. What it cannot do (audit the baseline's original certification, or notice a tape that was wrong on both runs) has to be handled elsewhere in the chain, by separate receipts kept for exactly that purpose. A lab that expects one artifact to cover the whole audit will under-build both the artifact and the audit.

A failed wave shows why that boundary needs its own evidence. Our deleted wave case study shows what happens when a pipeline loses it.

Three control blind spots: a mis-certified baseline, a tape that was wrong on both runs, and cells dropped by a selection bug
What a control cannot see. It certifies the run, not the truth — the baseline's own certification is a separate receipt, kept separately.

The control certifies the run, not the truth. The truth needs the baseline to have been right in the first place, which is a separate receipt, kept separately.

#The grader kept a stale result instead of the latest

The first bug was in the grading script, not the engine. The collector kept the first result file per setfile-and-terminal pairing, in directory listing order. When a wave was run twice (rebuilt to fix something, or simply re-run), the directory held two sets of result files with similar names, and glob order decided which one became the truth. It could pick the older run. The wave got graded against numbers from a previous life.

The symptom was specific and misleading. The control cell, which had actually reproduced perfectly, was graded as a failure. Reading that symptom as an engine problem would send a debugger hunting through a strategy that was never broken. The actual defect was a collector choosing a file by alphabetical accident.

SymptomObvious (wrong) readingActual cause
Control cell "failed" its reproducibility checkThe strategy drifted; re-certify the baselineThe grader kept the older result file and graded a stale run
A rebuilt wave got a garbage gradeThe rebuild was faultyTwo runs of the same cell existed; the wrong one won

The fix is one line of principle: the latest result wins, keyed on the run's own timestamp, never on listing order. And one line of procedure: delete a wave's folder before rebuilding it, so the ambiguity never exists. Both are now standing law.

Two grader failure paths: a stale control result winning over a fresh result, and a hardcoded date making a wave vanish
Two ways a grader lies. One manufactures a failure that did not happen; the other manufactures a silence where evidence should be. Neither raises an error.

#A hardcoded date erased the wave after midnight

The second bug was quieter, and quieter bugs are the dangerous ones. The grader had a date hardcoded into its selection logic. Any cell run after midnight matched nothing. The cells were not misgraded; they were not graded at all. No error, no warning, no empty-state alarm: the run simply produced a valid result that no report would ever name.

A wave that runs is not a wave that measured. Until the grade counts the rows, the wave does not exist.

The silent failure mode should worry anyone running long campaigns. A whole wave consumed its compute, its cells ran, its CSVs were written to disk, and the afternoon's grade table showed nothing because the selection key had rolled past midnight. Detecting it requires a habit, not a clever tool: before reading any grade, check the row count. A wave that legitimately ran does not become zero rows. Zero rows is a grader alarm, and a silent alarm is worse than no alarm because it looks like a result.

Execution conditions need their own check, too. The same discipline shapes our latency tolerance ladder, where each delay rung has a fixed purpose.

The fix mirrors the first bug's: the selection defaults to today's date, so the window rolls with the clock instead of freezing on the day the script was written. The lesson is the same one in both bugs: the selection logic is part of the instrument. An engine bug corrupts one number. A selection bug corrupts the entire table, and a table that is corrupt in both directions (stale rows and missing rows) is worse than no table at all.

#Raw rows are the only valid grading source

Raw rows are the only valid grading source. We learned this when a wave's summary block printed a row the result files did not contain. The summary is a convenience the producer renders for humans; the per-cell result files are the measurement. Grading is done from the files.

ArtifactWhat it isMay be graded?
Per-cell result files (raw rows)The measurement, written as the run concludesYes — this is the only source of truth
Wave summary blockA rendered view, printed for human scanningNo — it can print a stale row
Operator's memory of "what we saw"AnecdoteNo — a run is its files or it is fiction

#The rule that generalizes: instrument and experiment are one system

Both defects lived in the measuring layer, and neither lived in the engine. That is not a coincidence; it is the normal shape of the problem. The engine is exercised by every run, so its failures announce themselves early and loudly. The measuring layer is exercised only at the edges: a rebuild, a midnight, an unusual filename. Those are exactly the conditions nobody rehearses, so the bugs that live there survive longer and cost more.

Failure locationWhat it corruptsHow it announces itself
Engine / strategyOne number — a cell's resultEarly and loudly; every run exercises it
Grading / selectionThe entire table, in both directionsLate and quietly; only edge conditions exercise it
Operator habitThe interpretation of everythingNever; habit has no error channel

The control cell is the argument for checking the measuring layer actively. The measuring layer is the part that fails quietly, so it is the part we check first. A wave does not get to trust its own table. It earns that trust by running the known answer first, insisting it reproduce to the cent, and counting the rows before anyone reads a number.

#A clean wave starts with a control and ends with a row count

Put the three rules together and the wave has a shape. It starts with the control, grades from raw rows, and verifies the row count before anyone reads a number.

A clean wave flow: frozen control cell first, mutation cells next, latest-wins grading from raw rows, and row-count verification before any result is read
The clean shape. The control is cell one; the grade comes from rows; the row count is checked before the table is trusted.
  • Cell one is the control. If it does not reproduce the baseline to the cent, the wave stops there. Cells not yet run cost nothing.
  • Every mutation cell is one axis from the control. If two things changed, the cell measures an interaction, not a mutation: a different question, a different analysis.
  • Grading is latest-wins on run timestamps, from the raw per-cell files, never from a rendered summary.
  • The row count is verified before the table is read, because a table with missing rows and a table with stale rows look identical from a distance.

#The control cell is the cheapest tripwire in a wave

A control cell is cheap. A wasted wave is not. Two real defects in one measuring path (a stale file silently outranking a fresh one, and a date key silently erasing a night's work) proved that the measuring layer fails as surely as the strategy does, and usually more quietly. The control is what turns both failures into noise instead of evidence: run the known answer first, insist it reproduces to the cent, grade only from raw rows, and count the rows before you trust the table. Everything else is optimism with a spreadsheet.

Takeaways
  • The control is a drift detector, not a verdict: it certifies the run, not the truth.
  • Grade from raw per-cell rows with latest-wins timestamps, never from a rendered summary or directory order.
  • Count the rows before reading the table. A wave that ran does not become zero rows; zero rows is a grader alarm.

The next step is to inspect the live wave before trusting its score. The full campaign handoff, grader fixes, and wave-by-wave receipts are in the DAAVILEFX Quantum research dashboard. The broader published performance results show what survived those checks.

#Questions we get asked

#What is a bit-identical control cell?

A bit-identical control cell is a frozen copy of a verified baseline. It runs with zero mutation. It must reproduce the certified result to the cent before the wave continues.

#Why run the control before the other cells?

The control runs first because failure is cheap there. If the known result changes, we stop before spending compute on a wave we cannot trust.

#What did the two grader defects show?

Both defects came from grading, not the strategy engine. One collector chose the older result file, while a hardcoded date hid cells run after midnight.

#Can a control prove the strategy works?

A control proves repeatability, not truth. It shows that the current run matches a known result. The original baseline still needs its own certification and receipts.

Copied
// Software lab RYIUK — local LLM inference, backtesting pipelines, agentic workflows. No signals. No courses.
RYIUK.pro →