A failure that cannot be reproduced cannot be debugged. On silicon, many failures happen once in hours of running, and never at the same moment twice, because the chip interacts with clocks, memory and devices whose timing varies from run to run. This article covers where that variation comes from, how teams make failures repeat, and how they then shrink a long, complex failing test into a short one that points at a single block.
7 min read | Intermediate
What You'll Learn
- List the sources of non-determinism on silicon
- Explain how cycle-reproducible modes work
- Shorten a failing test by bisection
- Isolate a failure by turning features and units off
Why Failures Do Not Repeat
The same test can follow a different cycle-by-cycle path each run because of:
- Asynchronous clock domains: the phase between clocks differs each time, so crossings land on different cycles.
- Memory: refresh timing and training results change access latencies.
- I/O and interrupts: devices respond at their own pace.
- Analog effects: PLL lock time, temperature and supply noise.
- Uninitialized state: registers without reset start with different values.
A bug that needs two events to coincide within a cycle or two will then appear only when the timing happens to line up.
Making Failures Repeat
Teams fight variation in several ways. Some chips have a cycle-reproducible mode, designed in before tapeout, which locks clock ratios, fixes memory timing, makes I/O responses deterministic and resets all state, so that a run from reset follows exactly the same cycles every time.
Without such a mode, teams reduce variation on the platform: run from a known reset, disable asynchronous devices, fix memory training values, and hold the temperature and voltage steady. A failure that becomes reproducible this way can then be studied with trace and state dumps.
Quick Check
Q: A failure happens once in about 50 runs. After enabling a cycle-reproducible mode, it either happens every run or never. Why?
Show Answer
With timing fixed, every run follows the same cycles, so the alignment of events that causes the failure is either always present or always absent. If it is absent, the team changes the conditions, such as clock ratios or the test seed, until a configuration that fails every time is found.
Shrinking the Test
A failing test may run for billions of cycles. Bisection shortens it: run half the test and check the state; if it is already wrong, the error is in the first half, otherwise in the second. Repeat, halving each time.
Each step needs a way to check whether the state is still correct, such as comparing memory or register contents with a reference model, as in Checking Results on Silicon. Thirty halvings narrow a billion cycles to about one.
Isolating by Elimination
Turning features off one at a time shows which ones the failure needs. Chips provide controls for this: disable cores, caches, prefetchers, speculation, power states, or change clock ratios.
Each experiment either keeps or removes the failure. A failure that disappears when a second core is disabled points to a multi-core interaction.
One that disappears at lower frequency points to timing. Together, the results point to a block and a mechanism.
Quick Check
Q: A failure disappears when the L2 prefetcher is disabled. Does that prove the prefetcher has the bug?
Show Answer
No. The prefetcher may simply create the traffic that exposes a bug elsewhere, such as in the cache controller. It shows the prefetcher is involved in the failing scenario; the root cause still has to be found with trace or simulation.
In Practice
Reproduction and isolation take most of the debug time. Teams keep libraries of scripts that switch features, fix timing and run bisection automatically, often overnight. Disabling a feature that removes a failure can also become the workaround, covered in Workarounds, so these experiments serve two purposes.
Common Mistakes
Mistake #1: Debugging a failure that happens only occasionally
Make it reproducible first; otherwise every experiment is ambiguous.
Mistake #2: Changing several things in one experiment
Change one variable at a time, so each result is clear.
Mistake #3: Treating the feature that hides a failure as its cause
It may only expose a bug elsewhere. Confirm with trace or simulation.
Practice Exercise
Challenge: Bisection Cost
A test fails after 4 x 109 cycles. Each bisection run takes 20 minutes, including reloading and checking state. How many runs are needed to narrow the error to within 1,000 cycles, and how long does that take?
Hint
Each run halves the interval: find n with 4 x 109 / 2n below 1,000.
Solution
4 x 109 / 1,000 = 4 x 106, and 222 is about 4.2 x 106, so about 22 runs. At 20 minutes each that is a little over 7 hours, one overnight job. From there, trace capture around that window finds the exact cycle.
Summary
- Asynchronous clocks, memory, I/O, analog effects and uninitialized state make runs differ.
- Cycle-reproducible modes and controlled platforms make failures repeat exactly.
- Bisection shrinks a long failing test to a short window.
- Disabling features one at a time points to the blocks and mechanism involved.
