Debugging a silicon failure means getting from a symptom, such as a hang, a wrong result or a crash, to a root cause: the exact design error, electrical weakness or defect behind it. On silicon this is slow and hard, because the cause may have happened millions of cycles before the symptom and almost nothing inside the chip is visible.
This article walks through the stages of a silicon bug from the first sighting to a confirmed root cause, and what typically makes each stage take time.
7 min read | Intermediate
What You'll Learn
- Describe the stages from a failure to a root cause
- Explain why cause and symptom are far apart on silicon
- Use the sighting and bug terminology correctly
- Estimate how long silicon debug takes and why
From Symptom to Cause
- Detect: a test, an exerciser or software reports a failure.
- Reproduce: make it happen again, ideally every time, as in Reproducing and Isolating a Failure.
- Localize: narrow it to a block, a feature and a time window.
- Root cause: find the exact mechanism, using trace, state dumps and simulation.
- Confirm: show the cause explains the failure, for example by reproducing it in simulation or by testing a workaround.
- Resolve: a workaround, an erratum, or a fix in the next stepping.
Each stage narrows the search. Most of the time goes into reproducing and localizing, not into the final explanation.
Why It Is Hard
On silicon, the first visible symptom often comes long after the error. A wrong value is written to memory, sits there unused, and is read much later by software that then crashes.
The trace buffer holds only a short window, usually just before the symptom. Debug therefore works backward: find where the bad value came from, then move the trigger earlier and run again, until the capture window contains the real error.
Quick Check
Q: A crash happens at cycle 9,000,000, and the trace buffer stores the last 2,000 cycles before a trigger. Why is triggering on the crash rarely enough?
Show Answer
The trace then covers only cycles 8,998,000 to 9,000,000, near the symptom. The error that corrupted the state may have happened millions of cycles earlier. The team must find an earlier trigger, such as a write to the corrupted address, to capture the real cause.
Sightings and Bugs
Teams separate what they observe from what they know. A failure that has been seen but not yet explained is a sighting. Only when the root cause is confirmed as a design error does it become a bug.
Many sightings turn out not to be chip bugs at all: a test error, a board problem, a software bug or a known issue seen again. Classifying them quickly is the subject of Classifying Failures.
How Long It Takes
A published study of one large processor program gives a sense of scale. About 1 % of all its bugs were found after silicon, and half of those were found in the first three months.
Root-causing took around ten days on average for low and medium severity bugs, with one in ten taking three weeks or more. High severity bugs averaged about six days, because they got the most people and the best equipment. The program also relied on a cycle-reproducible mode to make failures repeat exactly.
Quick Check
Q: Why might high severity bugs be root-caused faster than low severity ones, even though they are not simpler?
Show Answer
They block progress or customers, so they get priority: more engineers, more debug platforms and more lab time. Low severity bugs wait in a queue behind them, which adds days without any technical cause.
In Practice
Every sighting is logged in a tracking system with its symptoms, platform, conditions and test. Daily triage meetings decide who takes it and how urgent it is. Good notes matter: a sighting that cannot be reproduced today may match one seen weeks later on another platform, and the match is often the key clue.
Common Mistakes
Mistake #1: Calling a sighting a bug too early
Many sightings are test, board or software problems. Confirm the root cause first.
Mistake #2: Debugging from the symptom only
The cause may be far earlier. Work backward and move triggers earlier.
Mistake #3: Skipping confirmation
A plausible explanation is not proof. Reproduce it in simulation or with a targeted test.
Practice Exercise
Challenge: Plan the Debug Effort
A program expects 40 post-silicon bugs, half in the first three months. Root-causing takes about 10 engineer-days per bug on average, and roughly three sightings are investigated for every real bug, each non-bug sighting costing about 2 engineer-days. How many engineer-days does the first three months need for this work?
Hint
Count bugs and non-bug sightings separately.
Solution
20 bugs x 10 days = 200 engineer-days. Three sightings per bug gives 60 sightings, of which 40 are not bugs, costing 40 x 2 = 80 days.
Total about 280 engineer-days in three months, roughly five engineers full time, before any fix work. The numbers are assumptions for the exercise.
Summary
- Debug moves from detection through reproduction and localization to a confirmed root cause.
- Cause and symptom can be millions of cycles apart; triggers are moved earlier to capture the cause.
- A sighting is an unexplained failure; it becomes a bug only when the design error is confirmed.
- Root cause typically takes days to weeks, and priority affects the time as much as difficulty.