In simulation, a reference model runs beside the design and checks every result. On silicon there is no simulator, and running a full reference model next to a chip at gigahertz speed is impossible.
So silicon tests need other ways to know whether the chip got the right answer. This article describes the four main techniques: self-checking tests, multi-pass consistency checking, signatures, and hardware checkers, and the trade-offs between them.
7 min read | Intermediate
What You'll Learn
- Explain why a full reference model cannot run beside silicon
- Describe self-checking tests and multi-pass consistency checking
- Explain how signatures compress results
- Choose a checking method for a given test
Why the Simulator Approach Does Not Work
A reference model is a program that computes what the design should do. In simulation, both run at the same slow speed and are compared step by step.
Silicon runs millions of times faster than any software model of it. The model cannot keep up, and the chip's internal state cannot be streamed out fast enough to compare. Checking must therefore be built into the test or the hardware.
Self-Checking Tests
A self-checking test knows its expected results and compares them itself. A directed test might compute a value two different ways, or check a known answer stored with the test.
Self-checks are precise and fail close to the problem, but they take manual work and only fit directed scenarios. They are often added to the most important scenarios to localize failures and stop errors from being masked.
Multi-Pass Consistency Checking
Exercisers often use a technique that needs no reference model at all: run each test case several times and compare.
- The first run, the reference pass, saves the final values of chosen resources: architectural registers and parts of memory.
- Later passes run the same test, often with some variation that should not change the result, such as different thread priorities or machine modes.
- Each pass's final values are compared with the reference. Any difference is a failure.
The idea is that a bug which depends on timing or on a rare interaction will hit some passes and not others, so the passes disagree. The generator must avoid anything with a legitimately unpredictable result, such as two threads writing the same location in an undefined order. Debug is also hard: the mismatch is found at the end of a pass, which can be millions or billions of cycles after the error.
Despite that, the technique has found some of the toughest functional bugs in large processors, especially in control paths and where control meets data.
Quick Check
Q: A multi-pass test has two threads write different values to the same address with no ordering between them. Why must the generator avoid this?
Show Answer
The final value depends on which write happens last, which can legitimately differ between passes. The passes would then disagree without any bug, producing false failures. Multi-pass checking only works on results that should be identical every time.
Signatures
When a test produces a huge stream of results, it can compress them into a short signature, such as a checksum or the value of a multiple-input signature register, which folds each result into a shift register with feedback. Only the final signature is compared, with a value computed once by a reference model offline or with another run. Any change in the stream changes the signature with very high probability.
The cost is localization: a wrong signature says that something went wrong, not where or when. Teams therefore take intermediate signatures, so a mismatch can be narrowed to an interval.
Hardware Checkers
Checkers built into the chip watch for generic errors on every cycle: accesses outside memory, transactions that never complete, timeouts in instruction completion, protocol violations. Monitors on the Die describes them. Failures caught by hardware checkers are the easiest to debug, because the chip stops close to the origin of the bug and the checker records the transaction involved. Their limit is cost: area, power and timing, so only generic, high-value checks are built in.
Choosing a Method
| Method | Strength | Weakness |
|---|---|---|
| self-checking | precise, fails near the cause | manual, directed scenarios only |
| multi-pass consistency | no reference model, catches timing-dependent bugs | restricts tests, fails far from the cause |
| signatures | checks huge result streams cheaply | says something is wrong, not where |
| hardware checkers | every cycle, stops near the cause | area and power; generic checks only |
Most programs combine all four: hardware checkers always on, multi-pass checking in exercisers, signatures in long data-heavy tests, and self-checks in key directed scenarios.
Quick Check
Q: A long test's final signature is wrong. How can intermediate signatures speed up the debug?
Show Answer
Comparing signatures taken at regular points shows the first interval in which they diverge. Debug then focuses on that interval, which can be rerun with trace triggered inside it, instead of on the whole test.
In Practice
When a check fails, the first step is to confirm it is real: rerun the test, on another core or another part. A failure that disappears on a different part points to a marginal or defective part; one that repeats everywhere points to a design or test bug. The test software itself is a frequent cause, which is why exercisers are run on emulators before silicon.
Common Mistakes
Mistake #1: Comparing results that may legitimately differ
Race-dependent results cause false failures in multi-pass checking. Keep compared resources deterministic.
Mistake #2: One signature for a whole long test
A mismatch then says nothing about where. Take intermediate signatures.
Mistake #3: Relying on one checking method
Each method misses something. Combine hardware checkers, multi-pass, signatures and self-checks.
Practice Exercise
Challenge: Narrow the Interval
A test runs for 10 billion cycles and takes a signature every 100 million cycles. The first mismatch appears at the 37th signature. Within which cycle range did the error first affect the results, and how much smaller is that than the whole test?
Hint
The 36th signature still matched.
Solution
Between cycle 3.6 billion and 3.7 billion, an interval of 100 million cycles, one hundredth of the test. The error may have happened earlier and only reached the signed results then, but the interval is where debug starts, for example by triggering trace inside it on a rerun.
Summary
- A full reference model cannot keep up with silicon, so checking is built into tests and hardware.
- Self-checking tests are precise but manual.
- Multi-pass consistency compares repeated runs and needs no reference model, but restricts tests and fails late.
- Signatures compress long result streams; intermediate signatures localize failures.
- Hardware checkers stop the chip near the cause; most programs combine all methods.
