Functional validation on silicon runs tests that look for logic bugs, using the silicon's huge speed to reach states no simulation could. The tests come in two broad kinds: focused tests written by experts for specific features, and random tests produced by generators.
On silicon, the most productive generators often run on the chip itself, creating, running and checking new tests endlessly. This article describes both kinds, how on-chip exercisers work, and the hardware tricks that push a chip into its hardest corners.
7 min read | Intermediate
What You'll Learn
- Compare focused and random tests on silicon
- Explain how a bare-metal exerciser generates, runs and checks tests
- Describe irritators and stress modes
- Explain why short bug-to-failure latency matters
Focused and Random Tests
Focused tests target a specific feature: a multiprocessor protocol, a register configuration, address decoding, a power state. Experts write them by hand, so they take effort, and they can run for hours on silicon.
Random and constrained-random tests exercise the chip in ways nobody planned: random instruction sequences, random interleavings of traffic from several agents. Constraints keep them legal and steer them toward interesting areas.
Silicon's speed changes the balance. A random generator can run hundreds of millions of seeds on silicon, so it explores far more than any hand-written suite.
Bare-Metal Exercisers
Loading each test from a host takes time, and lab time is scarce. So many teams use bare-metal exercisers: programs that run directly on the chip without an operating system and repeat a loop forever.
- Generate a random test case from a template, the architecture description and the system configuration, all built into the program image.
- Execute it on the cores.
- Check the results, as described in Checking Results Without a Simulator.
- Repeat, with no interaction with the host.
The exerciser is built offline into one executable image, so it needs no files or databases while running. Once loaded, it can run for days.
The generator must be simple and fast, because it shares the chip with the test. That is the main difference from pre-silicon generators, which run on separate servers and can afford slow, sophisticated generation.
Templates for Silicon
A test template describes the kind of test wanted, leaving the details random. On silicon a template is used for an enormous number of test cases.
A template that is too specific soon starts repeating similar tests; one for silicon must leave much more room for variation than a pre-silicon one. Teams that share one template language across simulation, emulation and silicon can move a failing scenario between platforms easily.
Quick Check
Q: Why does an on-chip exerciser use a simple generator instead of the detailed generators used in simulation?
Show Answer
It runs on the chip itself, so every cycle spent generating is a cycle not spent testing. A simple, fast generator keeps most of the time on executing tests, and the silicon's speed makes up for the simpler generation by running far more of them.
Irritators and Stress Modes
Some corners are hard to reach with any instruction stream. Designers therefore build small helpers into the chip:
- Irritators: logic that injects microarchitectural events at random, such as flushing a pipeline or invalidating a translation entry as if another chip had asked. The test does not have to create the conditions; the irritator does.
- Stress modes: non-functional settings that make a unit behave worst-case, for example a cache mode in which almost every access causes an eviction, piling traffic onto the next level.
Both cost area and power and must be designed in before tapeout, so they belong to Design for Debug.
Short Bug-to-Failure Latency
A bug can corrupt something long before anything visibly fails. A wrong value written to memory may not be read for thousands of cycles.
The longer that gap, the harder the debug. One technique, quick error detection, transforms tests so that each effect is checked soon after it happens, for example by reading back every memory location right after writing it.
Quick Check
Q: A cache bug writes a wrong value, but the test only reads it back 5 million cycles later. Why is that a problem, and how can the test be changed?
Show Answer
By the time the failure is seen, the trace buffer and most state show events millions of cycles after the cause, so finding it is very hard. Adding a read-back soon after each write, as quick error detection does, makes the failure appear close to the bug, where trace and state dumps still show what went wrong.
In Practice
Teams that rely on exercisers consider bugs found only by operating systems or real software as escapes from their strategy: the exercisers should have found them first. Exerciser runs are scheduled around the clock on as many boards as possible, with templates chosen from coverage results, as in Coverage on Silicon.
Common Mistakes
Mistake #1: Reusing narrow pre-silicon templates
They run out of new tests quickly on silicon. Give silicon templates room to vary.
Mistake #2: Loading every test from the host
The overhead wastes scarce lab time. Generate and check on the chip.
Mistake #3: Accepting long bug-to-failure latency
Failures far from their cause are slow to debug. Check results early and often.
Practice Exercise
Challenge: Throughput
An exerciser spends 2 ms generating a test case, 18 ms running it and 1 ms checking it. Loading a test from the host instead would take 200 ms per test plus the same 18 ms run and 1 ms check.
How many test cases per hour does each approach run? The times are assumptions for the exercise.
Hint
One hour is 3,600,000 ms.
Solution
Exerciser: 21 ms per test, so 3,600,000 / 21 = about 171,000 tests per hour. Host loading: 219 ms per test, so about 16,400 tests per hour. The on-chip exerciser runs about ten times more tests, and spends 86 % of its time executing them instead of 8 %.
Summary
- Focused tests target features; random tests from generators explore combinations nobody planned.
- Bare-metal exercisers generate, run and check tests on the chip in an endless loop.
- Silicon templates need more variation than pre-silicon ones.
- Irritators and stress modes push the design into rare corners.
- Checking soon after each effect keeps bug-to-failure latency short.
