A full SoC regression is expensive to run and slow to turn around, and "keep running it until coverage looks good" isn't a strategy - it's a way to burn compute budget without a clear answer to when the chip is actually ready. This article covers how regression gets structured at this scale, and what closure - deciding verification is actually done - really requires.

7 min read | Intermediate Level

What You'll Learn

  • Why an SoC regression needs to be tiered rather than run as one flat suite
  • Why closure needs several metrics to agree simultaneously, not one number crossing a threshold
  • How to read a bug-discovery trend as a closure signal, not just a coverage percentage
  • Why regression health depends directly on the coverage-closure and triage work covered earlier in this chapter

Why Regression Gets Tiered

A full SoC regression can take hours to run and consume enough compute that running it on every single commit isn't practical. The standard answer is tiering: a small, fast smoke suite - the handful of scenarios that would catch an obviously broken build - runs on every commit or every few commits, catching gross breakage within minutes. A fuller regression, covering a much wider slice of the scenario and configuration space, runs nightly. The complete regression, including the long-running negative tests, multi-core stress scenarios, and low-probability corner cases, runs on a slower cadence - weekly, or ahead of a specific milestone - where the turnaround cost is acceptable because the frequency is lower.

Tiered regression: a small smoke suite on every commit catches gross breakage in minutes, a wider nightly regression covers more scenarios and configurations, and the full regression with long negative, stress and corner-case tests runs weekly or before a milestone

This tiering only works if each tier's scope is deliberate rather than arbitrary: the smoke suite has to actually catch the failure modes a broken commit is likely to produce, or it stops doing its job of catching problems early and cheaply, and everything shifts onto the slower, more expensive tiers instead.

What "Done" Actually Requires

Closure isn't one metric crossing a threshold - it's several conditions holding true at the same time. A useful reference point is chipverify.com's own Sign-Off Criteria page, which lays out the gates a verification effort typically has to clear together: code coverage and functional coverage both closed against their targets (with any gap explicitly waived, not silently ignored), every assertion passing across the full regression with no exceptions on critical paths, zero open blocking or critical-severity bugs, the complete regression suite passing against one specific, frozen RTL commit rather than a moving target, and the verification plan itself checked off feature by feature.

The important part isn't the specific thresholds - it's that no single one of these is sufficient on its own. High coverage against a regression riddled with open critical bugs isn't done. A clean bug list against thin, stale coverage isn't done either. Closure is the point where all of these gates read green against the same frozen commit at the same time.

Reading a Bug-Discovery Trend

Coverage percentage answers whether the planned scenarios ran. It doesn't by itself answer whether the design is actually solid, and a verification effort's own history of finding bugs is a second, independent signal worth tracking alongside it. Early in a verification effort, coverage ramps quickly and new bugs turn up at a high rate, because almost everything the regression touches is new ground. As the effort matures, the rate of new bugs found per unit of regression time should decline, even as total test volume and coverage keep climbing - that declining rate, against sustained or increasing stimulus diversity, is itself part of what a mature team reads as evidence the design has actually stabilized, not just that coverage numbers look complete.

The trap is reading a flat bug-discovery rate as success without checking why it's flat. A declining bug rate against a regression that's still generating meaningfully new stimulus each run is a good sign. The same flat line against a regression that's quietly stopped exploring new scenarios - because coverage plateaued and nobody added new stimulus to push past it - means nothing at all; there's simply nothing new left in what's being run to find a bug in.

Regression Health Depends on What Came Before

A tiered regression strategy and a clean closure decision both depend on the coverage-closure and failure-triage practices already covered in this chapter, not on anything new. Directed, gap-aware coverage closure is what keeps the nightly and full-regression tiers actually productive as they scale, rather than spending most of their compute re-covering bins that closed weeks ago. Automated failure triage is what keeps a growing regression's failure list legible - without it, a large regression's own noise, the same known bug tripping many tests at once, buries the one new failure that actually matters under a pile of duplicates nobody has grouped together yet.

Common Mistakes

Treating overall coverage percentage as sufficient proof of closure by itself.

A high coverage number next to open critical bugs, or against a regression that isn't passing cleanly, isn't closure. All the sign-off gates have to hold together, not just the one that happens to look best.

Signing off regression results against a moving RTL target instead of one frozen commit.

A "green" regression that was actually run across several different RTL versions as fixes trickled in doesn't prove anything about the chip as it will actually tape out. Closure regression has to run against one specific, frozen commit.

Reading a flat bug-discovery rate as evidence the design is clean without checking whether the regression is still finding new ground.

A declining bug rate against a regression that's stopped exploring new scenarios reflects stalled stimulus, not a stable design. Check that coverage and test diversity are still moving before trusting a flat bug curve.

Summary

In this article, you learned:

  • SoC regression is typically tiered - a fast smoke suite on every commit, a fuller nightly run, and a complete regression on a slower cadence - because running everything on every commit isn't practical at this scale
  • Closure requires several gates to hold at once: code and functional coverage, a clean assertion pass rate, zero open critical bugs, a full regression pass against one frozen commit, and a checked-off verification plan
  • A declining bug-discovery rate against sustained or growing test diversity is a meaningful closure signal in its own right, separate from coverage percentage
  • A flat bug-discovery rate against a regression that's stopped exploring new scenarios is not evidence of a stable design
  • Regression health at scale depends directly on the coverage-closure and failure-triage practices covered earlier in this chapter, not on anything separate from them
Did you find what you were looking for?

Quiz

Test Your Knowledge

1 of 5 Score: 0 / 5