Everything in this chapter so far has looked at multi-core correctness from the hardware side - boot sequencing, coherency, lockstep comparison against a reference model. None of that tells you whether the hardware actually behaves the way real multi-core software depends on it behaving. That's what C-level multi-core tests are for: running genuinely concurrent code across multiple cores and checking that the synchronization primitives software relies on actually hold up under real contention.

9 min read | Intermediate Level

What You'll Learn

  • Why hand-written multi-core C tests struggle to hit the interleavings that actually matter
  • How to verify a spinlock or atomic primitive is doing its job, not just that the final answer came out right
  • Why barrier synchronization needs its own dedicated test, separate from locks
  • How to make a multi-core C test self-checking without a human watching waveforms

Why Hand-Written Multi-Core C Tests Don't Scale

A directed single-core C test is something an engineer can reason about line by line. A multi-core test is fundamentally different: the actual behavior depends on how instructions from independent cores interleave in time, and that interleaving is decided by the hardware, not by the source code. Writing a C test by hand that reliably lands on one specific, interesting interleaving out of the enormous number of legal ones is close to impossible - people are simply not good at predicting how several independently-timed streams of instructions will interact.

The practical answer is volume and variation rather than precision: run the same concurrent scenario many times, with randomized delays, randomized core-to-resource assignments, and randomized data, and let enough repetitions land inside the narrow timing windows where a real bug lives. This is the same principle already used for cache-coherency litmus tests earlier in this chapter, applied more broadly to whatever synchronization primitive is under test.

Verifying Synchronization Primitives Directly

Multi-core software depends on a small set of primitives - spinlocks, atomic read-modify-write operations, barriers - and if any one of them is subtly broken, every piece of software built on top of it inherits the bug. Verifying a spinlock means checking something stronger than "the program finished with the right answer": it means proving mutual exclusion actually held, meaning no two cores were ever inside the critical section at the same time, for the entire duration of the test, not just at the moment it happened to be sampled.

A useful, self-checking way to test this directly is a shared counter that every core increments a fixed number of times while holding the lock. If the lock genuinely provides mutual exclusion, the final count is exactly the number of cores times the number of increments per core, no matter how the increments interleave. If even one increment is lost to a race, the final count comes up short - a clean, deterministic pass/fail condition that needs no human judgment.


// Shared state (would live in a coherent, non-cached-locally region)
volatile int counter = 0;
volatile int lock = 0;

void spin_lock(volatile int *lock) {
  // Atomic compare-and-swap retry loop: only one core's swap succeeds
  // when the lock is 0, everyone else spins and retries
  while (__sync_val_compare_and_swap(lock, 0, 1) != 0) {
    // spin
  }
}

void spin_unlock(volatile int *lock) {
  __sync_lock_release(lock);
}

// Run on every core
void lock_stress_test(int increments_per_core) {
  for (int i = 0; i < increments_per_core; i++) {
    spin_lock(&lock);
    counter++;          // the operation the lock is supposed to protect
    spin_unlock(&lock);
  }
}

// Checked after all cores finish:
// counter must equal (num_cores * increments_per_core) exactly.
// Any smaller value means the lock let two cores in at once.

The compare-and-swap in this example is what a spinlock built on RISC-V's LR/SC (load-reserved/store-conditional) pair, or ARM's LDREX/STREX, ultimately compiles down to. On real hardware, the reservation a core takes with LR is invalidated the moment any other core writes to that address, which is exactly the mechanism that makes only one core's store-conditional succeed. Verifying that invalidation happens correctly across cores - not just that one core's retry loop eventually succeeds in isolation - is a large part of what this kind of stress test is actually exercising underneath the C source.

Barrier Synchronization Needs Its Own Test

A barrier is a different primitive from a lock, and it fails differently. Its job is to hold every participating core until all of them have arrived, then release them together - and the specific bug to hunt for is a core that gets released early, before every other core has actually reached the barrier. A lock-style stress test doesn't catch this, because a barrier bug doesn't corrupt a shared value; it just lets one core see stale state slightly too soon.

A direct way to test this is to have each core write a per-core "arrived" flag immediately before calling the barrier, then have one core - after the barrier releases everyone - check that every other core's flag was actually set before its own release happened. A barrier that occasionally lets a core through one instruction early will fail this check even though the program's final output might still look correct.

Testing a barrier: each core sets its arrived flag just before the barrier; after release a checker core confirms every flag was set, which catches a core let through before the last one arrived

Stress Tests as a Coverage Tool, Not a Smoke Test

It's tempting to treat a passing multi-core C test as a checkbox - it ran, it didn't hang, it printed the right answer. The real value of these tests is as a coverage tool aimed at timing windows that are otherwise nearly impossible to hit on purpose: many cores hammering the same lock, the same barrier, or the same memory region simultaneously, with randomized timing across many seeds, is what actually has a chance of landing inside the narrow interleaving where an arbitration, snoop, or reservation bug shows itself. A single clean run proves far less than the same test run hundreds of times with different random seeds.

Making the Result Self-Checking

None of this is useful if a human has to inspect a waveform to know whether it passed. The pattern that scales is: each core computes something locally, one designated core aggregates the results from every other core through the same shared-memory or mailbox mechanism being tested, compares the aggregate against an expected value computed independently - ahead of time, on the host, not derived from the same hardware being tested - and writes a single pass/fail status to a known register or memory location the SystemVerilog testbench can poll and report through the regression the same way it reports any other test.

Common Mistakes

Giving a multi-core test generous timing margin between cores instead of forcing contention.

A test where core delays are staggered comfortably apart, whether by accident or by an overly cautious synchronization design, never actually contends for the lock or barrier it's meant to be stressing. Contention has to be forced, not hoped for.

Not bounding an atomic retry loop, so a real bug shows up as a hang instead of a failure.

A spinlock that never terminates because of a hardware livelock looks identical to a testbench timeout from the outside. Give retry loops a bounded iteration count and report an explicit failure when that bound is exceeded, rather than relying on the simulation timing out.

Checking only the final aggregated result, not whether the primitive actually held throughout the run.

A subtle race can occasionally still produce the correct final answer by coincidence. Where practical, add an explicit, continuously-checked invariant - like the arrived-flag check for a barrier - rather than trusting the end state alone.

Summary

In this article, you learned:

  • Hand-written multi-core C tests can't reliably target specific interleavings; volume and randomization across many runs is what actually finds synchronization bugs
  • A shared-counter stress test under a lock gives a clean, deterministic way to prove mutual exclusion actually held, not just that the final answer looked right
  • Barrier synchronization fails differently from a lock and needs its own dedicated arrived-flag style check
  • Multi-core stress tests are most valuable as a coverage tool for narrow timing windows, run repeatedly with randomized seeds rather than once
  • A self-checking multi-core test aggregates each core's result, compares it against an independently computed expected value, and reports pass/fail through a location the testbench can poll automatically
Did you find what you were looking for?

Quiz

Test Your Knowledge

1 of 5 Score: 0 / 5