15 min read | Advanced Level
What You'll Learn
- Debug X-propagation issues using waveform analysis and tracing
- Diagnose retention and isolation cell failures systematically
- Trace power state transition violations
- Correlate UPF intent across simulation, synthesis, and P&R
The Elusive Power Bug
Your power-aware simulation fails with a cryptic error:
UPF-ERROR: X-propagation detected at time 15,432ns Signal: noc_router/input_fifo/write_enable Source domain: PD_GPU (state: OFF) ERROR: X-state in memory write enable causes undefined behavior Simulation STOPPED
You have isolation cells specified for GPU outputs. They worked in thousands of previous simulations. What changed? Where exactly is the X coming from? Why isn't isolation preventing it?
You load the waveform viewer, but with 50,000 signals across 8 power domains, finding the root cause feels like searching for a needle in a haystack.
Without systematic debugging techniques, power intent issues can consume days of engineering time. This article teaches you the five-step framework to diagnose and fix these issues efficiently.
The UPF Debugging Process
Five-Step Debugging Framework
1. IDENTIFY: What failed? (X-prop, retention, state transition, synthesis mismatch) 2. LOCALIZE: Where in the design? (Which domains, signals, cells) 3. ISOLATE: When did it fail? (Timeline, state transitions, power events) 4. ANALYZE: Why did it fail? (Root cause - UPF, RTL, or timing) 5. FIX: Correct the issue and verify the fix
Systematic Debugging in Action
In a networking processor project, an intermittent bug appeared in only 1 out of every 200 boots: the packet processor would occasionally drop the first packet after wake-up from deep sleep. The debug team enabled comprehensive UPF tracing (+upf_trace_x +upf_verbose) and ran 10,000 simulations with randomized timing. They captured 47 failures and analyzed waveforms. Root cause: FIFO pointers lacked retention cells, causing undefined behavior when packets arrived within 50ns of restore. The systematic debug process—from bug report to fix—took 3 days instead of months, saving an estimated $3M respin.
Debugging X-Propagation
Step 1: Identify the X Source
UPF-ERROR: X-propagation detected at time 15,432ns Signal: noc_router/input_fifo/write_enable Source domain: PD_GPU (state: OFF)
Question: Which signal in PD_GPU is driving the X?
Use UPF Tracing to Find X-Propagation Paths
# Run simulation with X-propagation tracing
./simv +upf_iso_warn +upf_trace_x
Time 15420ns: PD_GPU transitioning to OFF UPF-TRACE-X: PD_GPU output signals driven to X: - gpu_inst/dma_req (32 bits) -> X - gpu_inst/dma_addr (40 bits) -> X - gpu_inst/cache_snoop (1 bit) -> X Time 15422ns: X-propagation path: gpu_inst/cache_snoop (X) -> noc_router/snoop_controller/snoop_valid (X) -> noc_router/input_fifo/write_enable (X) ROOT CAUSE: Signal 'cache_snoop' has NO isolation cell!
Step 2: Localize Missing Isolation
Check UPF for isolation coverage:
# UPF file inspection
set_isolation ISO_GPU -domain PD_GPU \
-isolation_supply_set SS_TOP \
-isolation_signal gpu_iso_enable \
-location parent \
-applies_to outputs # Should cover ALL outputs
Problem: The isolation applies to "outputs", but is cache_snoop considered an output?
Check Actual vs. Intended Coverage
# Use simulator report to see actual coverage
./simv +upf_dump_isolation > isolation_report.txt
Isolation Strategy: ISO_GPU
Domain: PD_GPU
Covered signals (47):
- gpu_inst/dma_req[31:0]
- gpu_inst/dma_addr[39:0]
- gpu_inst/interrupt_req
... 44 more ...
Uncovered outputs (1):
- gpu_inst/cache_snoop ? MISSING!
Reason: 'cache_snoop' is inout port (bidirectional), not classified as 'output'
FIX: Use -applies_to "outputs inouts" or explicitly name the signal
Step 3: Fix and Verify
# BEFORE: Only covers outputs
set_isolation ISO_GPU -domain PD_GPU \
-isolation_supply_set SS_TOP \
-isolation_signal gpu_iso_enable \
-location parent \
-applies_to outputs # Misses inout ports!
# AFTER: Covers outputs and inout ports
set_isolation ISO_GPU -domain PD_GPU \
-isolation_supply_set SS_TOP \
-isolation_signal gpu_iso_enable \
-location parent \
-applies_to "outputs inouts" # Fixed!
Re-running simulation... Time 15420ns: PD_GPU transitioning to OFF Time 15422ns: Isolation ISO_GPU enabled UPF-INFO: 48 signals isolated (including cache_snoop) UPF-INFO: No X-propagation detected Simulation PASSED
Use +upf_trace_x simulator option to automatically trace X-propagation paths from powered-down domains to active logic. This saves hours of manual waveform analysis.
Debugging Retention Failures
Symptom: State Corrupted After Power-Up
Time 10000ns: Saving CPU state (PC = 0x8000_4A3C) Time 10020ns: CPU powered down Time 20000ns: CPU powered up Time 20040ns: Restoring CPU state Time 20060ns: Verification FAILED Expected PC = 0x8000_4A3C Actual PC = 0x0000_0000 ? State lost!
Step 1: Check Retention Cell Supply
Most common cause: Retention supply powered down with domain.
# Check retention UPF
set_retention RET_CPU -domain PD_CPU \
-retention_supply_set SS_CPU_RET \
-save_signal {cpu_save_enable high} \
-restore_signal {cpu_restore_enable high}
# Check SS_CPU_RET definition
create_supply_set SS_CPU_RET \
-function {power VDD_CPU} \ # BUG: Uses VDD_CPU (powers down!)
-function {ground VSS}
Problem: Retention supply set uses VDD_CPU, which powers down! Retention cells lose state.
Use Waveform to Confirm
Open waveforms and add these signals:
cpu_power_enable # Power switch control VDD_CPU # Primary supply (should go to 0V) VDD_RET # Retention supply (should stay at 0.6V) cpu_save_enable # Retention save trigger cpu_restore_enable # Retention restore trigger cpu_inst/pc # Program counter value
Waveform analysis:
CORRECT TIMING:
Time: 10000ns 10010ns 10020ns 20000ns 20020ns 20040ns
| | | | | |
save: ━━━━━━━┏━┓ ← Assert save
power_en: ━━━━━━━━━━━┓ ┗━━━━━━━━━━━━┏━━━━━━━ ← Power down AFTER save
└─ Gap: 10ns ─┘
restore: ━━━━━━━━┏━┓ ← Restore AFTER stable
power_en: ┗━━━━━━━━━━━━┏━━━━━━━━━━
└─ Gap: 20ns ─┘
INCORRECT TIMING (causes corruption):
Time: 10000ns 10010ns 20000ns 20010ns
| | | |
save: ━━━━━━━┏━┓ ← Save asserted
power_en: ━━━━━━━┓ ┗━━━━━━━━━━━━━┏━━━━━━━ ← Power down BEFORE save complete!
Gap: 0ns! (TOO SHORT)
restore: ┏━┓ ← Restore before supply stable!
power_en: ┗━━━━━━━━━━━━━━━━┏━━━━━━━
Gap: 10ns (TOO SHORT)
Root cause confirmed: VDD_RET collapses to 0V when CPU powers down, causing retention cells to lose state.
Step 2: Fix Retention Supply
# BEFORE: Retention supply powers down
create_supply_set SS_CPU_RET \
-function {power VDD_CPU} \ # Wrong!
-function {ground VSS}
# AFTER: Dedicated always-on retention supply
create_supply_net VDD_RET # Separate retention supply
add_power_state VDD_RET \
-state {ON 0.6} # Always on, no OFF state
create_supply_set SS_CPU_RET \
-function {power VDD_RET} \ # Always-on supply
-function {ground VSS}
associate_supply_set SS_CPU_RET -handle PD_CPU
Step 3: Verify Save/Restore Timing
Even with correct supply, timing issues can cause corruption:
CORRECT TIMING:
Time: 10000ns 10010ns 10020ns 20000ns 20020ns 20040ns
| | | | | |
save: ?????????? ? Assert save
power_en: ???????????? ????????????????????? ? Power down AFTER save
+- Gap: 10ns -+
restore: ??????????? ? Restore AFTER stable
power_en: ????????????????????????
+- Gap: 20ns -+
INCORRECT TIMING (causes corruption):
Time: 10000ns 10010ns 20000ns 20010ns
| | | |
save: ?????????? ? Save asserted
power_en: ???????? ?????????????????????? ? Power down BEFORE save complete!
Gap: 0ns! (TOO SHORT)
restore: ??? ? Restore before supply stable!
power_en: ?????????????????????????
Gap: 10ns (TOO SHORT)
Retention Timing Requirements: Save must be asserted at least 5-10 cycles before power-down. Restore must be asserted at least 10-20 cycles after power-up to allow supply voltage stabilization. Violating these timings causes intermittent state corruption.
Quick Check
Q: Why do retention failures often appear intermittent (passing some runs, failing others)?
Show Answer
Retention cells have varying capture times due to process variation and routing delays. If the save signal duration or gap before power-down is marginal, faster cells complete capture successfully while slower cells don't. Small timing variations (PVT, clock jitter) affect which cells complete, causing non-deterministic failures. The solution is to provide adequate timing margin: hold save for 5+ cycles, wait 10+ cycles before power-down.
Debugging Power State Transitions
Symptom: Illegal State Transition Error
Time 25000ns: GPU voltage transition UPF-ERROR: Illegal power state transition Current state: DVFS (VDD_GPU = 0.8V) Target state: POWEROFF (VDD_GPU = 0.0V) PST does not allow DVFS -> POWEROFF Legal path: DVFS -> STANDBY -> POWEROFF Simulation STOPPED
Step 1: Examine Power State Table (PST)
# GPU Power State Table
add_pst GPU_PST -supplies {VDD_GPU}
add_pst_state ACTIVE -state {VDD_GPU ON} # 1.0V
add_pst_state DVFS -state {VDD_GPU SLOW} # 0.8V
add_pst_state STANDBY -state {VDD_GPU SLEEP} # 0.6V
add_pst_state POWEROFF -state {VDD_GPU OFF} # 0.0V
# Legal transitions (directed graph)
add_pst_transition -from ACTIVE -to DVFS
add_pst_transition -from DVFS -to ACTIVE
add_pst_transition -from DVFS -to STANDBY
add_pst_transition -from STANDBY -to POWEROFF
add_pst_transition -from POWEROFF -to ACTIVE
# NOTE: No direct DVFS -> POWEROFF transition!
Step 2: Find Offending RTL Code
Which RTL code triggered the illegal transition?
// Power management controller RTL
always_ff @(posedge clk) begin
case (power_mode)
MODE_ACTIVE: gpu_voltage <= 1.0; // ACTIVE
MODE_SLOW: gpu_voltage <= 0.8; // DVFS
MODE_SLEEP: gpu_voltage <= 0.6; // STANDBY
MODE_OFF: gpu_voltage <= 0.0; // POWEROFF
endcase
end
// BUG: Direct transition from MODE_SLOW to MODE_OFF!
always_comb begin
if (deep_sleep_request && (current_mode == MODE_SLOW)) begin
power_mode = MODE_OFF; // Illegal: DVFS -> POWEROFF!
end
end
Step 3: Fix FSM to Follow Legal Transitions
// FIXED: Multi-step transition through STANDBY
always_comb begin
if (deep_sleep_request) begin
case (current_mode)
MODE_ACTIVE: power_mode = MODE_SLOW; // ACTIVE -> DVFS
MODE_SLOW: power_mode = MODE_SLEEP; // DVFS -> STANDBY
MODE_SLEEP: power_mode = MODE_OFF; // STANDBY -> POWEROFF
default: power_mode = current_mode;
endcase
end
end
Result: Transition now takes 3 clock cycles (DVFS ? STANDBY ? POWEROFF) instead of 1, following legal PST path.
Draw your PST as a state diagram during design. This makes illegal transitions obvious and helps you design the power controller FSM correctly from the start.
Debugging Synthesis/P&R Correlation Issues
Symptom: Simulation Passes, Synthesis Fails
Power-aware simulation: PASSED (100% coverage) UPF linting: PASSED (0 errors, 2 warnings) dc_shell synthesis: Error: ISO_GPU isolation strategy references signal 'gpu_iso_enable' Error: Signal 'gpu_iso_enable' not found in design Error: Cannot insert isolation cells Synthesis FAILED
Step 1: Check RTL vs. Synthesis Netlist
RTL has the signal, so why doesn't synthesis find it?
// RTL design
module power_controller(
output logic gpu_iso_enable // Signal exists in RTL
);
always_ff @(posedge clk)
gpu_iso_enable <= gpu_power_enable_n;
endmodule
Check synthesis elaboration:
dc_shell> elaborate design Warning: Signal 'gpu_iso_enable' has constant value 0 Info: Optimizing away constant signal 'gpu_iso_enable'
Root cause: Synthesis optimizer removed the signal because it's driven to constant 0 in testbench configuration!
Step 2: Preserve Control Signals
# Preserve power control signals from optimization
set_dont_touch_network [get_pins power_controller/gpu_iso_enable]
set_dont_touch_network [get_pins power_controller/gpu_power_enable]
set_dont_touch_network [get_pins power_controller/cpu_save_enable]
set_dont_touch_network [get_pins power_controller/cpu_restore_enable]
# Alternative: Use UPF-aware synthesis flow
set_app_var power_preserve_rtl_hier_names true
Common Synthesis/Simulation Mismatches
| Issue | Simulation | Synthesis | Fix |
|---|---|---|---|
| Control signal optimized away | Signal exists | Signal removed | set_dont_touch_network |
| Domain instance renamed | Uses RTL name | Uses syn name | Update UPF or preserve names |
| Hierarchical path changed | RTL hierarchy | Flattened | Use dont_touch on modules |
| Library cells missing | Behavioral model | No matching cell | Add cells to library or remove strategy |
Using UPF Reports for Debug
Generate Comprehensive UPF Reports
# VCS: Generate detailed UPF reports
./simv \
+upf_dump_power_domain \ # Domain hierarchy
+upf_dump_supply_net \ # Supply network topology
+upf_dump_isolation > upf_isolation_report.txt \
+upf_dump_retention > upf_retention_report.txt \
+upf_dump_level_shifter > upf_level_shifter_report.txt
Example: Isolation Coverage Report
=============== ISOLATION REPORT ===============
Strategy: ISO_GPU
Domain: PD_GPU
Isolation signal: gpu_iso_enable (active HIGH)
Isolation supply: SS_TOP (VDD=1.8V, always-on)
Clamp value: 0
Location: parent
Applies to: outputs
Covered signals (47):
gpu_inst/dma_req[31:0] -> ISO_CELL_0 (noc_router/dma_controller)
gpu_inst/dma_addr[39:0] -> ISO_CELL_0 (noc_router/dma_controller)
gpu_inst/interrupt_req -> ISO_CELL_LH (interrupt_controller)
... 44 more ...
Cells inserted: 47
Total area: 2,340 um²
Total power: 1.2 mW
=============== END ISOLATION REPORT ===============
Use this report to verify:
- All expected signals are covered
- Isolation cells use correct supply (always-on)
- Cells placed in correct location (parent domain)
- Area/power overhead is reasonable
Common Beginner Mistakes
Mistake #1: Debugging without UPF tracing enabled
# WRONG: Basic simulation without tracing
./simv
# Hard to debug - no X-propagation paths, no power state info
# CORRECT: Enable comprehensive tracing
./simv +upf_trace_x +upf_dump_power_states +upf_verbose
Why? Without tracing, you must manually search waveforms for hours. Tracing automatically identifies X sources and propagation paths.
Mistake #2: Not checking UPF reports before simulation
# WRONG: Jump straight to simulation
vcs -upf buggy.upf design.sv
./simv # Fails with unclear errors
# CORRECT: Static check first
vcs -upf buggy.upf -upf_check_only design.sv # Finds errors immediately
# Fix UPF errors
vcs -upf fixed.upf design.sv
./simv
Why? Static checks run in seconds and catch basic errors. Simulation takes minutes to hours.
Mistake #3: Debugging one issue at a time (waterfall approach)
# INEFFICIENT: Fix one error, re-run, repeat
# Run 1: X-propagation error on signal A -> fix -> re-run
# Run 2: X-propagation error on signal B -> fix -> re-run
# Run 3: Retention error -> fix -> re-run
# ... 10 iterations ...
# EFFICIENT: Enable all checking, collect all errors
./simv +upf_iso_warn +upf_ret_warn +upf_trace_x +upf_verbose
# Fix ALL reported issues in one pass
# Re-run once to verify
Why? Waterfall debugging wastes time with repeated simulations. Comprehensive checking finds all issues in one run.
Summary
In this tutorial, you learned:
- Five-step debugging framework: Identify ? Localize ? Isolate ? Analyze ? Fix
- X-propagation debug: Use +upf_trace_x to find sources automatically, check isolation coverage with +upf_dump_isolation reports
- Retention debug: Verify always-on retention supply, check save/restore timing with waveforms (5-10 cycle hold times, 10-20 cycle gaps)
- Power state debug: Examine PST for legal transitions, fix FSM to follow required paths
- Synthesis correlation: Preserve control signals with set_dont_touch_network, match hierarchical paths, verify library cells
- UPF reports provide detailed coverage, area, and power information for verification
- Enable comprehensive tracing (+upf_trace_x, +upf_verbose, +upf_dump_*) to debug efficiently
