Integrating the CPU Subsystem

A CPU subsystem is the processor core together with everything it needs close by: fast local memory, the logic that sends each access to the right place, the bus port to the rest of the chip, interrupt inputs and a debug connection. Integrating it well matters, because nearly everything the chip does starts with an instruction, and every cycle the core waits for memory or the bus is lost. This article explains the parts of a CPU subsystem, how an access is routed, what the bus port decides about performance, how interrupts and debug are connected, and what goes wrong at these boundaries.

11 min read | Beginner to Intermediate

What You'll Learn

• Name the parts of a CPU subsystem and what each does
• Explain how an address decides between local memory and the bus
• Estimate how the bus port's latency and outstanding transactions affect the core
• Describe how interrupts, debug and reset connect to the core

What a CPU Subsystem Is

A processor core on its own is a pipeline with an instruction port and a data port, as The Datapath explains. To be useful in an SoC, it needs more around it. Vendors usually deliver cores as a complete subsystem, sometimes called a core complex or a CPU cluster, with these parts already connected:

• The core itself.
• Local memory: a tightly coupled memory (TCM), caches, or both.
• An address decoder that sends each access to the local memory or out to the bus.
• A bus port, or manager port, that speaks the chip's interconnect protocol.
• Interrupt inputs, and often a small local interrupt controller and timer.
• A debug module, reached through JTAG or a similar port.
• Clock and reset inputs, and sometimes power control.

Routing Each Access

Every load, store and instruction fetch carries an address. A small decoder inside the subsystem looks at the high bits. If the address falls in the TCM range, the access goes to the TCM, which answers in one cycle. Otherwise it leaves the subsystem through the bus port and crosses the interconnect, as Memory Maps and Address Decoding explains.

In one small example SoC, a TCM access costs 1 cycle, an access to the on-chip SRAM across the bus costs 4, and an access to a peripheral behind a slower bus bridge costs 6. Those differences show up directly in run time. A test program that copied a buffer ran about 22 percent slower when the destination was in the SRAM rather than the TCM, with almost the same instructions.

This is why the first integration decision is what goes where. Code, the stack and data used in tight loops belong in the TCM or behind a cache. Large buffers and data shared with other blocks go further out. The linker script, described in Assembling, Linking and Loading, puts that decision into effect.

Instruction and Data Paths

A pipelined core fetches an instruction and accesses data in the same cycle, so the local memory needs two ports, or the two paths need separate memories. If both paths had to share one single-port memory, every load or store would stall a fetch. A TCM with one read-write port for data and one read port for instructions is a common answer, and Memories in a System describes that organization.

The Bus Port

The bus port decides how well the core can use the rest of the chip. Two properties matter most.

The first is latency: how many cycles a request takes to come back. Each stage of the interconnect, each clock crossing and each bridge adds to it. A simple in-order core waits for every load, so its CPI rises with each cycle of latency.

The second is how many transactions can be outstanding, that is, in flight at once. A simple port waits for each response before sending the next request. In one small SoC with such a port, a test program spent over a third of its run time with the core waiting on the bus. A port that allows several outstanding transactions can overlap them, but only if the core can issue another request while one is pending. A simple in-order core usually can't, which is one reason large cores have non-blocking caches. Outstanding Transactions and Bursts works through the numbers.

The protocol matters too. A simple protocol such as AXI4-Lite, or APB for peripherals, carries one word per transaction. A full protocol such as AXI carries bursts, so a cache line can be filled with one request. A core with caches needs a port that supports bursts, or every line fill turns into many slow single transfers.

Caches or TCM?

A small embedded core often has a TCM and no caches: its timing is fully predictable, and it is cheaper. A larger core uses caches to reach external memory, and may also have a TCM for code that must run with known timing, such as an interrupt handler. If a subsystem has caches, they create work at integration: DMA buffers must be handled carefully, as DMA Engines explains, unless the interconnect keeps the caches coherent.

Interrupts

A core has a small number of interrupt inputs. In RISC-V, a core has a machine external interrupt, a timer interrupt and a software interrupt, as Interrupts and Interrupt Latency describes. An SoC with dozens of interrupt sources combines them in an interrupt controller, which drives the core's external interrupt line. Interrupt Controllers explains that block.

Interrupt lines often cross clock domains, because devices may run on their own clocks. Each must be synchronized into the core's clock domain, as Clock Architecture and Clock Domain Crossings describes. The integrator also decides whether each line is level-sensitive or edge-triggered, which must match what the device and the controller expect.

Debug

A chip that can't be debugged can't be brought up. Most cores include a debug module that can halt the core, read and write registers and memory, set breakpoints and single-step. RISC-V defines a standard debug specification, so one debugger works with many cores. The debug module is usually reached through a JTAG port at the pins, or a two-wire variant, and in larger chips through a debug network that reaches every core.

Debug has security implications. A debugger can read anything, so production chips often lock the debug port, or require a key to unlock it. That must be decided at integration, as Security Architecture discusses.

Reset and Boot

At reset, the core starts fetching from a fixed reset vector. The integrator decides what is there: a boot ROM, the start of flash, or a TCM loaded by other means. Some cores take the reset vector as an input pin, set by the SoC, which makes this choice a wiring decision. Boot ROM and the Boot Flow covers the rest. Reset must also be released in the right order: the memories and bus the core needs must be out of reset before the core makes its first fetch.

Configuration Choices

A licensed core usually comes with many options to set at integration time. Typical ones include:

• which instruction set extensions to include, such as multiply, floating point or compressed instructions;
• TCM and cache sizes;
• the number of interrupt lines and priority levels;
• debug features, such as the number of hardware breakpoints and trace;
• the bus protocol, width and number of outstanding transactions;
• whether there is a memory protection unit or an MMU.

Each one affects area and power, and some affect software. Compiling software for an extension the core lacks makes it fail, as What an ISA Defines explains. So the configuration is recorded in the spec, and the software build is set up to match.

More Than One Core

Many SoCs have several CPU subsystems, each with a different job: a large application cluster, a small real-time core for control, a tiny core in the always-on domain to manage power. Each is integrated the same way, but they must also cooperate. They need a way to send each other messages, usually a small hardware mailbox that raises an interrupt on the other side. They need agreed rules about which memory each may use. And at boot, one core usually starts first and releases the others. Multicore Integration and Coherent Interconnects covers cores that share data through caches.

Common Mistakes