Accelerators and Hardware/Software Partitioning

An accelerator is a block of hardware built to do one kind of work much faster, or with much less energy, than software on a processor. Hardware/software partitioning is the decision about which parts of a product's work go into such blocks and which stay as software. The trade-off is fundamental: hardware is fast and efficient but fixed, while software is flexible but slower. This article explains why accelerators win, how to decide what to accelerate, how setup and data movement can cancel the gain, and how an accelerator is connected to the rest of the chip.

12 min read | Intermediate

What You'll Learn

• Explain where an accelerator's speed and energy advantage comes from
• Use profiling and Amdahl's law to choose what to accelerate
• Calculate the break-even point where offload starts to pay
• Describe the common ways to connect an accelerator to the system

Why Accelerators Win

A processor is general-purpose, and that generality has a cost. For every operation, it fetches an instruction, decodes it, reads registers, and decides where the result goes, as The Instruction Cycle describes. The arithmetic itself is a small part of the energy: studies of processors find that the overhead of instruction handling and data movement uses most of the energy of a simple operation such as an addition.

An accelerator removes that overhead. Its datapath is wired for one job, so there is no instruction to fetch. It can do many operations at once, because the work is known in advance. And it can keep data in small local memories next to the units that use it, rather than in a general register file. For a suitable task, these together give gains of 10 to over 100 times in performance per watt over a general processor, and even more for very regular work such as video decoding.

The price is flexibility. Once a chip is made, the accelerator does exactly what it was built for. If the standard changes, a new video codec or a new neural network layer, the hardware may not support it, and the work falls back to software.

A Spectrum of Choices

Partitioning isn't a choice between two extremes. The options run along a spectrum, each more efficient and less flexible than the one before:

1. Plain software on a general processor.
2. Software using special instructions, such as SIMD or vector extensions, as SIMD, Vectors and Multithreading explains.
3. Custom instructions added to the processor for one task, which RISC-V explicitly allows.
4. A programmable accelerator, such as a DSP, a GPU or a neural network engine, which runs its own programs but is built for one kind of work.
5. A fixed-function accelerator, which does exactly one job, such as a video decoder or a crypto block.

Moving down the list raises efficiency, development cost and risk, and lowers flexibility. Many products use several levels at once.

Choosing What to Accelerate

The first step is to profile: run the real software on a model or an existing chip, and measure where the time goes. Usually a few small kernels, such as filters, transforms, or encryption, take most of it. Those are the candidates.

Then apply Amdahl's law, from PPA Trade-offs and Amdahl's Law. If a kernel takes 60 percent of the run time and an accelerator makes it 20 times faster, the whole task becomes 1 / (0.4 + 0.6 / 20) = 2.3 times faster. An infinitely fast accelerator would give only 2.5. These numbers are assumptions for this example. The 40 percent left in software sets the limit. That is why a good partition often moves more than the obvious kernel, or speeds up the surrounding software too.

Good candidates share some features:

• a large share of time or energy;
• regular, predictable work on lots of data;
• a stable algorithm or standard that won't change over the product's life;
• a clear interface, with few decisions in the middle of the data flow.

Setup Cost and Break-Even

An accelerator doesn't start instantly. Software must prepare the data, write the configuration registers, start the engine, and later collect the result, often through an interrupt. If the data has to be copied into the accelerator's memory and back, that costs time too. For small jobs, this overhead can be larger than the work itself.

Here is a simple model. Software takes 50 cycles per item. The accelerator takes only 5 cycles per item, but needs 2,000 cycles to set up and finish each call. These numbers are assumptions for this example.

The lines cross at 2,000 / (50 - 5), about 44 items. Below that, software is faster. Above it, the accelerator wins, and for 1,000 items it takes 7,000 cycles against 50,000. A driver therefore often runs small jobs in software and sends large ones to the hardware. Designers try hard to lower the setup cost, with command queues, descriptor lists and shared memory, as DMA Engines describes for a related case.

Data Movement Can Dominate

An accelerator that computes 100 times faster is useless if it waits for data. If each item needs 16 bytes, and the accelerator processes one item per cycle at 1 GHz, it needs 16 GB/s of memory bandwidth, which may be most of what the chip's DRAM can deliver. Memory Bandwidth Budgets shows how to check that. Accelerators therefore often keep data in local SRAM, reuse it many times before writing results out, and process data as it streams from another block, rather than going through DRAM.

Connecting an Accelerator

There are several ways to attach an accelerator, and they trade speed against simplicity:

Most accelerators are memory-mapped devices. Software programs them through registers, as Memory-Mapped I/O and Registers explains. They read and write memory directly, through a manager port on the interconnect, and raise an interrupt when done. If the CPU has caches, the shared buffers must be kept coherent, by flushing caches in the driver, or through a coherent connection, as Multicore Integration and Coherent Interconnects describes.

Designing the Hardware/Software Interface

The interface between the driver and the accelerator decides how easy it is to use and how fast it can be. Good interfaces share a few features. They let software queue several jobs, so the hardware can move to the next one without waiting. They put large parameters in memory, as a descriptor, rather than in many registers. They report status and errors clearly, and they support being stopped and reset cleanly. And they are written down before the RTL, in the spec, because the software team and the hardware team must build to the same contract.

Accelerators can also be verified early with a software model. A model of the accelerator in C, run against the same tests as the RTL, lets driver writers start before hardware exists, and gives a reference to compare with the RTL. System Performance Modeling covers this kind of model.

A Worked Partition: A Camera Pipeline

Consider the path from a camera sensor to a compressed video file in a phone. A sensor delivers raw pixel data, which must be cleaned up (noise removal, color correction, lens correction), scaled, and then compressed by a video encoder. A typical partition looks like this:

• Image signal processor, in fixed-function hardware: the pixel cleanup runs on every pixel of every frame, tens of millions of pixels per second, with a stable algorithm. Software couldn't keep up within the power budget.
• Tuning parameters, in software: how much noise to remove, how to set exposure. These change with the scene, and the camera team improves them for years after the chip ships. The hardware has registers for them, and software updates them for each frame.
• Video encoder, in fixed-function hardware, for a standard codec, with a software driver that sets the bit rate and handles the files.
• New features, such as a new computational photography effect, start in software or on a programmable accelerator, and move into hardware in a later chip once they are proven.

The pattern is typical. The heavy, stable, regular work goes into hardware, while decisions, tuning and new ideas stay in software, and the interface between them is a set of registers and shared buffers. Data flows from block to block, often without passing through the CPU at all, which saves both time and memory bandwidth. When the frames do go through DRAM between stages, the bandwidth they use must be counted in the chip's memory budget, which is often what limits the camera's frame rate.

Common Mistakes