DMA Engines

A DMA engine, short for direct memory access, is a small block that copies data between memories and devices by itself, so the processor doesn't have to move every word. The processor tells it what to move and where, then goes on with other work, and the engine raises an interrupt when it is done. Almost every SoC has several. This article explains what a DMA engine contains, how descriptor chains work, how it saves processor time, the pitfalls of caches and ordering, and how to decide when to use one.

11 min read | Intermediate

What You'll Learn

• Describe the parts of a DMA engine and how software programs it
• Explain descriptor chains and scatter-gather transfers
• Estimate how much processor time DMA saves
• Handle cache coherence and ordering when using DMA

Why DMA Exists

Moving data with the processor is expensive. Each word needs a load and a store, plus loop instructions, and each access may wait on the bus. Worse, when data comes from a slow device, the processor spends most of its time waiting. Polling, Interrupts and DMA measured this: in one small SoC sending a 256-byte message to a UART by polling, 72 percent of the instructions executed were the polling loop itself, and the processor could do nothing else.

A DMA engine moves the data instead. It is a manager on the interconnect, like the processor, but it does only one thing: read from a source address, write to a destination address, and repeat. Because it is built for that, it can keep many transfers in flight and use long bursts, as Outstanding Transactions and Bursts explains, so it often moves data faster than the processor could, as well as freeing the processor for other work.

What It Contains

A simple engine has:

• Registers for the source address, the destination address, the length, and a control register to start it and select options, plus a status register.
• A read port and a write port, or one port used for both, as a manager on the interconnect.
• A small buffer, so data read can wait until it is written, letting reads and writes overlap.
• An interrupt output, raised when a transfer finishes or fails.
• Often several channels, each an independent set of registers, sharing the ports through an arbiter.

Software sets the registers and writes "start". The engine reads from the source in bursts, fills its buffer, writes to the destination, updates the addresses and the count, and repeats until the length reaches zero. Then it sets a status bit and raises its interrupt.

Addressing Modes

A copy between two memories increments both addresses. A transfer from memory to a peripheral, such as a UART's transmit register, increments the source and keeps the destination fixed, since every byte goes to the same register. Engines also support strides, to copy a rectangle out of an image, and different widths on each side, such as 32-bit reads and 8-bit writes.

Handshaking with Peripherals

A peripheral can't accept data faster than it sends it. So a DMA engine serving a peripheral waits for a request signal from it: the UART raises its request when its FIFO has room, and the engine sends one chunk each time. This handshake replaces the processor's polling loop with a wire.

Descriptor Chains

Setting registers for every transfer still costs processor time, and real data is often not in one piece: a network packet may be spread over several buffers, and a video frame may be stored in tiles. Scatter-gather DMA handles this with descriptors: small structures in memory, each describing one transfer, with a pointer to the next one.

struct dma_desc {
    u32 src;        // source address
    u32 dst;        // destination address
    u32 len;        // bytes to move
    u32 ctrl;       // options: interrupt when done, increment modes
    u32 next;       // address of the next descriptor, or 0 to stop
};

Software builds the chain, writes the address of the first descriptor to the engine, and starts it. The engine fetches each descriptor, does its transfer, and follows next until it reaches 0. One start can move dozens of pieces. Network and storage controllers use rings of descriptors that software keeps refilling, so the hardware never stops. This is how high-speed devices move data with very little processor involvement.

What It Saves

Take a peripheral that must send 1,000 bytes. With polling, the processor spends the whole transfer time on it. With interrupts but no DMA, the processor runs a short handler every time the FIFO needs refilling, say every 8 bytes: 125 interrupts. With DMA, the processor spends a few hundred cycles setting it up, and one interrupt at the end.

Some numbers make that concrete. Assume each interrupt costs 60 cycles of overhead plus 8 bytes x 15 cycles of work, and the DMA setup and completion together cost 400 cycles. Interrupt-driven: 125 x (60 + 120) = 22,500 cycles. DMA: about 400 cycles. These numbers are assumptions for this example. DMA uses about one fiftieth of the processor time, and the processor can sleep or do useful work meanwhile. For large or fast transfers, the difference is decisive, which is why every high-speed peripheral has DMA.

DMA isn't free. It costs area, a share of the interconnect's bandwidth, and software complexity: drivers must build descriptors, handle completion and errors, and manage buffers. For a few bytes, setting up DMA costs more than copying them, the same break-even idea as for accelerators in Accelerators and Hardware/Software Partitioning.

Caches and DMA

DMA creates the most common bug in driver code. If the processor has a write-back data cache, the newest data may be in the cache, not in memory. A DMA engine reading memory then sees old data. The reverse is just as bad: after DMA writes new data into memory, the processor's cache may still hold old copies of those addresses and return them. Replacement and Write Policies explains why.

There are three standard solutions:

• Cache maintenance: before DMA reads a buffer, the driver cleans the cache for that range, writing dirty lines to memory. After DMA writes a buffer, the driver invalidates the cached copies before reading it.
• Uncached buffers: buffers shared with DMA are placed in memory marked as not cacheable. This is simple, but processor access to them is slow.
• Coherent DMA: the engine's accesses snoop the caches, as Multicore Integration and Coherent Interconnects explains, so no maintenance is needed. This costs hardware but removes a class of bugs.

Ordering and Completion

When the interrupt arrives, is the data really in memory? It depends. If writes are posted, buffered on the way, the engine may report completion while the last data is still in flight. Engines that are used correctly only report completion once their writes are acknowledged by the destination. Likewise, software must make sure that its writes to a descriptor are visible in memory before it starts the engine, which may need a memory barrier. Getting this wrong gives rare, timing-dependent corruption, which is one of the hardest kinds of bug to find.

Errors need handling too. If a DMA access hits an address that belongs to nobody, the interconnect returns an error response, as Buses and Transactions describes. The engine must stop, record the failing address, and tell software, rather than carrying on and corrupting memory.

DMA and Virtual Memory

On a system with an operating system, programs use virtual addresses, translated by the processor's MMU, as Why Virtual Memory explains. A DMA engine without translation uses physical addresses. That creates two problems. A buffer that is contiguous in virtual memory is usually scattered over many physical pages, so the driver must build a descriptor for each piece. And a DMA engine can reach any physical address, so a buggy or malicious device could overwrite the kernel.

An IOMMU, an MMU for devices, solves both. It sits between DMA-capable devices and memory, and translates the addresses they use through page tables set up by the operating system. A device then sees a contiguous buffer, and it can reach only the pages it has been given. IOMMUs are standard in servers, laptops and phones, and they are essential when devices are given to virtual machines. RISC-V defines a standard IOMMU specification for the same purpose.

Devices that don't need the full range of memory, or that run in a secure area, can also be limited by simpler hardware: a memory protection unit at the device's port that only lets it reach listed address ranges.

Translation adds latency to each DMA access, so IOMMUs have their own TLBs, and device drivers try to reuse mapped buffers rather than mapping new ones for each transfer.

Common Mistakes