Memory Controllers and Scheduling

A memory controller sits between the SoC and its DRAM. It receives read and write requests from the interconnect, turns them into DRAM commands, obeys dozens of timing rules, refreshes the memory, and chooses the order in which to serve requests. That last decision, scheduling, can change the useful bandwidth of the same memory by a factor of several. This article explains what a controller does, how page policies and scheduling work, shows with a model how much scheduling matters, and covers the controller's other duties.

11 min read | Intermediate

What You'll Learn

• Describe the parts of a memory controller and what each does
• Compare open-page and closed-page policies
• Explain FCFS and FR-FCFS scheduling and their effect on bandwidth
• Describe how controllers handle writes, refresh, fairness and power

What a Memory Controller Does

DRAM Organization explained that a DRAM access is not a single step: a row must be activated, a column read, the row eventually precharged, and every command must respect minimum times. Managers on the interconnect don't want to know any of this. They send "read 64 bytes at address X". The memory controller does the translation:

• It accepts requests from the interconnect, often from several ports, and puts them in queues.
• It maps each address to a channel, rank, bank, row and column.
• It tracks the state of every bank: which row is open, and when each command is next allowed.
• Each cycle, a scheduler picks which command to issue next.
• It issues refresh commands on time, and manages power modes.
• It returns the data and the responses to the right managers, in the order the protocol requires.

Behind the controller is the PHY, which drives and samples the pins at gigatransfers per second. At power-up, the PHY and controller run training: they measure the delays on each wire, and adjust their timing so that every bit is sampled at the right moment. Training is often the hardest part of bringing up a new board.

Page Policy

After a column access, should the controller leave the row open or close it? This is the page policy.

• Open page: leave the row open, betting that the next access to this bank is to the same row. A hit then costs only the column access. If the bet fails, the controller must precharge first, which costs more than if the row had been closed.
• Closed page: precharge right after each access. Every access then costs an activate plus a column access, but never a conflict.

Open page wins when accesses have locality, such as streams and cache line fills of nearby data. Closed page wins when accesses are random, such as many independent managers in a server. Many controllers are adaptive: they keep rows open for a while, and close them after a timeout, or when a bank's recent history suggests that hits are unlikely.

Scheduling

Requests arrive in some order, but the controller doesn't have to serve them in that order. The simplest scheduler, FCFS (first come, first served), serves the oldest request first. It is fair and simple, but it wastes time: an old request to a closed row forces a precharge and activate, while a younger request to an already open row waits.

FR-FCFS (first ready, first come, first served) is the classic improvement, used in some form by nearly every modern controller. It prefers requests that hit an open row, and only among those does it pick the oldest. Hits are served quickly, and rows stay open as long as they are useful.

A Model

A model of one DRAM channel shows the effect. It has 8 banks, 2 KB rows and timings of 14 cycles each for tRP, tRCD and CL, with 4-cycle data bursts and a queue of 16 requests. These values are assumptions for this example. Three access patterns were run through it with each scheduler:

For one sequential stream, nearly all accesses hit the open row, about 97 percent, and the data bus is busy about 84 percent of the time with either scheduler. There is nothing to reorder. For random addresses, there are no row hits at all, and utilization falls to about 39 percent. Scheduling can't help, because there are no hits to prefer. The bank parallelism is what keeps it from being worse.

The interesting case is two streams interleaved, for example two managers each reading their own buffer, and mapped to the same banks. In arrival order, every request alternates between two rows, so each one closes the row the other needs: zero hits, and only about 13 percent utilization. FR-FCFS looks ahead in the queue, groups the requests by row, and brings the row hit rate to about 94 percent and utilization to about 76 percent, nearly six times better. This pattern, several managers each with good locality on their own, is exactly what an SoC creates, which is why every serious controller reorders.

The Price of Reordering

Reordering has costs. A request that keeps missing can be passed over again and again, which is a form of starvation. So controllers add aging: once a request has waited long enough, it goes first regardless. They also respect the QoS fields of requests, so that the display or CPU can be served ahead of bulk traffic, as Arbitration and Quality of Service describes. And they must keep the right order where it matters: two accesses to the same address must not be swapped, and responses must return in the order the protocol requires.

Writes

Writes are handled differently from reads. The data bus must turn around between reads and writes, which costs several cycles each time. So controllers collect writes in a write queue, and serve reads first, since processors wait for reads, but not for writes. When the write queue fills past a high-water mark, the controller switches to write drain mode, and writes a batch at once, until the queue falls below a low-water mark. This keeps turnarounds rare. A read to an address that is still waiting in the write queue must get the new data from the queue, not stale data from DRAM.

Refresh and Power

The controller must refresh every row on schedule. It can delay refreshes a little when it is busy and catch up later, within limits the standard allows. Some DRAM types can refresh one bank while others stay in use, which hides much of the cost.

The controller also manages power-down modes. When a rank is idle, it can be put into a low-power state, which saves energy but costs a delay to wake up. In self-refresh, the DRAM refreshes itself while the controller and PHY sleep, which is how a phone keeps its memory alive in standby. Choosing when to enter these modes is a trade between power and latency, much like the trade in Power Domains and Power Management.

Measuring a Controller

Controllers are judged on sustained bandwidth for realistic mixes of traffic, average and worst-case latency per class of client, and power. Benchmarks such as STREAM measure sustained bandwidth on simple patterns. A system architect also measures the latency the CPU sees when the GPU and display are busy, which is often much higher than when the system is idle. These figures feed the bandwidth budget in Memory Bandwidth Budgets.

Latency under load deserves its own measurement. As utilization rises toward the controller's limit, queues grow and each request waits longer, so the latency a processor sees can double or triple on a busy system. A controller that keeps latency low for the CPU while the GPU streams at full rate is worth more than one with a slightly higher peak bandwidth.

Several Channels

An SoC with several DRAM channels has a controller per channel, or one controller managing several. The address mapping decides which channel each request goes to, and a good mapping spreads traffic evenly. Mappings usually interleave channels at a fine granularity, such as every 256 bytes or every few cache lines, so that even a single stream uses all the channels at once. Coarse interleaving, such as one channel per gigabyte, would let one busy buffer overload one channel while the others sit idle. Some systems keep the option of mapping specific regions to specific channels, for example to isolate a real-time client from interference.

Verifying a Controller

A memory controller is checked in several ways. A model of the DRAM, which flags any violation of the timing rules, runs alongside it in simulation, so that every command sequence is checked against the standard. Random traffic from several managers, with random QoS and address patterns, looks for ordering bugs and starvation. Data checking compares every read with what was last written. And on silicon, long stress tests with training at different temperatures and voltages make sure the PHY's timing margins hold, because memory errors in the field are among the hardest problems to diagnose.

Common Mistakes