Arbitration and Quality of Service

Arbitration is the decision, made thousands of times per microsecond inside an SoC, about which of several requesting blocks gets a shared resource next. Quality of service, usually shortened to QoS, is the set of mechanisms that make sure the blocks that need it most get enough bandwidth or a short enough wait. A display that misses its deadline shows a glitch on the screen, while a background copy that waits a little longer harms nobody. This article explains the common arbitration policies, measures how they treat three competing blocks, and explains how real chips manage priority and bandwidth.

11 min read | Intermediate

What You'll Learn

• Compare fixed priority, round robin and weighted arbitration
• Explain starvation and how to prevent it
• Distinguish latency-sensitive, bandwidth-sensitive and best-effort traffic
• Describe how QoS mechanisms work across an interconnect and memory controller

Why Arbitration Exists

Whenever two managers want the same thing in the same cycle, such as a shared bus, one port of a crossbar, or the memory controller, only one can have it. An arbiter picks. Buses, Crossbars and Networks-on-Chip showed that even a crossbar must arbitrate when two managers target the same subordinate, and DRAM is shared by nearly everyone. The arbiter's policy decides who waits, and for how long.

A good policy must balance several goals: keep the resource busy, give each manager a fair share or the share it needs, keep the wait short for urgent traffic, and never let any manager wait forever, which is called starvation.

The Basic Policies

Fixed Priority

Each manager has a fixed rank, and the highest-ranked requester always wins. It is the simplest arbiter, just a priority encoder, and it gives the top manager the shortest possible wait. Its weakness is starvation: if the high-priority managers are busy enough, a low-priority one may never get through.

Round Robin

Managers take turns. After a manager is served, it goes to the back of the line. Every requester is served within a bounded number of turns, so none starves, and busy managers share fairly. It costs a little more logic, a pointer to the last winner.

Weighted Round Robin

Round robin, but each manager gets a number of turns per round, its weight. A manager with weight 2 gets twice as many grants as one with weight 1 when both are busy. This divides the bandwidth in chosen proportions, without starving anyone.

Measuring the Policies

A model compares the three. A CPU and a DMA engine each request a shared target on 35 percent of cycles, and a display controller on 25 percent, so together they ask for 95 percent of the target's capacity. These numbers are assumptions for this example. The target serves one request per cycle, and the model ran for 200,000 cycles:

With fixed priority, ranking the CPU first, the CPU never waits, the DMA engine waits about one cycle on average, and the display waits about 21 cycles on average, with 1 in 100 requests waiting over 100 cycles. All three are still served, because the total load is below capacity, but the lowest-priority manager absorbs all the waiting.

With round robin, the waits even out: about 7 cycles for the CPU and DMA, and about 2.6 for the display, which asks less often and so rarely finds itself behind a long queue. With weighted round robin, giving the display twice the weight, its average wait drops to well under a cycle, and the others wait slightly longer.

No policy is best overall. The right one depends on what each manager needs. Here the display's needs, a steady stream that must never fall behind, are served best by giving it weight or priority, and accepting a slightly longer wait for the others.

Kinds of Traffic

To set policies, architects sort the traffic into classes, according to what each needs:

A display controller is the classic hard real-time case. It reads a frame from memory to send to the screen at a fixed rate. It has a buffer, but if memory falls behind long enough to empty that buffer, the screen shows garbage: an underrun. So the display must be guaranteed its bandwidth even when the GPU is busiest. The CPU needs something different: a short wait for each miss, because the core stalls while it waits, as Integrating the CPU Subsystem explains.

QoS Mechanisms

Real chips go beyond a single arbiter's policy. Common mechanisms include:

• Priority values on transactions. AXI carries a QoS field with each transaction, a 4-bit value that tells arbiters along the path how urgent it is.
• Dynamic priority. A display controller raises its priority as its buffer empties, and lowers it when the buffer is full. It is low priority most of the time, and urgent only when needed.
• Rate limiting. A regulator at a manager's port limits its bandwidth, so a busy GPU can't crowd out everyone else. A common form is a token bucket: the manager earns tokens at a set rate, and spends one per transaction.
• Separate paths. Real-time traffic gets its own queues or virtual channels in the interconnect, so it is never stuck behind bulk traffic.
• Memory controller scheduling. The DRAM controller is the most contended point, and it uses QoS values when choosing what to serve next, as Memory Controllers and Scheduling explains.

QoS only works end to end. A display given top priority at the interconnect, but treated as ordinary at the memory controller, can still underrun. Architects therefore define a policy for every arbitration point along each critical path.

Designing the Arbiter

An arbiter must make its decision quickly, usually within one clock cycle, because it sits on the path of every transaction. A fixed-priority arbiter for a few requesters is a handful of gates. Round robin adds a pointer and some rotation logic. Weighted arbiters and those that compare QoS values need counters and comparators. For many requesters, arbiters are built as trees of smaller ones, which keeps the logic fast.

Arbiters also decide at what granularity to switch. Switching after every beat of a burst gives the fairest sharing, but costs efficiency. Switching only between bursts is simpler, and keeps bursts intact, but a long burst can block others. Interconnects often limit burst length partly for this reason.

Locks and Exclusive Access

Some transactions must not be interleaved with others. An atomic read-modify-write needs exclusive access to a location. Old protocols allowed a manager to lock the bus, which blocked everyone and hurt QoS badly. Modern protocols use exclusive access monitors instead: the read and write are separate, and the write only succeeds if nobody else wrote in between, much like RISC-V's load-reserved and store-conditional in Memory Consistency, Atomics and Fences. Nobody is blocked while it happens.

Aging: Raising Priority Over Time

A simple way to keep fixed priority without starvation is aging. Each waiting request carries a counter that increases every cycle it waits. Once the counter passes a threshold, the request is treated as high priority, whatever its normal rank. Urgent traffic still goes first almost all the time, but no request can wait longer than the threshold plus a little. Aging is common in memory controllers, where a request for a closed DRAM row can otherwise be passed over again and again by requests that hit the open row, as Memory Controllers and Scheduling explains.

Aging needs care with its threshold. Set too low, it turns every request urgent and undoes the priority scheme. Set too high, it allows long waits before it acts. Designers usually choose it from the longest wait each client can tolerate.

Checking That QoS Works

A QoS design is verified by measurement, under the worst load. Engineers build traffic generators that imitate each client, such as a display reading at its exact rate or a GPU issuing as much as it can, and run them together in simulation or emulation for long periods. They record the distribution of each client's wait, not just the average, and check the worst cases against the deadlines: the display buffer must never empty, and the audio stream must never miss a sample. The model in this article reports both the average and the 99th percentile wait for this reason, since a client that is fine on average can still miss deadlines in its slowest moments.

Common Mistakes