Pipelining and Outstanding Transactions

A memory may need several cycles to answer a read. If the master waits for every answer before asking again, the bus sits idle most of the time. Pipelining splits a transfer into an address phase and a data phase so that the next address can go out while the previous data is still coming back. Allowing several transactions to be outstanding takes the idea further. This article measures both on a memory with a latency of three cycles.

9 min read | Intermediate

What You'll Learn

• Explain latency and throughput on a bus
• Describe pipelined address and data phases
• Explain outstanding transactions and how many a master needs
• Measure the effect in a simulation

Latency and Throughput

Two numbers describe a bus slave. Latency is the time from a request to its answer. Throughput is how many requests it can finish per cycle. A memory with a latency of three cycles can still have a throughput of one read per cycle, if it accepts a new request every cycle while older ones are still in progress, just like an assembly line.

The memory model in this article does exactly that. Each request enters a three-stage pipeline and comes out three cycles later, in order:

module lat_mem #(
  parameter LAT = 3
) (
  input            clk,
  input            rst,
  input            req_valid,      // a read request is presented this cycle
  input      [7:0] req_addr,
  output           rsp_valid,      // read data returns LAT cycles later, in order
  output     [7:0] rsp_data
);
  reg [LAT-1:0] v;                 // one stage per cycle of latency
  reg [7:0]     a [0:LAT-1];
  integer       i;

  always @(posedge clk) begin
    if (rst) begin
      v <= 0;
    end
    else begin
      v[0] <= req_valid;
      a[0] <= req_addr;
      for (i = 1; i < LAT; i = i + 1) begin
        v[i] <= v[i-1];
        a[i] <= a[i-1];
      end
    end
  end

  assign rsp_valid = v[LAT-1];
  assign rsp_data  = a[LAT-1] ^ 8'hA0;   // stand-in for memory contents
endmodule

Address and Data Phases

A pipelined bus separates every transfer into two phases. In the address phase, the master presents the address and control. In the data phase, which follows later, the data moves. Because the phases use different signals, the address phase of the next transfer can overlap the data phase of the current one. AHB has exactly this structure, with a data phase one cycle behind the address phase. AXI goes further and uses entirely separate channels for addresses and data.

Outstanding Transactions

A transaction is outstanding from the moment its request is accepted until its response returns. A master that allows only one outstanding transaction waits for each response before the next request. A master that allows several keeps issuing requests while earlier ones are still in flight. The master model limits itself to MAX_OUT transactions in flight:

wire [3:0] outstanding = issued - done;

assign req_valid = (issued < N) && (outstanding < MAX_OUT);

Measuring It

The testbench does eight reads from the 3-cycle memory three times, with 1, 2 and 4 reads allowed in flight:

1 read in flight:  8 reads took 32 cycles
2 reads in flight: 8 reads took 17 cycles
4 reads in flight: 8 reads took 11 cycles

With one read in flight, every read costs four cycles: one to issue, three of latency, and the next request only after the response. With four in flight, a request goes out in every cycle, and eight reads take eight cycles of requests plus the three-cycle latency of the last one. The memory was the same in all three runs; only the master's willingness to have reads in flight changed.

The rule of thumb: to keep a pipelined slave busy, a master needs at least as many outstanding transactions as the round-trip time in cycles. Here, the memory answers three cycles after a request, and the master can react one cycle after that, so four in flight are enough and more would not help.

Running It in ChipVerify Lab

The example runs the three masters side by side, each with its own copy of the memory. Run it and compare the request and response patterns in the waveform.

Common Mistakes