SIMD, Vectors and Multithreading

A processor can do more work per cycle in other ways than finding parallelism between ordinary instructions. SIMD and vector instructions apply one operation to many values at once, which suits graphics, audio, machine learning and any work over arrays. Simultaneous multithreading lets one core run two programs at the same time, filling slots that would otherwise be empty. This article explains how each works, shows a real compiled SIMD loop, compares fixed SIMD with RISC-V's vector approach, and looks at when these features pay off.

12 min read | Intermediate

What You'll Learn

• Explain how SIMD processes several values with one instruction
• Read a real auto-vectorized loop
• Compare fixed-width SIMD with vector-length-agnostic designs
• Explain how simultaneous multithreading shares a core

Data Parallelism

Superscalar Issue looked for independent instructions in ordinary code. There is often a better opportunity. Many programs do the same thing to many values: add two arrays, brighten every pixel, multiply a large matrix. This is data parallelism, and it needs only one instruction telling many adders what to do.

SIMD means single instruction, multiple data. A SIMD register holds several values, called lanes, packed side by side, and one instruction works on all of them:

A 128-bit register can hold four 32-bit integers, or eight 16-bit ones, or sixteen bytes. A single paddd instruction on x86 adds four pairs of 32-bit integers in one go. Compared with four separate adds, it takes one fetch, one decode and one issue slot, so the control work is shared. That is why SIMD gives so much performance per watt: the energy of an operation goes mostly into controlling it rather than into the arithmetic itself.

A Real Compiled Loop

Here is a small C loop that fills an array with a[i] = 3 * i + 1, compiled by GCC 13 at -O2 for x86-64, where SSE2 SIMD is always available. This is the loop the compiler produced:

30:  movdqa %xmm1,%xmm2         # copy the four values of i
34:  add    $0x10,%rax          # move the pointer on by 16 bytes
38:  paddd  %xmm4,%xmm1         # the next four values of i
3c:  movdqa %xmm2,%xmm0
40:  pslld  $0x1,%xmm0          # 2 x i, in each lane
45:  paddd  %xmm2,%xmm0         # 3 x i
49:  paddd  %xmm3,%xmm0         # 3 x i + 1
4d:  movaps %xmm0,-0x10(%rax)   # store all four results
51:  cmp    %rax,%rcx
54:  jne    30

The compiler rewrote the loop to compute four elements per pass, using 128-bit xmm registers. This is auto-vectorization, and it is one of the optimizations mentioned in Compiler Optimization Levels. The programmer wrote nothing special. For the compiler to do this, it must prove that the elements are independent. If two pointers could overlap, or there is a branch inside the loop, or the elements are not in a neat row in memory, it may give up. Then programmers can use intrinsics, C functions that map directly to SIMD instructions, or hand-written assembly. Very fast libraries, such as those for video and cryptography, usually do.

What Holds SIMD Back

Several things limit the gain:

• Alignment and edges. The array may not start on a boundary, and its length may not be a multiple of four, so the compiler has to add extra code before and after the main loop.
• Non-adjacent data. An array of structures has its fields spread out, while SIMD wants all the x values together. Rearranging the data can matter more than anything else.
• Branches. A condition inside the loop must become a mask: compute both results, and let the mask choose per lane.
• Amdahl's law. Only the vectorizable part speeds up, as PPA Trade-offs and Amdahl's Law explains.
• Memory bandwidth. A loop running four times faster needs four times the data, and memory may not deliver it.

Fixed Width and Vectors

x86 has grown its SIMD in steps: MMX at 64 bits, SSE at 128, AVX at 256, AVX-512 at 512. Each step added new instructions and new registers, and code must be rewritten or recompiled to use the width. Arm's NEON is 128 bits. This creates a problem: a program built for one width can't use a wider one, and instruction sets fill up with similar instructions.

The older idea of a vector processor, going back to Cray supercomputers in the 1970s, solves this. The program does not state the hardware's width. In the RISC-V Vector extension (RVV), a loop asks the hardware how many elements it will handle this time, with the vsetvli instruction, processes that many, and repeats:

loop:
    vsetvli t0, a2, e32, m1, ta, ma   # t0 = elements this pass, up to a2
    vle32.v v0, (a0)                  # load t0 elements from a[]
    vadd.vx v0, v0, a3                # add a scalar to each
    vse32.v v0, (a1)                  # store t0 elements to b[]
    slli    t1, t0, 2                 # bytes done
    add     a0, a0, t1                # advance the pointers
    add     a1, a1, t1
    sub     a2, a2, t0                # elements left
    bnez    a2, loop

The same code runs on a core with 128-bit vectors and on one with 1,024-bit vectors. It is vector-length agnostic. The last pass needs no special code for the leftovers, since vsetvli simply returns a smaller number. Arm's SVE takes a similar approach. A small core can implement vectors with short registers, which saves area, and the same software will run faster on a larger one.

GPUs and Accelerators

Graphics processors take data parallelism further. A GPU has thousands of simple lanes, built for workloads that are almost entirely parallel. Machine learning accelerators are specialized further, as matrix multipliers. For work with huge amounts of data parallelism, those beat a CPU by a wide margin in performance per watt. A CPU's SIMD unit is for the moderate cases, close to the data, with no need to hand work to another device.

Inside a SIMD Unit

A SIMD unit is less exotic than it sounds. Its 128-bit adder is mostly four 32-bit adders side by side, with the carry between them cut. The same hardware can act as eight 16-bit adders, just by cutting the carries in more places, which is how one unit serves many element sizes. Add a register file of wide registers, a load path as wide as a register, some shuffle logic to move lanes around, and multipliers for each lane. The cost is mostly the wide register file and the wider data path to memory. Its small control logic is shared with the rest of the core, because a SIMD instruction goes through the same fetch, decode and rename as any other.

Width is a trade-off like any other. Going from 128 to 256 bits doubles the data path, and it only helps code that vectorizes well. Many designs instead keep the registers wide, and run the operations in two halves on narrower hardware, which saves area and power but gives no speed gain. RISC-V's vector extension makes this explicit: the register length and the width of the data path are separate choices. An implementation can have long registers and a narrow data path, taking several cycles per instruction, and software does not need to change. That is a good fit for small cores, which get the benefit of compact loops at little cost.

Simultaneous Multithreading

Now a different idea. An out-of-order core with a width of 4 often uses only 1 or 2 of its slots, because the current program has no more ready work, for example when it is waiting for memory. Those empty slots are wasted hardware. Simultaneous multithreading (SMT) fills them with instructions from a second program.

The core keeps two sets of architectural state: program counter, register map and CSRs, one for each hardware thread. To the operating system, it looks like two processors. In RISC-V these are called harts, hardware threads. The fetch stage alternates between threads, and everything else is shared: the execution units, caches, predictors and queues. The issue logic does not care which thread an instruction came from, since renaming keeps their registers apart.

The hardware cost is small: a few percent of the core's area. The gain depends on the workload. When Intel introduced it, the company quoted gains of up to 30 percent on well-threaded programs, and independent tests often measured around 20 percent or less. On a program that already uses the core fully, it may give nothing, and it can even slow each thread down, because they fight over caches and predictors. SMT raises throughput, not single-thread speed, and each thread alone usually runs slower than if it had the core to itself.

SMT also brings a security concern. Two threads sharing caches and predictors may be able to observe each other, which matters if they belong to different users. Some cloud providers avoid sharing a core between customers for this reason.

Other Kinds of Multithreading

Simpler designs share a core between threads over time. Fine-grained multithreading switches threads each cycle. Coarse-grained multithreading switches only on a long stall, such as a cache miss. An in-order core with a few threads can stay busy during misses with little extra hardware, which suits network processors and some throughput-oriented embedded designs.

Flynn's Taxonomy

A classic classification from 1966 sorts machines by how many instruction and data streams they have. SISD, single instruction and single data, is a plain processor. SIMD is the case in this article. MIMD, multiple instruction and multiple data, is several processors, or threads, each doing its own thing, which is covered in Multicore Processors and Shared Memory. A modern chip mixes them all.

The two features also meet inside a real core: each hardware thread can issue SIMD instructions, and the threads then compete for the same wide execution units.

Common Mistakes