How data moves inside an AI cluster

An AI training job isn’t running on one chip, or even one rack — it’s spread across hundreds or thousands of GPUs that have to exchange data with each other constantly, sometimes many times per second, to stay in sync. How that data moves, and over what physical medium, is a big part of why an AI cluster looks so different from a normal data center, and why some of the network gear inside it exists at all.

Two different networks, not one

It’s easy to think of “the network” as a single thing, but an AI cluster actually runs two networks with different jobs.

scale-up fabric links GPUs to each other inside a single rack or pod — this is the tightest, fastest connection in the system, built for the kind of constant, low-latency exchange a training job needs between GPUs working on the same problem. As models split across more GPUs per job, this fabric needs more bandwidth and has to reach further within the pod than earlier generations required.

scale-out switch asics is a separate job: routing traffic between racks and pods, connecting the pieces of a cluster that scale-up fabric doesn’t reach directly. AI clusters need far more inter-rack bandwidth than general-purpose computing, which pushes scale-out switch silicon toward higher port counts and higher per-port speeds sooner than the rest of the industry needs them. Those switch ASICs ship inside switch systems & white-box odms, physical switches built by both branded vendors and white-box manufacturers working from merchant switch silicon.

Copper versus optics

Both networks move data over one of two physical media: copper cable or optical fiber, and the choice between them comes down to a simple trade-off between cost, power, and reach.

Copper is cheap and needs no power-hungry conversion at either end — a signal just travels down a wire. Inside a rack, copper: dac / acc / aec cables connect GPUs, NICs, and switches over short distances, and AI racks pack far more of these high-speed links per GPU than general-purpose servers do. As per-link speeds rise, volume is shifting from passive DAC cables toward active ACC and AEC assemblies, which add signal-boosting electronics inside the cable itself to push a usable signal further down the same copper.

Optics trade a bigger cost and power bill for reach and density: a laser converts an electrical signal to light, which travels down fiber with far less loss over distance than an electrical signal on copper. optical transceivers do this conversion in a pluggable module at the end of a fiber run, currently transitioning from 400G through 800G toward 1.6T speeds as per-GPU bandwidth needs climb. Behind each transceiver sits a lasers generating the light and an optical dsps, drivers, tias conditioning the electrical signal on either side of it.

The package-level path

Zoom all the way in, and data movement starts before it ever leaves the chip package. A GPU die doesn’t sit alone — it’s built alongside hbm, stacked directly on the same package to give the accelerator very fast, close-range access to the memory it needs to stay fed, since moving data off-package to conventional memory would be far slower. Connecting the compute die to the HBM stacks (and to other dies on the same package) is the job of interposers, silicon bridges, rdl, silicon interposers and bridges that route those signals within the package before anything reaches a board-level connector at all. All of this is assembled by 2.5d/3d advanced packaging, the packaging process that turns a compute die and a set of HBM stacks into one finished, shippable accelerator — and one of the more consistently supply-constrained steps in the whole chain.

Once data leaves the package, it reaches the board through nics / supernics, which connect a server’s CPU and GPUs to the cluster network, sometimes paired with a dpus that offloads networking, storage, and security work from the host CPU so those cycles stay available for orchestrating accelerators instead.

Why the copper spine pushes toward co-packaged optics

The clearest example of copper’s limits shows up in nvlink copper spine / backplane cartridges, the cable assemblies and backplane cartridges that physically wire a scale-up fabric together inside a rack. A single NVL72-class rack uses roughly 5,000 of these copper cables to connect its GPUs into one scale-up domain. That’s a genuinely large amount of copper — and copper cable has hard physical limits: at high enough signal speeds, a copper trace or cable needs to be shorter and thicker to carry a clean signal, and packing enough of it into one rack to serve 72 GPUs is already close to what’s practical to route, connect, and keep cool.

As per-GPU bandwidth keeps climbing, that reach-and-density ceiling is what makes an optical alternative necessary rather than optional. silicon photonics & co-packaged optics is the industry’s answer: instead of a pluggable transceiver sitting at the edge of a board, the optical components move directly onto or next to the switch or GPU package itself, shortening the electrical path a signal has to travel before it’s converted to light. The power and cable-count savings from doing this only outweigh the simplicity of ordinary pluggable optics once link speeds get high enough — which is exactly the point AI-scale scale-up fabrics are now reaching. It’s the same story as liquid cooling: rising density didn’t create a new problem so much as it pushed an old one past the point where the existing solution still works.

Where to go next

• nvlink copper spine / backplane cartridges — the copper spine that’s running into its own limits.
• silicon photonics & co-packaged optics — where optics is moving to replace it.
• hbm — the memory sitting right next to the die that makes package-level bandwidth matter.