Burn-in, goodput, and the seven-day warning

What actually happens to a GPU between the loading dock and your training job — and why the metric that matters after handoff isn't uptime.

There's a version of the AI infrastructure conversation that stops at procurement. You secured allocation, the racks landed, the press release went out. Congratulations — you now own several hundred million dollars of hardware that has never proven it works.

The gap between "the GPUs arrived" and "the GPUs are earning" is a real engineering discipline, and it's one of the least discussed parts of this industry. It has two halves. Before handoff, you have to prove the hardware is good. After handoff, you have to prove it stays good while someone else's job is running on it — which is a completely different problem, because now you can't just reboot things.

Google Cloud recently walked through how they handle both for their GB300 fleet. It's a useful reference architecture, so I want to unpack the model, explain why each piece exists, and pull out what transfers if you're building or buying GPU capacity anywhere.

Why acceptance testing is not paranoia

Start with the arithmetic, because it reframes everything that follows.

Meta published detailed reliability data from training Llama 3 405B on 16,384 H100s. Over a 54-day window they logged 419 unexpected job interruptions, with roughly 78% traced to hardware — GPU faults being the single largest category. That's about eight unplanned interruptions a day, on a fleet that was professionally built and operated.

Scale multiplies fragility. Synchronous distributed training has no partial credit: one GPU throwing an uncorrectable ECC error stops all 8,192 of them.

So acceptance testing isn't about catching dead-on-arrival cards. Those are easy. It's about catching the marginal ones — the GPU that passes idle diagnostics and fails at 95% sustained load, the NVLink connection that's fine until it's saturated, the thermal solution that's adequate until the rack next door heats up. Marginal hardware is worse than dead hardware, because dead hardware fails during your testing and marginal hardware fails during your customer's training run.

The pipeline

Phase one: foundational testing

Three tools, three depths. They aren't alternatives — each catches a class of problem the others miss, which is why the order matters.

And then NVLink, separately

On rack-scale systems this deserves its own callout. A GB300 NVL72 rack is 72 GPUs in a single NVLink domain, wired over a passive copper backplane, moving roughly 130 TB/s in aggregate. That interconnect is the difference between a rack and 72 separate computers.

NVLink faults are also the nastiest failure class, because they often don't fail — they degrade. A link that negotiates at reduced width still works. Every collective operation just gets slower, and since collectives are synchronous, one degraded link paces the entire job. Nothing errors. Nothing alerts. Your customer just quietly gets 70% of the throughput they paid for.

Which is why link width, error counters, and measured bandwidth all have to be validated explicitly, and why the validation has to keep running after handoff.

Phase two: the bill of health

Here's the part most operators skip, and it's the part that separates a serious platform from a rack of tested components.

Diagnostics test components. They tell you each GPU is healthy, each link negotiates correctly, each node passes stress. They do not tell you the system works, because real AI workloads produce a load pattern that no synthetic test replicates.

Real training has a distinct signature: synchronized power ramps as thousands of GPUs enter a forward pass together, all-reduce collectives that saturate the fabric in bursts, checkpoint writes that hit storage simultaneously from every rank, and thermal soak that builds over hours rather than minutes.

That's why the second phase runs actual model workloads rather than synthetic load. Google's framework uses NVIDIA's Nemotron models to drive representative training and inference as the final gate — real architectures, real collective patterns, real memory pressure. The point isn't the specific model family; it's that the validation workload has the same shape as the customer workload.

What this phase catches that diagnostics can't:

• Stragglers. One node consistently 5% slower than its peers. Nothing is broken; a 512-node job is now 5% slower. Only visible when you compare nodes running identical work.
• Fabric behaviour under real collectives. Congestion, adaptive routing decisions, and RoCE flow control tune differently under bursty all-reduce than under a synthetic bandwidth test.
• Thermal behaviour at rack scale. One node at full power is a different cooling problem from eighteen nodes at full power, synchronized, for six hours.
• Storage under burst. Checkpointing is the pathological case — every rank writing terabytes at the same instant.

What changes at handoff

Now the priorities invert. Before delivery, you optimize for thoroughness — take as long as you need, break things deliberately. After delivery, you optimize for non-disruption, because there's a 30-day training run on the machine and the customer has thousands of GPU-hours of state at risk.

Two metrics carry the weight, and both are better than uptime.

The telemetry that makes it computable

You cannot calculate MTBI or goodput from one data source. Both are derived metrics that require correlating signals from the silicon all the way up to the training loop — which is why observability in this domain is a layered architecture, not a dashboard.

Two design notes worth stealing:

Passive tests run continuously. Validation doesn't stop at handoff. Lightweight health checks and periodic bandwidth conformance runs keep executing against live infrastructure, because degradation is a process, not an event. The NVLink-at-reduced-width problem is invisible to any test you only run once.

Straggler detection is its own signal. Google's monitoring treats stragglers as a first-class metric precisely because they're non-crashing failures that drag down an entire synchronous workload. Nothing errors. The job just gets slower. In a fleet where every rank waits for the slowest, a straggler is a fault even though every component reports healthy.

The seven-day warning

Now the part that's genuinely a product decision rather than an engineering one, and the reason I wanted to write this up.

Telemetry flags a GPU crossing its thermal threshold. It's not dead. It's not throwing errors. But per vendor guidance it needs to come out for diagnosis. A customer's job is running on it right now.

The reflexive operator answer is to take the machine. Safety first, hardware protected, VM terminated. And that answer is defensible engineering — and terrible for the customer, whose 30-day run just lost everything since its last checkpoint, because you made a scheduling decision on their behalf with no notice.

The alternative: send a notification, and give them seven days to choose the moment.

What makes this work is that it converts an unplanned interruption into a planned one. The customer stops at a checkpoint boundary, drains cleanly, maybe migrates to spare capacity. Lost work approaches zero. MTBI improves without a single hardware change — because MTBI counts interruptions, and a scheduled stop the customer chose isn't one in any way that matters.

What to take from this

If you're buying GPU capacity, these are fair questions for any provider, and the quality of the answers tells you a lot:

• What's your acceptance test process, and will you share the bill of health for the hardware I'm assigned?
• What's your measured MTBI on fleets my size? Not node uptime — interruptions per job.
• Do you expose goodput, or only allocation?
• How much notice do I get before you take a machine for maintenance?
• Do you detect stragglers, or is that on me?

Any provider that can only answer with an uptime SLA is selling you allocation and calling it reliability. If you want the financial half of this picture — what a megawatt costs and what it returns — that's covered in What $2.3 Billion Actually Buys.

If you're building the platform, three things transfer regardless of scale:

• Component health and system performance are different claims. Test both. Diagnostics prove the first; simulated workloads prove the second.
• Re-entry needs a gate. Repaired hardware passes the same qualification as new hardware. No exceptions, no "it's probably fine."
• Detection and disruption should be decoupled. Finding a fault and acting on it don't have to happen in the same instant. That gap is where you give the customer agency — and it's the cheapest reliability improvement available to you.

The industry spends enormous energy on peak FLOPS. Almost none of that number survives contact with a real 30-day training run. What survives is goodput, and goodput is an operational discipline — built at the loading dock, defended continuously, and measured honestly.

Frequently asked questions