A $3 Clock Buffer Nearly Brought an Entire Computer Company to a Standstill
After AI demand exploded, ninth-generation PCIe clock buffers were swept off the shelves, and the substitute part tripped over a hidden difference in HCSL termination. What saved the line in the end wasn't a chip — it was a single 0201 resistor.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Fix the problem and you're further behind than before
The time spent troubleshooting was time that should have gone into building boards. By the time the fix was validated and the contract manufacturer could run the normal process again, the backlog of lost capacity hadn't disappeared — it had just been compressed into a shorter window. Bryan's words were "Now you're behind": every board that should have been produced still has to be built, but the cycle time is drastically compressed. This is the second-order cost of a hardware incident that's easy to overlook: not the fix, but the debt you owe after the fix.
— Bryan CantrillThe oscilloscope did the chip's termination for it
When Eric measured clock jitter he used an 8 GHz oscilloscope, and the scope itself terminates to ground at 50 ohms — which meant it supplied the termination this driver chip needed. So the waveform looked fantastic, performance was excellent. RFK nailed it in one line: this is the probe effect in the literal sense — the act of measuring made the thing being measured look normal.
— Eric AasenSwitching to ninth-gen parts ran straight into the AI demand explosion
Steve Tuck took the timeline back to the summer over a year earlier: a power electronics component supplier, taking advantage of a demand trough, switched a ninth-generation part to a tenth-generation one, sent the switch signal up to its wafer fab, and then relied mainly on inventory to bridge until the new part went into production. Less than thirty days after the switch, AI infrastructure demand exploded, ninth-generation parts were bought up, and the already-thin inventory overlap period drained fast. This PCIe clock buffer was one of those parts; the supplier later sold its entire timing business to another company, and during the handover supply information got even messier.
— Steve TuckThe clock was stuck high, and then he remembered HCSL is pull-up only
On the Winona line, a great production colleague looked at the waveform on an 8 GHz scope and found the clock was stuck high the whole time. Eric said this triggered a memory: HCSL is pull-up only. He realized on the spot that what had looked normal before was entirely the probe effect, and with that came a huge sinking feeling. Then the scramble began for 0402 resistors around 49.9 ohms.
— Eric AasenTwo kinds of output driver decide whether a substitute works at all
Robert pointed out the real trap in this part: the difference between a low power HCSL driver and an ordinary HCSL driver. Some output drivers need a ground reference, some don't — fundamentally a difference in how the line is terminated. These are 100-ohm signals, corresponding to 50-ohm differential termination; one kind requires you to provide the termination, the other doesn't. This difference wasn't spotted at selection time, and Bryan said "therein lies a critical small difference that God you wish you saw." It's the most hidden technical constraint along the entire substitution path.
— Robert "RFK" KeithRework 30 boards, and the resistance is off by three orders of magnitude
The supplier's inventory only had 45.3-ohm 0201s, so they reworked. The boards went into the programming station and didn't work at all, same as before. RFK measured that tiny resistor with a multimeter and read 25.6 k; he swapped in another one, switched to another clock output, still 25.6 k. The cause was a missing letter K in a copied description: they wanted 45 ohms, they installed 44 k ohms, and 30 boards had already been reworked that way over the weekend.
— Robert "RFK" KeithThe screening criteria for a substitute are harsher than you'd think
Eric laid out the hard constraints on finding a substitute: first, footprint compatibility, because building an interposer board is painful, and this part sits in a narrow gap between fans — a low-clearance space. Second, the input and output counts have to match: they needed one input and four outputs, and each output had to be individually enableable. Worse was the lead time — the boards themselves take six months, six months of supply was already queued in the supply chain, and changing the design meant not getting anything until next year. Even if they were willing to switch to an incompatible new-technology part, there was none to buy, because hyperscalers and big PC makers had swept the market clean.
— Eric AasenChange the stencil so the 0201s can go through normal reflow
They didn't want to hand-rework every board, so they changed the CAD and sent it to the factory, where the factory's quick-turn prototype shop made a new stencil with the 0201 resistor positions opened precisely. Because the resistors can sit directly on the vias, this batch of boards could go through the normal reflow process instead of having someone place each one by hand — doing that 30 times was already absurd enough, let alone for thousands of boards.
— Robert "RFK" KeithIn their own words · checked verbatim
It's the I mean, it is truly for want of a resistor. You do not have a computer company.
Bryan Cantrill0:00
we keep thinking we're kinda like past the point of like, oh, a resistor is actually standing between us and death. But we we were definitely on the operating table seeing the light on this one.
Bryan Cantrill0:00
It was like, buy every part you can get on the planet.
Steve Tuck2:00
And then Google is like, what happened here is a classic AI hallucination that it puts in quotes and bolds.
Bryan Cantrill6:30
And that like, know, end number of months of overlap of inventory until next gen is online. It was already starting to look extremely, extremely thin.
Steve Tuck12:00
So, this largely simple part, they can have different types of output drivers.
Robert "RFK" Keith20:00
It was like a cacophony of failures on all sides, The description was copied incorrectly, but the part number that was shown right above the description was correct and had the K in it.
Eric Aasen20:00
Figures
| Part unit price | About $3 | 32:00 |
| Number of substitutes found | About 3000 | 32:00 |
| Board lead time | About 6 months | 26:00 |
| Normal price of the clock buffer | About $34 | 0:00 |
| Price of the part before it went out of stock | About $70 | 0:00 |
| Size of the microchip that failed | 4 mm × 4 mm | 0:00 |
| Microchip part number | ZL4264 | 0:00 |
| Team's firefighting period | Five days | 0:00 |
Glossary
- clock buffer
- An analog chip that copies one reference clock into multiple outputs and distributes them, with almost no programmability.
- HCSL
- High-speed current steering logic, a differential clock signaling standard commonly used for PCIe; output drivers come in two kinds, one requiring termination and one not.
- probe effect
- The act of measuring itself changes the state of the thing being measured, making the measurement look normal.
- 0201
- An imperial-size surface-mount component package, roughly 0.02 inches by 0.01 inches, smaller than 0402.
- stencil
- A template used during SMT reflow to print solder paste precisely, determining where components are placed.
How to listen
Engineers and founders working in hardware, supply chain, or data center infrastructure — especially anyone currently selecting parts, hunting for substitutes, or stuck behind lead times.
The first ten minutes or so of internet gossip and AI-hallucination chit-chat can be skipped.