On a single board sit two large FPGAs, a Zynq UltraScale+ RFSoC and a Kintex UltraScale, joined by eight high-speed serial lanes. The goal for this session was to turn the second chip into an extension of the first: a place to run logic, and four gigabytes of memory beside it to use as remote storage. An AI drove the whole bring-up and the whole hunt in our Python2Verilog framework; the engineer only set the direction at each step.
Before any of that could work, the link between the two chips had to carry real bytes both ways. One direction came up quickly. The other refused, and refused in the most expensive way a problem can: no error, no crash, every check green.
One direction would not link, and every layer passed
The lane runs a standard line code: each block of 64 data bits carries a 2-bit sync header, and the receiver locks once it sees enough legal headers in a row. One direction locked. The other kept sliding, always a hair short of lock. So we measured, layer by layer, and every layer passed: the vendor's own example simulation built the link; a loopback simulation locked and carried a frame bit-exact; a cross-chip simulation wired to the real connection locked both ways and looped back byte-exact; an analysis of 400,000 bits on the simulated wire found the sync headers 100% legal at the one correct offset; and a comparison of the placed-and-routed netlists, property by property and pin by pin, found the good and bad builds identical.
The eye told the same story. A scan of the receiver showed an eye wide open and clean: 77.8% horizontal, 93.6% vertical. The signal quality was flawless. The problem was not signal quality.
Catch the real bytes on the wire
Every reading so far was a statistic. So we added a one-shot capture that read three consecutive raw 64-bit words straight off the locked lane. They came back as a repeating 0011 pattern. Scrambled data should look random; this was a plain square wave, a 2.5 GHz clock. And 66 divides by 4 with a remainder of 2, so slicing the wire into 66-bit frames slides the phase by 2 bits each frame and the sync header always lands on 01 or 10. Both are legal.
One capture ruled out an entire class of guesses. There was no data on the wire at all, only a free-running clock pattern that had been impersonating a healthy link the whole time.
The map was reversed
So we measured the lane wiring again, from scratch, by physical name rather than tool index, sending a different binary-coded marker down each lane. Three rounds resolved all eight. The result was the exact opposite of the record we had been trusting. The transmitter had been sending correct data down a lane nobody was listening to, and the monitored lane was connected to one that had never been configured, free-running that square wave. Every simulation had wired itself from the same record, the record was internally consistent, so every simulation passed.
An earlier bit-error-rate test had already run on these eight lanes, seven of them with zero errors, so it is fair to ask how it missed a reversed pair. Because that test sends the same pseudo-random sequence down every lane: whatever connects to whatever, each receiver locks and reports zero errors. It is blind by construction to which lane connects to which. The copper was fine on every lane. What was wrong was the table of which one went where. The fix was to rebuild on the two correct lanes, and it linked on the first try, byte-exact. Three genuine bugs found along the way were kept, but none of them was the cause.
Pull the DDR4 pinout out of a PDF, by machine
Remote memory was next. The four gigabytes of DDR4 beside the second chip had no machine-readable pinout, only a 75-page schematic PDF, and its 64-bit interface has 119 pins where one hand-copied wire means a failed calibration. So the AI turned it into a geometry problem: extract every word on every page with its coordinate box, pair each net name with the nearest pin number on the same row, and require every net to pair twice to the same pin or the whole thing is rejected. That zero-conflict gate resolved 117 nets with no conflicts, including the address lines hidden inside multiplexed names. The vendor's memory-controller IP was the arbiter: it ran to placement and passed its own byte-group and bank legality checks, with no human eyeballing a schematic. Burned to the board, the DDR4 calibrated on the first power-up.
The second chip becomes an extension
With the link carrying real bytes and the memory calibrated, the second chip took on three roles, each proven on the real hardware. A 256-kilobyte round trip over the link came back byte-exact. A Wi-Fi LDPC decoder placed on the second chip took in 1,296 bytes of soft information and returned 164 bytes of decisions with zero errors against the reference model, and still zero errors when run twice without a reset in between. A four-kilobyte write across the link and back read out byte-exact, and again at a deep address. From then on the second chip is three things to the first: a 10-gigabit pipe, a compute node, and four gigabytes of remote memory.
There was a loose end. The earlier bit-error-rate test had flagged a trace of errors on one of the eight lanes. Before closing out, that lane was soaked on its own for ten minutes: six trillion bits, zero errors, with an eye as open as the other seven. That lane was the one the reverse link had been using. The link then went one step further, bonding several lanes into one wider channel, and the two-lane version already passes byte-exact. Bonding surfaced one more assumption, that the two chips number their lanes in opposite order, which a single lane could never reveal. The fix has each lane announce its own identity in the data and the receiver reorder by what it hears. The wiring is now measured, not assumed.
Three rules we kept
Each of these went back into the framework as a check that now runs on every build.
Lock is not data. A square wave can satisfy sync-header lock, CDR lock, a clean eye, and a perfect header-legality rate all at once. A link is only accepted when it decodes real payload and matches byte for byte.
A map is trusted only by physical name. Lane wiring is recorded by the physical names on both ends and measured with a distinct marker per lane. A tool's numbering is not evidence of what connects to what.
A pinout is trusted only by machine. Schematic pins are extracted by geometric pairing, admitted only when every net resolves with zero conflict, and validated by the vendor IP rather than by reading the drawing.
Why this matters
The most expensive failures are the ones where every layer says pass. When the circuit checks out at every level, the thing to question is the measurement itself. A record with one reversed line survived a full day of work precisely because every simulation dutifully wired itself from it. The same discipline runs through everything we build: prove each layer against a real reading, and trust the state of the hardware over the document that describes it. That is how a second chip became a proven extension of the first, one measured step at a time.