This post will be an advanced deep dive of a state-of-the-art co-packaged optical system based on NVIDIA’s Dense-Wave Division Multiplexing (DWDM).

This post is organized in the following sections:

• What is Dense-Wave Division Multiplexing?

• A complete architecture breakdown of NVIDIAs DWDM with key physics constraints:

  • PIC - Ring modulator physics

    • Ring coupling theory - splitter and phasor theory

    • Key ring properties

    • How PN junctions shift the resonant frequency by changing the refractive index

    • Nonidealities - Thermal sensitivity and crosstalk

  • EIC - Clock forwarding and jitter analysis

    • Conventional clocking techniques - Forward clocking and embedded clocking

    • NVIDIA’s clocking architecture - half rate DWDM link architecture

  • 🔒 A Breakdown of the DWDM CW-DFB Laser Array and performance results

  • 🔒 Overlay of two critical spectrums that illustrate precise alignment challenges:

    • 🔒 NVIDIAs MRM transmission spectrum from the ISSCC paper

    • 🔒 NVIDIAs DFB Laser Array power spectrum from the ECTC paper

  • 🔒 A brief high-level overview of key CPO packaging features with TSMC COUPE

DWDM is fascinating because it blends together the knowledge amongst different specialties who all work together to create a successful product:

• The on-chip mixed signal SerDes designers designing the high speed TX and RX circuits

• The material scientists/solid state physicists quantifying the behavior of PN junctions and SiPh materials

• The packaging engineers who integrate everything together

The primary sources of this post are two NVIDIA papers on the circuit architecture and packaging aspects of DWDM from conferences I attended the sessions for in-person. These are:

• ISSCC 2026: “A 32Gb/s/λ 256Gb/s/Fiber Half-Rate Bandpass-Filtered Clock-Forwarding DWDM Optical Link in a 3D-Stacked 7nm EIC/65nm PIC Technology”

• ECTC 2026: “Design and Packaging of a DWDM CW-DFB Laser Array for Co-Integrated Optical Interconnects”

I also integrate relevant material from relevant ISSCC forums and short courses as well.

I find that one of the best ways to learn more advanced concepts is to work backwards and figure out what simpler prerequisite concepts you need to learn first. The free half covers a huge technical breadth of material with links to several of my fundamental posts I wrote leading up to this. This way, people of all skill levels can work backwards and figure out what they need to learn to understand more complex concepts.

I want to challenge you to try to learn one new concept in depth from this post or my body of work, even if everything else doesn’t make sense.

Please note that I do not consider myself a photonics expert. I wrote this post using the sources listed above as sources-of-truth to base my commentary off of. If you’re an expert familiar with this material and notice a mistake in this post, please reach out to me so I can have this promptly corrected.

For a broad overview of optical communications, including VCSELs, optical modes, and key impairments, I recommend reading this post where I dive into these fundamentals in the free section:

What is Dense-Wave Division Multiplexing (DWDM)?

Optical interconnects in co-packaged optics help overcome the energy dissipation and frequency scaling challenges inherent in copper interconnects. Co-packaged Optics moves the optical engine as close to the chip as possible to minimize the length the electrical signal travels before being converted to the optical domain.

Optical communications enable low-energy and low-latency data transmission. However, optical communications is limited by a few constraints:

• The physical edge density. The optical signals need to be propagated through waveguides into a fiber optic connector. How and where the signal flows from the laser source to the electrical devices matters a lot.

• Frequency constraints of individual devices. The electrical as well as the optical bandwidths needs to be fast enough to handle the data throughputs needed.

Several companies propose solutions that push all available scaling dimensions. One of the most promising solutions to scale data speed involves densely multiplexing light signals of different wavelengths on the same waveguide.

Dense-Wave Division Multiplexing consists of individual optical channels of equally spaced wavelengths. Electrical signals are frequency multiplexed in a waveguide through a series array of micro-ring modulators. Ring modulators act like on-off switches for each of their respective channels. Ring modulators are ultra-compact, enabling many channels to be packed into a common waveguide.

Pros and Cons of DWDM

Compared to other modulation options, DWDM has several benefits:

• Scalability. DWDM performance offers scaling potential with the # and size of devices.

• Low latency for XSR due to no DSP

• High energy efficiency

• High bandwidth density

However, there are several challenges with DWDM that involve the material physics aspect of these devices:

• Temperature sensitivity. Lasers and MRMs are inherently sensitive to temperature and must be well controlled

• Design Complexity - DWDM moves a lot of the complexity from the electrical domain to the optical domain, and imposes unique clocking and jitter tracking challenges at high speed

• Laser Integration - DWDM needs a multi-wavelength laser source that has to be brought onto the chip

  Silicon Co-Design is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Architecture Breakdown - EIC, PIC, Laser Array, and Packaging

Next I’ll discuss NVIDIAs DWDM multiplexing architecture. I’m going to describe the physics of the MRM on the PIC first to show how it works at a fundamental level. I’ll then discuss the clocking scheme of the EIC and show how the PIC requirements influence the EIC requirements.

At an architecture level, the job of an optical transceiver is to make sure that the data sent through an optical laser is received by a photodetector on the other end with acceptable BER. When selecting components, link budgeting is performed to take into account all sources of losses and optimize the performance of the entire system. I discuss link budgeting in the free section of this post:

PIC - Ring Resonator Physics

In a nutshell, microring modulators (MRM) accept or reject specific wavelengths of light going through a nearby waveguide. MRMs couple evanescently to nearby optical waveguides and act like band-stop/pass optical filters, depending on the way you look at it. Light is normally guided through an optical waveguide and passes right through, but the MRM acts like a low impedance for light at specific wavelengths.

MRMs consist of the following layers:

• A circular optical waveguide (grey) that is “sandwiched” between two regions:

  • p-doped region (blue)

  • n-doped region (red)

• A heater (green) that surrounds the MRM to regulate the temperature

A voltage source is attached to the n-doped region (yellow) to switch the optical signal going through the waveguide by shifting the resonant frequency over.

Lets dive into each of these elements in more detail.

Ring Coupling - the Splitter

The ring can be thought of as an splitter with two input and two output terminals. . Think of the ring like a “race track” that the coupled signal enters, runs laps around, and returns back onto itself until it ultimately decays due to losses.

In this splitter network, there are two constants that define the input-output characteristics:

• t is the transmission coefficient and represents how much of the signal gets through. Ideally, t should alternate between 0 and 1 at specific wavelengths.

• k is the coupling coefficient and is affected by how closely spaced the ring is to the optical line.

By conservation of energy, t^2 + k^2 = 1.

Once the optical signal enters the loop, it experiences loss and phase shift represented by a(λ). If the ring is unwound, the path of the light is represented by the phasor:

where the “a” term factors in the loss factor α and the propagation constant β that represents the wavelength-dependent phase shift:

The transmission formula is given by the ratio of the light that makes it through the optical waveguide vs the light coupled into the ring across wavelengths, which is given by:

Through substitution, as well as simplifying the phasor into a cosine, we arrive the following:

This formula lends itself to the following properties:

Rings Resonance Frequencies

In a microring modulator, the ring resonates when the diameter is equal to an integer number of wavelengths. These wavelengths is given by

where

• n_eff is the effective refractive index

  m = integer modes (1,2,3…)

  r = ring radius

Due to the periodic nature of the ring, the resonances are contained within a free-spectral range that limits the wavelength window signals can couple into it. Any signal outside of this window will alias into it.

One important property of the ring is the Full Width at Half Maximum (FWHM). This is the “width” of the resonance measured at half the max power (or 3dB on a log scale) at both edges of the resonances. FWHM is a useful qualitative reference to measure how “wide” the resonance is. This is given by:

The quality (Q) factor describes how narrow the resonance width is compared to the frequency it is centered at. This is given by:

where

• c is the speed of light in vacuum

  τ is the cavity photon lifetime that lumps in the above parameters

In NVIDIA’s implementation, the TX ring modulator quality factor Q is ~4.5K, and RX filter Q is ~4K.

In order to switch electrical signals into this ring, the ring bandwidth needs to be wide enough. The ring bandwidth is given by

where

• R is the ring junction resistance

  C is the ring junction capacitance

NVIDIA switches into each ring at 32Gbps, corresponding to a switching frequency of 16GHz.

PN Junction shifts the frequency by changing the refractive index

PN junctions normally act as electronic diodes that, electrically speaking, act as one way current flow with a small built-in voltage drop (~0.7V).

In conventional PN junctions, the p side is doped with type III elements (such as boron, gallium, or indium) with free holes in the valence band, and the n side is doped with type V (such as phosphorus or arsenic) atoms with free electrons in the conduction band. When a p and n material are sandwiched together, it creates a depletion region of “built-in” electrical potential due to the flow of carriers between the materials.

However, when PN junctions are used in SiPh, we care about the solid state physics properties that affect the index of refraction. In optical, these junctions are biased in reverse bias. As more voltage is applied to the n-doped region, the depletion region widens. This lowers the carrier concentration, causing the refractive index of the silicon to spike upward. This shifts the resonant frequency and moves it just enough to allow a light source to pass through. Think of this operation like a door opening and closing really fast.

In fact, these PN structures can be altered and routed in series with the optical signal itself. Distributed PN structures are used in Mach-Zehnder modulators that modulate travelling waves along long lengths and unique structures such as p-i-n are used in electro-optic modulators. Each structural variation has their own unique characteristics and tradeoffs. I discuss these options behind the paywall in this post:

Nonidealities

I’ll discuss two nonidealities of MRMs: Thermal sensitivity and crosstalk.

Thermal Sensitivity and Tuning Loop

Ring modulators are sensitive to temperature primarily because of the thermo-optic effect. The thermo-optic coefficient of silicon is ~1.8 x 10⁻⁴ K⁻¹ and changes in temperature affects the index of refraction.

Here, the thermal tuning loop measures the current at both TX and RX through:

• A drop-port photodetector (PD) at the TX to sense how much optical power is leaking out of the ring

• The TIA DC-loop at the RX

The PD current value is first digitized with an 8 bit SAR ADC, then processed digitally to produce a pulse density modulated signal to drive an integrated metal heater.

The paper doesn’t give too much detail beyond this, but I would imagine the ADC/DAC precision and loop bandwidth needs to be acceptable to correct for the expected temperature fluctuations with an acceptable wavelength error.

I dive into depth regarding the SAR ADC in this post:

Crosstalk

When multiple microrings are in close proximity to each other on the same substrate, cross talk is a key first-order phenomenon. The frequency characteristics of microrings tend to have long tails that poorly suppress sidebands and leak into other rings as well.

When deciding on what the Q factor should be, there is a tradeoff between crosstalk and the bandwidth:

• A higher Q lowers crosstalk and bandwidth

• A lower Q increases crosstalk and bandwidth

Modelling crosstalk is not that intuitive since there are many crosstalk paths among different rings that occur at different wavelengths.

Summary

Ring modulators appear simple at first but are highly affected by material properties and their tightly coupled nature. They are subject to device-to-device variation, so appropriate tuning circuits and control loops are needed to account for this, adding additional design complexity.

EIC - NVIDIA’s half rate DWDM link architecture

Now I will talk about the electrical side, more specifically the clocking architecture.

Of all the topics I’ve written about, I found that clocking is perhaps one of the trickiest subjects to explain in depth. As a simple analogy, think of clock synchronization like an orchestra conductor making sure all the instruments in the orchestra are in sync with minimal variation in when notes are played. There are many outside sources that can affect each individual musicians rhythm.

Each ring modulator is driven through their own SerDes electrical circuit. Compared to standalone high speed SerDes, each ring’s SerDes is slower and doesn’t have the DSP due to the XSR nature, easing some complexity. However, parallelizing 8 data lines requires synchronization and jitter control that adds additional complexity to the clock network.

Lets break down NVIDIA’s clocking architecture in more detail and see if we can make sense of this.

Conventional Clocking Techniques

When digital data is transmitted through a channel, there needs to be some way to clock the data at the RX to recover the data. There are two conventional ways of doing this:

• In a forward clocking scheme, the clock is sent alongside the data in a separate channel.

• In an embedded clocking scheme, the clock is “integrated” into the data stream. The data stream is serialized in a way that ensures enough voltage transitions (edges) occur to represent the timing information. This clock is then recovered at the receiver side through a clock data recovery (CDR) circuit.

I discuss the conventional circuits used in CDR circuits (such as bang-bang phase detectors) after the paywall in the following post:

No matter which scheme is used, each channel will couple in noise that appears as clock jitter, which is a deviation of the clock edge from its ideal location. There are two types of jitter:

• Correlated jitter - this is jitter that is generated from device operation such as TXPLL jitter and supply noise induced jitter. This jitter is identical lane to lane.

• Uncorrelated jitter - this is jitter that is random in nature, such as thermal, flicker, shot, etc, and is mostly generated through devices. This jitter is different from lane to lane.

Jitter show up as “tails” in the frequency content of a high speed signal. If jitter is high enough, it can degrade BER from the value being outside the threshold used to read 0 or 1. I discuss more about jitter in this post:

When clocking parallel channels, it is important to track the jitter and filter out unnecessary sources of it. In general, it is easier to track correlated jitter than uncorrelated jitter, and filter out low frequency sources of jitter.

There are a few pros and cons of each scheme:

• Forward clocking has excellent jitter tracking bandwidth, but requires a separate clock line. Additionally, uncorrelated jitter can accumulate.

• Embedded Clocking saves on IO, but CDR circuits have the potential to be exposed to jitter beyond its limited bandwidth.

DWDM has the unique benefit of embedding the clock in as a separate wavelength along the data lines, but is does come with a a few unique challenges:

• The data at the TX needs to enter the rings synchronously

• When data is sent down through the optical fiber, there will be delay mismatches due to various non-idealities such as chromatic dispersion and polarization mode dispersion.

• The thermal noise-induced jitter is dominated by the transimpedance amplifier (TIA) at the RX

Lets discuss what NVIDIA does to overcome these challenges.

NVIDIAs clocking architecture - half rate DWDM link architecture

NVIDIA uses a half-rate FC DWDM link architecture that embeds the clock inline with the data. In this architecture, there are 9 rings on each of the TX and RX that combines for a total data throughput of 256Gbps:

• 8 rings are for data at 32Gbps each

• 1 ring is for FWDCLK at 16GHz

On both the TX and RX side, the signal will traverse different lengths depending on which ring it coupled into. As a result, the TX clock needs to be de-skewed to account for known path length misalignment.

At the RX end, the transimpedance amplifier (TIA) adds a lot of thermal-noise induced uncorrelated random jitter. This TIA noise primarily comes from the feedback resistor (circled above) which is necessary to provide a low impedance input from a current sourced photodetector into the TIA.

To overcome these jitter challenges, NVIDIA uses an Injection locked oscillator (ILO) - based bandpass filter to bandpass filter the clock with BW ≈ 1-2GHz. This filters out jitter beyond 1GHz to remove most of the uncorrelated jitter and tracks most of the correlated jitter.

This clock is then distributed to the RX as illustrated by one example setting of the RX CLK distribution. Each RX has a local ILO that acts as another jitter filter. The clock distribution is segmented into two distinct spinal wires CLKA and CLKB that cut the capacitive load per wire in half. The RXPLL acts as a on-chip frequency synthesizer that provides the local timing reference upon startup until the system is locked to the main timing source.

I write about phase-locked loops in more depth:

DWDM Laser Source and Packaging of EIC and PIC

After the paywall, I will cover the packaging part that represents a major constraint in the physical realization of this system. I will cover the following:

• A breakdown of the DWDM CW-DFB Laser Array and performance results

• A brief high-level overview of key CPO packaging features with TSMC COUPE

In between these two discussions, I will overlay NVIDIAs measurements of the ring spectrum from the ISSCC paper with the DFB Laser Array from the ECTC paper, and discuss the challenges with aligning both of these. Alignment between these two spectrums is crucial across multiple devices because misaligned channels don’t transmit data.

Here are several other posts I’ve written that cover the fundamentals of advanced packaging in-depth:

Share

Read more

Editors Note (6/23/26) - The last half of the post will be paywalled. Before each paywall, I link several of my other posts in domains adjacent to the topic you are reading to give you a well rounded flavor of the areas adjacent to packaging.

At ECTC 2026, TSMC noted that many package designers are being too aggressive with their package designs and that a lot of communication is needed to balance the design performance with the manufacturing and reliability constraints of the package.

This post helps reduce that communication overhead.

In this post I’ll be diving into the key mechanical and thermal factors needed for co-designing advanced packaging for high power. The outline of this post is as follows:

• I will explain the current I-shaped paradigm in most technology companies and why engineers will need to embrace T-shaped skillset to effectively co-design tightly integrated systems

• I’ll discuss the key challenges that GPUs are placing on both the package and circuit side

• Then I’ll discuss two fundamental concepts and associated material properties for successful co-design of sub-components with advanced packaging:

  • Thermodynamics, including the three sources of heat transfer and thermal resistance

  • Stress-strain and associated material properties such as CTE

• Then I’ll describe the:

  • 🔒Three main sources of warpage

  • 🔒Four common failure modes

  • 🔒Challenges with modelling cross coupling failure modes

• 🔒Last, I’ll discuss a few warpage mitigation options

The research from this post comes from both from my conference experience at ECTC and the book “Hybrid Bonding, Advanced Substrates, Failure Mechanisms, and Thermal Management for Chiplets and Heterogeneous Integration” (2025). I found this book to be a useful mix of both high-level fundamental overviews of advanced packaging and current in-the-weeds academic research on key areas related to packaging.

I previously wrote a masterclass post on the fundamentals of advanced packaging that covers what packaging “is”. This post builds off that by covering the physics dimension when it comes to CTE, warpage, and failure modes. I recommend you read that post along side this one to build a more complete view of advanced packaging.

First I want to describe what I believe effective hardware co-design requires.

A New Approach for Hardware Co-Design: The T Shaped Engineer

Currently, most engineers at major tech companies operate in the “I” shaped paradigm where engineers are highly specialized in their field of choice, and interface with other engineers through requirement specifications and handoff at interfaces.

As systems start become more tightly coupled across domains, many leaders are calling for more cross domain awareness and co-design. However, I believe that most engineers in the “I” shaped paradigm treat co-design as more frequent and earlier meetings.

I believe that effective co-design requires you to become more ‘T” shaped, that is, depth in one domain and breadth in others. This requires you to build a semi-independent knowledge base of high level concepts in other domains to be able to intuitively assess how factors from your domain affect others. This allows you to anticipate issues and ask key questions from domain experts early on.

Becoming T shaped is actually quite difficult to do in an I shaped paradigm, both cognitively and organizationally. Vikram Sekar described these challenges best when he gave career advice because often times, it might be possible to “step” on adjacent teams who are CTA.

I totally understand this, and I’m not saying you should ignore your orgs politics. In any case, here is my blueprint for people who want to become more “T” shaped:

Ground yourself in fundamental tradeoffs in silos adjacent to your domain of expertise

Effective co-design across multiple domains requires you to first ground yourself in the fundamentals of domains that relate to each other in some way. This requires deep work to analyze the technical fundamentals in domains adjacent to your own.

For me, the bulk of my co-design posts revolve around four interconnected domains in hardware:

• On-chip mixed signal (SerDes architecture, ADC, PLL)

• High speed communications (Optical, high speed wireline, and signal integrity)

• Power integrity / management

• Advanced packaging

Each specialty has their own unique set of tradeoffs, such as:

• Gain vs bandwidth in Analog

• Interconnect distance vs BER in SI

• Switching frequency vs ripple vs loss in Power

• Substrate area vs bump stress in Packaging

Understanding the fundamentals in each domains is not an easy process because it requires to you sort through domain specific knowledge from experts and try to extract out the important insights. I found that the tutorials at the beginning of the conferences are excellent at providing a framework to understand more complex conference material that build upon the tutorial material; these are what the tutorials are specifically made for. That way, when I read dense technical papers or conference digests, I can efficiently filter through information that doesn’t matter. I find that the bulk of the writing in conference papers is really dancing around the nuances of previous works while making claims of proposed solutions.

I’m not saying you should become an expert in domains outside of your own; that is overkill for practical purposes. However, I do think you should work backwards to figure out what baseline knowledge you need to understand more domain-specific knowledge from other people.

Understand how adjacent domains couple together

Then, after understanding tradeoffs in each domain, try to understand how effects from each domain “couple” to other domains. Based on my previous writing, these are some interactions I can think of off the top of my head:

• Power supply induced jitter and crosstalk lower BER of the signal path

• Fast current transients in AI workloads cause localized joule heating that can worsen reliability

• Ring modulators in optical communications are sensitive to thermals that need to be corrected with heater feedback loops

Understanding interactions is quite tricky that I admittedly don’t really understand fully quite yet and am trying to understand better myself.

My advice: start small and teach others

As a start, I recommend people first ground themselves in ONE field. Power and packaging are excellent interdisciplinary fields to start for anyone involved in semiconductors in anyways. I have several posts that provide a good starting point.

Then, to really test if you have the knowledge, try to teach it to others, whether that be through writing or a seminar. This way, others can mutually benefit from the insight in your own process of becoming more T shaped. It’s quite easy to fool yourself in thinking you know something from reading someone’s article until you go and sit down and explain that topic to another person.

The Co-Design Problem: High Power Density Stresses the Package

GPU roadmaps are projecting a huge increase in computational power that comes with higher performance requirements, including:

• High power usage. GPUs draw huge current density that is lost in the PDN and dissipates heat.

• High transient response. The critical frequency band of AI workload transients is between ~0.1-10 MHz, and the PDN needs to provide low enough impedance at these frequencies to meet the on-chip droop specs. This is typically done by placing localized decoupling capacitors as close to the chip as possible.

• Large Area. More GPUs per package increase the package area that increases the Distance to the Neutral Point (DNP).

To meet these performance demands, innovations are both needed on the package and circuit design side.

On the circuits side, much work has been done on optimizing specialized power conversion topologies such as multi-phase buck/TLVRs at the PoL and LLC DCX for the IBC. I dive deep into these fundamentals of the circuit architectures in more detail:

On the packaging side, there are several well established failure mechanisms and integration techniques. I briefly touched upon these in these posts:

The issue with high power is that they stress the package, enhance the already established failure mechanisms, and opens up possibilities for new ones. Large BGAs with up to 2000 - 6000 balls face mechanical challenges with heavy heat sinks and continuous vibration.

Lets dive deeper into important packaging considerations.

Fundamental Thermal and Mechanical Concepts: Thermodynamics and Stress-Strain

Co-design of the high power with the package involves knowledge of fundamental concepts in thermodynamics and stress-strain. These are pretty well known to people in those domains and I realize there’s a lot more depth underneath these equations I am simplifying.

As an EE myself who is more specialized on the circuit side, I’m framing these as high level mental models for people outside of these domains to have a first order understanding of the mechanical and thermal effects of packaging.

Likewise, if you are very familiar with these but not familiar with those concepts on the electrical side like signal integrity, power electronics, or the intricacies of high speed SerDes, well, lucky for you, I have mental models on those on my Substack as well that are linked throughout this post.

The Three Sources of Heat Transfer

Conduction

Conduction is heat transfer within a stationary solid, liquid or gas due to temperature differences. Energy is “diffused” from more energetic regions to less energetic particles.

Conduction is governed by Fourier’s law:

where

• q′′ is the heat transfer per unit area (W/m² )

  k is the thermal conductivity of the material (W/m K)

  dT/dx is the temperature gradient (K/m)

Common values for thermal conduction range from:

• 0.024 (air),

• ~0.2 - 0.35 (for epoxy, polymide, and FR4)

• 120 (silicon)

• 390 (copper)

Conduction matters when dissipating heat from:

• The GPU - Rubin generates 1,800W to 2,300W per GPU

• Joule heating in the interconnects - this is self-heating due to IR drop that increases as current density increases

Convection

Convection refers to heat transfer between a surface and a moving fluid at different temperatures

Convection is governed by Newtons laws of cooling:

where

• q′′ is the heat transfer per unit area (W/m²)

  h is the convective heat transfer coefficient (W/m² K)

  Ts is the surface temperature of the solid (K or ⁰C)

  Tinf is the temperature of the surrounding fluid far form the surface (K or ⁰C)

Convection can be classified either as free (such as hot air rising from a heater) or forced (such as cooling a CPU with a fan).

Common values for convective heat transfer coefficient include:

• Free: 2 - 25 for gasses and 1000 for liquids

• Forced: 25 - 250 for gasses and 100 - 20,000 for liquids

Convection cooling can be facilitated through the use of heat sinks, liquid cooling or immersion cooling with single and two phase variants.

Radiation

Radiation is the transfer of thermal energy through EM waves without a need for a medium. It is governed by:

where

• E is heat transfer rate of emission (W)

• ε is the emissivity of the surface (between 0 to 1)

• σ is the Stefan-Boltzmann constant (5.67e-8 W/m² K⁴)

• A is the surface area of the radiating body

• T_s is the absolute temperature of the surface (K or ⁰C)

Compared to conduction and convection, radiation is not a primary heat transfer mechanism for ground based AI data centers, but could potentially be a dominant heat mechanism for space based AI data centers. We’ll see how that plays out.

Note that radiation also includes the absorption of incident radiation, so the net heat transfer depends on the balance between the inflow and outflow of radiation.

Conduction and convection must be in balance for ground based AI data centers

Fundamentally, for a GPU to maintain a thermal equilibrium below a threshold temperature, the conduction and convection equations must be in balance with each other for the cooling system to handle the heat dissipation of the GPU and subsystems.

Of course, there are many other factors such as boundary conditions, unique material properties, and CPU loads that are important and must be taken into account. If, for whatever reason, the heat removal becomes a bottleneck of GPU performance, then tricks need to be played on the SW side to manage the workload until the cooling or packaging system is able to catch up.

Thermal Resistance

Thermal resistance quantifies the opposition to heat flow at any two points where there is a temperature difference. It is given by

where

• R_th is the thermal resistance (K/W or ⁰C/W)

  ∆T is the temperature difference between two points (K or ⁰C)

  q is the heat transfer rate (W)

Thermal resistance is a convenient first order approximation because it almost resembles an Ohms law analogy to heat transfer, with analogous parameters:

• Electrical current is the heat transfer rate

• Electrical resistance/conductivity is thermal resistance/conductivity

• Electrical potential is the temperature

Heat sink fins can be modelled approximately using this model to estimate heat transfer. Of course, this approximation is valid if the thermal “circuits” are contained within a lumped approximation, where the heat flow is uniformly distributed in confined materials. Anything that is distributed or highly coupled will require sophisticated modelling.

Stress-Strain

All materials with a PCB stack experience strain / material deformation under some kind of loading. There are two major categories of materials:

• Ductile material - A material that initially deforms linearly as more stress is applied, then after a certain threshold, it undergoes plastic deformation over a wide range until it ultimately fails. This is the dynamic when you pull hard taffy or gum. Most metals fall into this category.

• Brittle materials - Brittle materials show a steep linear curve with a sharp fracture at the top. Glass falls into this category.

For ductile material, the left most region is governed by Hookes law which describes material deformation in the elastic region. It is given by

where

• σ is the average normal stress across a cross section area of a material

• ε is the average normal strain, give by (L - L0) / L0

• E is the modulus of elasticity (or Youngs modulus)

On the right side of the linear region, the material starts deforming and ultimately fails to a fracture. Of course, we hope that packaging materials don’t enter this region, but performance requirements can make it difficult to design around this.

Another form of strain, shearing, changes the angle between two normally perpendicular lines of a material. It is given by:

Bumps in between package layers experience both shearing and compressive / tensive forces.

There are a few material properties that matter:

• Youngs modulus - Describes the ability for a material to resist deformation when subjected to stress. Typical values include:

  • 6 for underfill epoxy

  • 8 - 22 for FR4

  • 131 for silicon

  • 132 for copper

• Poisson’s ratio - the ratio of the strains in the axial direction and the lateral direction. The max value is 0.5. Typical values include:

  • 0.1 - 0.14 for FR4

  • 0.28 Silicon

  • 0.35 for copper

• Creep - a time dependent permanent deformation of a material under a constant load or stress at a constant temperature. Creep is common at elevated temperatures.

• Stress relaxation - a gradual decrease in stress by a material under a constant strain over time.

• Viscoelasticity - a nonlinear, hysteretic curve to loading and unloading where a material recovers its shape over time after unloading

• Viscoplasticity - when a material suffers permanent, rate-dependent plastic deformation that does not recover

  • Viscoplasticity is important when describing the behavior of solder since it is subject to strain and creep deformation.

  • One popular viscoplasticity model of solder is the Anand model that governs the rate of change of the inelastic strain. It takes multiple effects such as strain rate sensitivity and deformation resistance into account.

• Coefficient of Thermal Expansion (CTE)

  • CTE is perhaps one of the most important and most talked about material properties for packages because it describes material behavior at the interface of thermal and mechanical properties.

  • CTE is a form of material “strain” where an increase in temperature causes the material to expand, and a decrease causes it to contract. It is given by:

    

    where

    • εth is the strain due to temperature with no mechanical load

      α is the coefficient of thermal expansion

      ∆T is the change in temperature

  • Common CTE values include:

    • 2.6 for silicon

    • 17 for copper

    • 20, 70 for epoxy

    • FR4:

      • 10, 15 in plane

      • 65, 180 in z-axis

  • For any given package size, material with a higher distance from the neutral point will experience higher deformation since it elongates / contracts more, thus increasing the stress on solder joints in that area.

Warpage

Now that I've established some thermal and mechanical fundamentals, we can discuss how these play a role in warpage, and failure modes after the paywall.

Share

Read more

Editors Note (6/22/26) - I am paywalling the last half of this post and rearranging the post outline at the beginning.

Before the paywall, I link several of my other posts in adjacent domains in power electronics, high speed communications, and signal integrity that related to packaging in some way.

In this post I’ll build intuition of the key tradeoffs and constraints surrounding advanced packaging.

• I’ll define each of the classifications at a high level (from 2D to 3.5D)

• Then I’ll do a breakdown of each of the key options of a TSMC 2.5D CHIP on WAFER on SUBSTRATE (CoWoS). We will examine this package architecture through the following lens:

  • TSV Fabrication

  • RDL Fabrication (with BEOL Process for Si)

  • Interconnect Options (C4 Bump, C2 Cu Pillar)

  • 🔒Substrate Options

    • 🔒Glass Substrate and the unique challenges: SeWaRe and bump reliability

    • 🔒Panel or Wafer - Why we didn’t do panels in the first place

    • 🔒Embedded bridge in Organic Substrate

      • 🔒TSMC CoWoS-L vs Intel EMIB-T

  • 🔒The Process Steps for Hybrid bonding and why it isn’t widely adopted

The reason I write this post is to establish some fundamentals of advanced packaging and communicate what a packaging engineer would like other engineers to know who are interfacing with packaging. My goal is to build on top of the knowledge on this post as well as my existing posts to be able to understand how to properly co-design electronics with the package.

Please note that I do not consider myself an expert in advanced packaging by any means. I reference John Lau’s slides from his seminar at ECTC 2026 and my overall conference experience to better understand the interactions with advanced packaging and other domains.

If you’re an expert and notice something incorrect, please reach out to me so I can have it promptly corrected.

Before I wrote this, I touched upon high level packaging issues regarding reliability, multiphysics simulation, and co-packaged optics in this post I wrote shortly after I attended ECTC 2026:

Advanced Packaging is like City Planning

Advanced Packaging is a lot like city planning:

• In rural areas, houses and buildings are distributed across a large plot of land (2D)

• In suburbs, houses and mid-sized buildings are bundled closer together and connected (2.1, 2.3, 2.5D)

• In cities, floors are stacked both above and below with means to access them (3D, 3.3D, 3.5D)

City planning is all about managing the flow of people to and from desired destinations.

Advanced packaging follows the same logic: signals need to get to and from compute devices. However, there is an an order of magnitude difference between the dimensions of the on-chip interconnect and the PCB/package interconnect. As speed and power requirements increase, more sophisticated and elegant means of package interconnect are needed to bridge this order of magnitude difference.

Just like any building, as you start stacking floors, the structural stability needs to be considered with appropriate safety factors designed in. This logic also applies to advanced packaging as factors like reliability, thermal, and mechanical characteristics become more important.

High Level Classifications: From 2D to 3D

There are several structural variants of advanced packaging. All of these build the system on top of a build-up package substrate on top of a PCB.

Why do you need a buildup package substrate? There are a few reasons:

• Package substrates help “bridge” the dimensional order-of-magnitude gap with the chip level interconnects and the board level traces. This is done through “fan out connections” where the package solder balls spread the signals out from underneath a chip.

• Mechanical Buffer. The package acts as a buffer to protect the chip against stress, vibration and thermal effects. The coefficient of thermal expansion is an important consideration. We’ll see how, quite paradoxically, the low CTE of glass can have a negative impact on reliability.

• Local interconnect amongst amongst chiplets. This is done through a combination of Re-distribution layers (RDLs) and Back-end-of-line (BEOL) metallization on Si Wafers

2D Integration

2D packaging represents the historical and architectural “baseline” packaging technology that more advanced variants build off of. In 2D we have the following classifications:

• 2D - the chip is bonded directly on top of a build-up package substrate.

• 2.1D - the RDLs (Redistribution Layers) are built as thin film layers directly on top of the organic substrate itself.

• 2.3D - The RDLs are fabricated separately on an organic interposer on top of a build-up package substrate. This is also known as the “hybrid approach”.

• 2.5D - The chips themselves sit directly on top of a massive, monolithic Silicon Core (Silicon Interposer) or a Glass Core (Glass Interposer)

In 2.1D and 2.3D, two popular materials used in the dielectric films are ABF and PID.

• Ajinomoto Build-Up Film (ABF) is a resin-based epoxy film embedded with silica fillers.

  • ABF has exceptional mechanical stability, excellent thermal resistance, and a low Coefficient of Thermal Expansion (CTE)

• Photoimageable Dielectric (PID) is almost chemically similar to photoresist; it is a liquid polymer or dry film that is chemically engineered to react directly to light.

  • PID can be made much finer than ABF, but is traditionally mechanically weaker than ABF.

There is a core mechanical stability vs feature size tradeoff between both films. ABF tends to be the dominant choice of packaging due to its ability to handle many metal layers.

2.1D / 2.3D are considered to be low cost alternatives to 2.5D, which is considered the standard for AI/HPC.

3D Integration

3D enables more computational density to be stacked in the vertical direction. The higher order classifications in 3D are still emerging and contested, but this is what Lau preliminarily defines:

• 3D - stacking chiplets directly on top of each other vertically on an active interposer (compared to 2.5D which is passive). These active interposers can contain IVRs, NoCs, BIST, or other peripheral circuitry that complement the functions of the SoC.

• 3.3D - consists of HBM on top of an SoC without a TSV interposer

• 3.5D - Involve hybrid bonding chiplets before integrating heterogeneously all components onto a single substrate.

As packages becoming increasingly integrated with the chip themselves, this leads to one of the biggest packaging challenges: whose responsibly should be for final assembly of the packaging: the fab, or the OSAT? Historically, both fab and OSAT specialized in different parts of the semiconductor production flow. The fab specialized in short pitch, finer interconnects and the OSAT specialized in higher pitch, coarser interconnects.

When hybrid technologies such as bridges and LSI in organic substrates come along, who will assume responsibility for final package integration? This is a question that is still being fiercely debated among the fab and packaging community and will be left for another time.

A Package Teardown: 2.5D

In the next section we will do a deep dive into the fabrication process of 2.5D and examine the fabrication process for each part. 2.5D is considered the package of choice for HPC due to its form factor and performance. For example, the NVIDIA H100 “Hopper” GPU uses this package and a cross section shows key features between the build-up package substrate and logic.

TSV Fabrication

A through-silicon via (TSV) transports signals and power completely through the thick structural floor (the raw Silicon Substrate). In CoWoS, the diameter is ~ 5-10um and the pitch is ~40-50um.

Lets go through the process steps:

• First, we start with standard lithography to add SiO2 on the Si surface that acts as the “hard mask” for TSV:

  • Deposit SiO2 by thermal oxidation of Plasma-Enhanced Chemical Vapor Deposition (PECVD)

  • Apply Photoresist

  • Pattern with lithography through a photo mask

  • Etch the SiO2

• After the SiO2 is deposited, the via hole is formed using Deep Reactive Ion Etching (DRIE). The most commonly used process is the Bosch process.

• After this hole is formed, it is filled with copper:

  • An SiO2 layer is added with PECVD/Sub-Atmospheric CVD to act an an electrical insulator against Si and Copper

  • Then, a barrier (Ti/Ta) and seed (Cu) added to prevent copper poisoning

  • Then, the hole is filled with copper, annealed to stabilize the grain structure, and polished with CMP.

Important thing to note: The more power TSVs you add to feed a power-hungry GPU, the more you “choke” the horizontal routing channels available for the RDLs to connect the GPU to its HBM memory.

RDL Fabrication

Then the RDLs are fabricated on top of the TSV using Standard BEOL Processes using dual damascene. Dual damescene refers to etching both the via and horizontal line trench before adding any metal. Heres how it works:

• Form the via and M1 trenches with standard lithography. This is the same process as with TSVs: Deposit SiO2 with PECVD + photoresist on top, etch the photoresist then the oxide

• Line trenches with a thin Ti/Cu seed layer with PVD

• Overfill the whole trench with electroplated copper

• Polish the excess copper on the surface with CMP to give you a planar surface

Compared to standard packaging methods that limit pitch to 2-5um, this method can get down to 0.4um (at the time of writing this paper).

A passive silicon interposer typically features 4 to 6 BEOL metal layers on the front-side (top), and 2 to 4 coarser packaging RDL layers on the back-side (bottom).

As more layers are added, film stress is added. With HBM4, foundries like TSMC and UMC are actively extending front-side BEOL capabilities to 8, 9, or 10 layers to accommodate the massive explosion of traces.

Interconnect Options

Bumps are formed by first forming an under/bump metallization with both a thin barrier layer (Titanium, tungsten, chromium) and seed layer (Copper/nickel). There are two major categories of bumps:

• C4 (Controlled Collapse Chip Connection):

  • C4 bumps are great for macro power delivery and board-level packaging

  • However, the pitch of C4 bumps are limited to 50-150um because they can cause “solder bridging” if they are too close

• C2 Pillar

  • The actual solder is just a tiny cap sitting on top of a rigid, electroplated copper that does not melt during reflow

  • These allow you to jam interconnects much closer together, but are much more expensive, primarily due to plating time of the Cu pillars

The other major option is Hybrid Bonding, which I’ll discuss at the end.

At higher current densities, both bump resistance and CTE affect reliability. I noticed a few posters at ECTC that demonstrate how a few small effects can magnify:

• Bumps further from the neutral point of a chip experience more deformation as substrate/chip area get larger

• The direction current flows into and out of the bump affects the equivalent resistance of the bump by a few percentage points

• The elemental composition at the bond interface can affect current density as well.

At higher current densities, these effects can lead to localized joule heating that can create “hotspots” and can increase the probability of failures.

Substrate Options

Silicon substrates are great for fabricating the fine pitch interconnect needed in between chiplets. However, silicon substrates are running into a few problems:

• They are expensive

• Fabrication is limited to major IDMs

• Substrates are running into photolithography reticle size limits

• Silicon everywhere is frankly overkill when the fine pitched interconnect is limited to specific areas in between chips.

After the paywall, I’ll discuss the following:

• Two alterative options to silicon: Glass and organic with bridges.

• Panels vs wafers, and why we didn’t do panels in the first place

• A brief history of hybrid bonding, the mechanism, and why its not widely adopted yet

Share

Here are several of my other posts I’ve written that provide the technology fundamentals of power, high speed communications, and AI compute that packaging is the physical container for.

Read more

Editors Note (6/22/26) - I am paywalling the last half of this post. Before the paywall, I link several of my other posts in adjacent domains in power electronics, high speed communications, and signal integrity that related to packaging in some way.

In this post I discuss the following topics:

• Clear definitions of five buzzwords used in advanced packaging from John Lau’s tutorial

• How Reliability is one of the biggest concerns for engineers on the floor

• 🔒The Wild West of Co-packaged optics - Different CPO Integration Options

• 🔒Why Multiphysics simulation is not that easy in practice

Hardware is becoming the new bottleneck in hyperscalers.

That was was the underlying message at ECTC 2026, the flagship electronics packaging conference, as stated in a plenary session by Dr. Tien Wu, CEO of Advanced Semiconductor Engineering, the largest OSAT in the world.

I attended ECTC 2026, not as a pure packaging engineer, but as a Analog/Mixed signal system architect who is curious about how advanced packaging is interrelated with other elements.

Before I came to this conference, I went in knowing a bit about advanced packaging from other writers. I studied mixed signal architectures in high-speed communications and power systems through self-directed conference travel to ISSCC, DesignCon, and APEC. I wrote comprehensive architecture breakdowns specifically to bring awareness to key constraints so that engineers can ask more informed questions earlier in the design process. I wrote about why I am doing this in my post:

As an EE myself, we sometimes take packaging for granted. However, ECTC made me realize the sheer amount of depth and mechanical considerations that go into packaging. The performance demands of GPUs are placing a lot of stress on the packaging itself and shifts the bottleneck toward hardware, not software, as the limiting factor in scaling AI compute performance.

In this post I’ll give a very broad high level overview of my experience at ECTC and the key challenges engineers on the floor are facing.

John Lau Tutorial on Advanced Packaging

At the beginning of the conference, I acquainted myself with advanced packaging trends through a tutorial given by John Lau. He is considered a legend in the packaging community because he wrote virtually every major book on advanced packaging and heterogeneous integration. He is very practical and as close to the manufacturing reality. His lecture was an experience; he tends to “beat” concepts into your head through repetition; I am not joking. I felt like I passed through a rite of passage.

He broke down the classifications (2D, 2.1D, 2.3D, 2.5D, 3D, 3.5D) as shown in the figure above, as well as described key issues in hybrid bonding and glass substrates. These topics are covered quite extensively by other writers and I will not go into depth here.

He notes that several terms in advanced packaging are often misused. He clearly defines these terms as the following:

He is also well versed in the second-order effects of manufacturability. He mentioned how larger panel sizes (600x600mm) that pack more parts per panel are not always better because it takes longer for the pick and place machine to traverse the distance to move the parts between panel and product.

After the seminar, I personally ordered up a book myself “Hybrid Bonding, Advanced Substrates, Failure Mechanisms, and Thermal Management for Chiplets and Heterogeneous Integration” (2025) to understand the interactions between advanced packaging and adjacent domains. I anticipate this book will strike a balance in describing the interactions with advanced packaging with other domains and will be a valuable resource. I’ll use the material in this book to inform my future writing.

In my view, there is a ton of noise and hype around advanced packaging. At the end of this post, I’ll also list some advanced packaging books by John Lau so that if you’re deciding to write or understand this space better, you have a source of truth to reference.

Two of the major ones high volume advanced packaging techniques are TSMC CoWoS and Intel EMIB, and is clear that Intel is becoming popular as supply chain bottlenecks for CoWoS are coming into play.

Reliability is one of the biggest concerns for engineers on the floor

Most people out there hyperfocus on the raw performance metrics of the GPUs, CPOs, the potential of hybrid bonding and glass substrates, or what Jensen is doing; they are clearly some of the flashiest subjects.

However, as GPU power increases demand more power and signal speed, these effects put more stress the package and “magnify” the existing failure mechanisms! Reliability is especially a concern as package sizes get bigger; they get heavier and introduce more points of failure.

Reliability is something historically thrown over the wall in the semiconductor industry as basically a cost center with a “line” to hit.

I was quite surprised by how much reliability was discussed among the conference among special sessions and technical sessions. There were a new notable observations amongst panelists in the special sessions:

• Tsun-yen Wu (in place of Kathy Yan), Department Manager at TSMC, pointed out two things:

  • First, there is a tradeoff between four domains in packaging:

    • Routability

    • Crosstalk (NOTE: This doesn’t appear correct; to first order, when spacing goes down, crosstalk goes up since there there is higher capacitance between the lines. However, lower line width decreases the fringing capacitance as well, so overall crosstalk depends on the dimensional scaling factors and also ground shielding used)

    • Structure stress/tension

    • Process difficulty

  • He also notes that many package designers tend to design their packages aggressively and not taking manufacturability into consideration early in the design, such as stacking as many vias as possible. This requires constant back and forth to ensure the package is manufacturable. This wastes communication time, especially across time zones, if a working knowledge of reliability is not known by packaging designers beforehand.

• Dr. Richard Rao, Distinguished Engineer at Marvell, notes that transient switching can cause stress, and that a change from a stress strain approach to fracture mechanics approach is needed.

• Gang Duan, EVP, Samsung, points out that embedding components in substrate is aggressive and can affect yield and performance

• Dr. Choong Un Kim, Professor, Material Science at UT Arlington, points out that joule heating leads to local stress which leads to cracking

I also learned of several failure mechanisms in advanced packaging. Here are a few examples:

Much like how we personally take reliability into account when we buy a car, investors and engineers should also take reliability of systems into account. You can have a well performing part or tightly integrated system that generates buzz, but if it causes you reliability problems down the line, it might not be the best long-term solution.

I do think reliability engineering will become much more important than ever, or at the very least, a topic board level engineers MUST know as systems become more tightly integrated. I think that reliability training will go a LONG way to help minimize the cost of constant back and forth as systems become more tightly integrated.

I dive into more depth regarding the fundamental structures and physics behind these failure modes:

The Wild West of Co-packaged optics

Co packaged optics is one of the hottest, yet diverse, topics of the conference. CPO consists of three major blocks:

• Photonic IC (PIC) - This consists of the photonic IC such as photodiode, laser, etc.

• Electrical IC (EIC) - This consists of the electrical circuits such as the transimpedance amplifier (TIA), laser driver, etc.

• Application-specific IC (ASIC) Switch

There are a number of clear performance advantages of CPO that make it such an attractive option compared to pluggable transceivers, such as lower length, reducing energy to drive the signal, and lower latency. Many content creators have written exhaustively on this.

Though the physics of devices themselves can be a bit tricky, the architecture behind optical communications has a simple mental model:

• You have a laser source that is routed through a PIC.

• This laser is generated from a VCSEL, micro led, or other laser source.

• Then, this laser source is modulated externally with either a MZM, micro-ring, or electro absorption modulator from a high frequency signal from an EIC.

• This signal is sent through the optical channel at a particular “band” (850n, O,C,L), detected with a photodiode, and amplified with a trans-impedance amplifier.

I cover these in more detail in my other posts:

What makes co-packaged optics so interesting and yet confusing are the vast amount of packaging techniques involved. Each system has a own unique set of requirements for high speed data rates, system constraints, and a power/ BER link budget to handle.

Share

For instance, here are a few options for 2D CPO structures with bridging and bumping techniques:

Read more

Editors Note (5/31/26): I wrapped up ECTC, so this post reflects my experience when I originally wrote this before ECTC.

I added details about key substack articles I wrote along the way, and a link to an Asianometry video on superconducting electronics to get a flavor of what I worked on.

I’m currently attending ECTC, the flagship electronics packaging conference, from May 26-29 in Orlando, FL, to learn more about advanced packaging. Packaging represents the “container” that the chip sits within and interfaces with most HW domains.

I’ll be writing more on advanced packaging after this conference, but first, I want to take a moment in this post to reflect on the past three conferences I’ve attended, why I attended these, what I learned, and the advice I would give to engineers thinking of doing the same.

Why ISSCC, DesignCon, and APEC?

I recently had the unique opportunity to attend three major technical conferences back-to-back: ISSCC, DesignCon, and APEC.

I selected these specific conferences because, together, they develop a holistic, cross-functional view of the critical hardware challenges facing today’s AI data centers. Specifically, I wanted to understand the abstraction levels of the mixed-signal portions of the data center, which bridge two traditionally isolated domains: high-speed communications and power electronics.

But, there is a deeper reason why I attended these conferences. I anticipate the biggest technology risk in advanced electronic systems is not understanding the interactions amongst domains as domains become more tightly integrated. Many leaders throughout my conference travels urge the need for more cross functional awareness and that silos will no longer cut it. Many call for a “digital twin” as a simulation model that models all of these interactions. This is much needed, but just like any simulation, its important to develop a solid theory of these interactions to be able to interpret the simulation results and propose fixes.

To be completely honest, taking this on conference gauntlet was brutal—at least at first. But it got easier over time.

Not many people get the opportunity to do this. Companies are often hesitant to send engineers to conferences due to cost, inertia, or fears of intellectual property leaks. Because my circumstances were unique, I had to self-fund this journey.

Obtaining Interdiscplinary breadth: From Design, Applications, and Validation

The catalyst for this journey came after I was let go from my previous job. However, I build up a rich set of experiences leading up to it.

I gained a ton of interdisciplinary breadth while attempting to scale up superconducting computing from low level IP to EDA integration. Check out this video by Asianometry for a more in depth look into this technology.

One thing I noticed in any novel deep tech is that there is a HUGE disconnect between the potential for a technology from the financiers and its actual technological readiness from the bottom. I have my fair share of war stories dealing with the integration challenges when taking thinking from CMOS specialties and applying that to scale the EDA flow of a low impedance, highly coupled technology with unique first principles constraints.

In addition to superconducting electronics, I also have pretty extensive experience doing IC design for mixed signal, board design for power, and audio applications. Its quite a well-balanced range of activities that gave me a wholistic sense of the design process.

However, I was unexpectedly let go at my last job. I won’t go through all the details, but in short, it turned out not to be a good role fit for my skillset they brought me in for.

Since then, I’ve taken the time to go on a year long sabbatical to not only expand my own knowledge, but also to share it with others. I also wanted to provide a mental map for other engineers to become more T shaped or directors who want to bridge domains and need a technical reference point to understand engineers concerns across domains. The industry is screaming for cross-functional awareness, and I decided to cultivate it in myself.

The Strategy and Preparation

Before diving into the conferences, I took a few months for personal travel around the world in places in Europe and Asia, including key countries in the semi supply chain like Germany, Japan, Taiwan, and Singapore. I did cultural tours in each place to gain a sense of the historical and cultural influences, such as how key events such as the 228 incident influenced Taiwans politics today. It was a much-needed mental break from work, but it isn’t a lifestyle I’d want forever—the logistics and constant travel become a hassle.

When I returned, I committed to building my Substack. Over the previous year, I had accumulated a wealth of insights from technical textbooks and business books, including books on strategy, business, VC, and innovation. Check out some high level overviews for the some of the books I read and highly recommend:

Before I attended the technical conferences, I first started writing foundational, subsystem-level posts covering ADCs, PLLs, bandgaps, and AI accelerators from textbooks and key papers.

Building this technical baseline gave me the mental models I needed to absorb the conference material, starting with the toughest one of all.

1. ISSCC (International Solid-State Circuits Conference)

ISSCC is considered the “Olympics of chip design.” The overarching theme was “Advancing AI with IC and SoC Innovations,” focusing heavily on high-speed optical and wireline systems and associated high speed subblocks. It is heavily academic, dominated by professors and PhDs from top-tier universities.

Holding a Master’s degree myself, I’ll be honest: I would NOT recommend ISSCC for a first-timer. It was rough and felt like I was drinking from a firehose!

The plenary sessions covered a broad range of topics, including a talk given by Hope Giles, VP of PM at Apple, on why systems thinking and cross functional thinking are critical skills. Bernhard Wicht won a “Best book award” for his book “Design of Power Management Integrated Circuits” which is one of the best analog design books out there for not just PMIC design but also circuits that share the same blocks.

I purchased an All-Access Education Pass, which granted me entry to 10 tutorials and 6 forums/short courses, in addition to the technical sessions. While my previous writing on PLLs and ADCs helped me follow some of the technical sessions, there was still a massive gap in technical depth that made it hard to follow the advanced papers for long.

However, the tutorials and short courses were incredibly valuable. I attended deep dives on biomedical circuits, logic and memory, AI accelerators, and clocking/CDR techniques for wireline transceivers. The forums comprehensively covered both the power and high speed optical communications side of AI data centers, as well as a short course on optical communications. I also caught a fascinating panel on brain-machine interfaces featuring Jan Rabaey and the CEO of Neuralink.

As a result of this conference, I synthesized this firehose of information to write primers on optical communications and high-speed SerDes—creating the exact frameworks required to understand complex ISSCC papers.

2. DesignCon

DesignCon is the premier high-speed communications conference, famously themed around “where the chip meets the board.” While ISSCC is more posh and focuses on on-chip architectures, DesignCon is more laid back and focuses on the board-level implementation of high speed communications. It was a bit more industry-focused and accessible, though still a challenge coming the following week after ISSCC.

The event kicked off with an excellent tutorial by Dr. Mike Peng Li, a foundational figure in jitter and signal integrity. he has written extensively on jitter. Check out this chart of his that highlights all the possible sources of jitter:

The core technical sessions revolved around the practical challenges of handling 224 Gbps signaling and the scaling paths toward 448 Gbps. There was a huge variety of topics from co-packaged optics, power integrity, EMI mitigation, and AI applications in signal integrity. The massive expo hall was particularly enlightening, showcasing real-world in-rack and interconnect solutions. I learned about the existence of many lesser known companies in the high speed communications value chain such as Molex and Samtec.

The result of this experience allowed me to write my deep dives on high-speed signal integrity and silicon photonics. I don’t think every engineer needs to know every single source of jitter, just the fundamental concepts of DJ and RJ.

3. APEC (Applied Power Electronics Conference)

APEC is the flagship conference for power electronics. I kicked things off by attending three tutorials specifically focused on power delivery for AI data centers, as well as digital controllers for power electronics.

A massive portion of the conference focused on optimizing the data center power chain. Sessions broke down every major conversion step: 800V to 48V, 48V to 12V, 12V to 1V, and various intermediate bus voltage permutations. To be fair, the material felt a bit repetitive after a while; once you understand the underlying power architecture, many papers are just permutations of the same fundamental topologies. There was also a heavy focus on wide-bandgap semiconductors like GaN and SiC, as well as a focus on optimizing magnetics.

Check out this image from STMicroelectronic’s Booth on the power architecture:

I really appreciate the power domain because the board structure is a tightly coupled system. It forces designers to have a holistic sense of how every single component interacts. It’s an area you can nerd out on for an entire career because there is always a creative optimization trick to find.

This conference led to my article, “AI Data Center Power System: A Complete Overview.” It maps out the intricate tradeoffs between converter topology, footprint, and power delivery network (PDN) variables, explaining not just what the architecture is, but why it has to be that way.

Advice for Engineers

Navigating these conferences as an independent engineer taught me several lessons about career development, corporate dynamics, and learning strategies:

1. Try to get your company to pay first (and how to pitch it)

If you want to attend a conference, ask your company to sponsor you. Even if its for your own benefit, you should frame the request entirely around how the knowledge benefits the company. Sometimes management is simply unaware of the existence of these conferences, or might not see it as a big priority in your day job. Offer to give an internal presentation to your team afterward. This shows you want to distribute the value, rather than “hoarding” knowledge for your own resume.

2. Navigate corporate sensitivities carefully

Remember, your company’s name is printed on your badge. Have a clear understanding of who your competitors are before walking in. When networking, err heavily on the side of caution and silence. It is incredibly easy to let an innocuous statement about your current project slip—especially when alcohol is being served at networking events. Competitors can easily piece these details together into competitive intelligence.

3. If you self-fund, prepare in advance

If your company won’t pay and you decide to fund it yourself, make sure your finances are secure (I relied on index fund savings and 10 years of industry experience to comfortably fund this).

However, attending a few conferences is not something worth quitting your job over. If your company refuses to pay, it might be worth using vacation time and a bit of your own money. This money should be viewed as an investment in your own education.

Self-paying also has the added benefit of forcing you to take the conference more seriously. I found myself more actively engaged in the conferences, sitting near the front of most sessions, diligently taking notes. Its not the same experience as when a company pays for you to go because you have more skin in the game.

4. Consider if you’ll gain more value cheaper from purchasing and studying technical books

Conferences are excellent sources of truth for what is going on in the industry. With that said, conferences are probably not the first thing you should do until you have good fundamentals of the background technical material surrounding the conference. You might be better off going to an engineering library, picking up a few excellent technical textbooks, and thoroughly outlining them. I spent a year doing exactly that before I ever stepped foot in these conferences. That helped to close the knowledge gap with the conference material that builds up top of the fundamental textbook material.

Check out my post “Learning Analog/ Mixed Signal Design the efficient way” for my methodology that can apply to other technical areas.

5. Aim to become “T-Shaped”

As a near term career goal, try to attend one conference inside your core domain, and one that is interdisciplinary or adjacent to it. This builds a “T-shaped” skill set (deep expertise in one area, broad literacy across others). If you are looking for excellent interdisciplinary conferences, check out the ones I mentioned, but also consider:

• VLSI Symposium (excellent to pair with a vacation to Japan or Hawaii)

• DAC (Design Automation Conference)

• SEMICON (various locations)

• IEDM - IEEE International Electron Devices Meeting

6. Lean into the haze

Don’t be discouraged if you feel overwhelmed or hazy during the technical sessions. Even though I prepared quite diligently and my published posts look polished, I really felt incredibly out of my depth during the events at first. I needed weeks of breathing room afterward to digest the material. Take full advantage of the tutorials at the start of the week, and remember that you get access to the conference digest (papers and slide decks) afterward. Don’t put too much pressure on yourself to fully understand the technical sessions.

However, it does get easier over time because you’ll learn how to prepare a game plan to frame the material in the conference better.

Real growth happens when you sit slightly outside your comfort zone and process the gaps in your knowledge after the chaos subsides.

Conclusion

Its unfortunately very common that many companies nowadays are very execution focused and push the burden of responsibility on you to develop your own career. When I ran into organizational barriers trying to add value inside my company and let go, I decided to use the unemployment time as a “sabbatical” and add value to a broader audience.

Ultimately, the best path is to figure out how to add value to your current company in a way that is mutually beneficial. If you find yourself in a mismatched role, don’t just check out. Try everything in your power to add value to other around out first.

With that said, you’re ultimately responsible for investing in your own education. I, and many others, can provide this information, but cannot make you learn it if you don’t seek it out. Take coursera courses. Take advantage of engineering libraries. I would literally go to an engineering library and just casually browse through books and make notes on ones to dive deeper into later.

But just don’t hoard the insights for yourself. Find ways to share those insights with others so they might benefit as well. As they say, you learn 90% of the things you teach others.

Share

Here are some of the articles I was able to produce as a result of these conference travels, including the methodology I used to self-study:

But I wonder to myself, with all the risks of AI, why do people love AI so much? I contend that we are deep in the throes of cognitive offloading. We are outsourcing our thinking to agents to reclaim time and money, especially as companies face financial pressures to adopt it.

But as AI moves into the “reasoning era,” we have to ask: Will this offloading eventually atrophy human cognition?

My theory is that our intellectual survival depends on how much intellectual agency we maintain over quick dopamine hits and the comfort of not learning.

The Danger of Cognitive offloading

Cognitive offloading isn’t inherently “bad.” Humans have always outsourced cognition—we use calculators for math, GPS for navigation, and specialists (doctors, lawyers) for expertise we can’t master ourselves. There is simply too much information in the world for one brain to hold.

However, offloading to AI carries unique risks to human agency.

Overconfidence

There is a massive gap between interfacing with AI and understanding AI.

This is a vitally important point because AI has caused people to be overconfident about their knowledge of AI. When people claim to know AI, they are knowledgeable about the latest tools used to interface and control AI. These tools also have a very low barrier to entry to them. Anyone can learn Python and ML techniques in a reasonable amount of time. They might also be familiar with basic concepts such as neural networks.

Knowing AI requires understanding the mathematical scaffolding—linear algebra for parallelization, calculus for derivatives, and discrete mathematics. It also requires you to have knowledge of key technical papers and books that lay the foundation for human cognition.

Take, for instance, the Bellman Equation, a cornerstone of reinforcement learning:

When we treat the output as magic without acknowledging the math, we develop an unearned confidence. We mistake the tool’s competence for our own.

Hollow thinking

In software engineering, there is a phenomenon called "tutorial hell"—you follow a guide, feel like a genius, but realize you can't build anything from scratch once the instructions disappear.

AI creates a similar trap. Because LLMs have a low barrier to entry, we often skip the "hard work" of forming mental models. If you ask an AI to write your strategy memo, you aren't doing the heavy lifting of synthesizing information. You end up with a hollow body of knowledge: you have the result, but not the neural pathways required to defend or evolve it.

Cognitive Surrender

The ultimate risk is cognitive surrender. We already see this with "algorithmic feeding" on TikTok or Netflix, designed to exploit our neurochemistry. When we stop choosing what to consume and let an agent decide for us, we trade our agency for dopamine.

The Solution: Reclaiming Intellectual Agency

The solution to this is to embrace curiosity, expand your mental bandwidth, and embrace the discomfort of learning.

Prioritize Curiosity over Ambition

Most self-help content out there promotes developing motivation and ambition. This is good for people starting, but can quickly become a “tutorial” hell where people consume videos to feel good rather than doing something about it.

Ambition is important, but a narrow focus on ambition often narrows your focus to surface-level tactics to project it. It can also lead to “hustle-type” behavior that don’t create long term value.

Curiosity does the opposite: it forces you to be foundationally sound. It allows your brain to ask yourself how do things in the world work, admit to your self what you don’t know, and lets you seek out meaningful information.

Curiously can look like going to a library or bookstore and skimming books and seeing which books best satisfy one curiosity to know how the world works or how to operate in a specific niche. It allows you build mental models of how the world actually works.

Curiosity allows the knowledge accumulate so that when interfacing with AI, you can question why the AI gave that answer, rather than just hitting "copy-paste."

Expand Your “Mental Bandwidth”

Agency is the capacity to act independently, but it is easily eroded by stress and financial pressure. When you are stuck in a cycle of “survival mode,” you are more likely to surrender your thinking to the easiest path (the algorithm).

One way to practice “negative bandwidth” protection is to control your spending and limit mindless consumption. Most marketing is designed to tap into your insecurities; by opting out, you reclaim the mental energy needed for deep thought.

Embrace Cognitive Friction

Cognitive friction is that uncomfortable “blank” feeling when you encounter information that doesn’t fit your current mental model. In the AI era, we try to actively avoid this feeling.

Instead, embrace this discomfort. That friction is new neural connections forming. As an adult, this can feel embarrassing—like we “should” already know these things. But whether it’s learning geopolitics or advanced physics, leaning into the discomfort is the only way to ensure the knowledge actually “sticks.”

How to Learn Efficiently Alongside AI

I’d say that self-directed learning is the most valuable skill of the 21st century because knowledge of how to use new technologies is not quickly feeding back into education and academia to inform the next generation of workers properly. Here is how I suggest balancing AI utility with human growth:

Learn foundational material from the bottom up

Don't just learn the "how," learn the "why." If you’re using AI for data science, go back and brush up on the underlying calculus or statistics. Work backward from what you don't understand.

Stretch Your Cognition

Put yourself in situations where the knowledge is just out of reach—a new role, a technical conference, or a difficult book. But make sure the information is not completely “over your head” - ensure that the difficulty level is slightly higher so that you can raise your level gradually.

Use AI As a “Critic” against your thinking

Instead of asking AI to write your explanation, write it yourself first. Then, feed your thoughts into the AI and ask it to critique your logic or find gaps in your mental model.

Conclusion

I’d argue that the world will eventually belong to the generalists—those who can use AI effectively to bridge gaps — but possess the foundational curiosity to know when the AI is hallucinating or oversimplifying.

If you want to learn more about the technical (such as transformer architectures) and corporate forces (such as DeepMind vs OpenAI), check out my other post here:

If you want to learn more about the Hardware Side of AI Accelerators (such as DRAM/SRAM, MAC operations, and a foundational architecture, Eyeriss), check out my post here:

Editors note (5/31/26) - Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

Before each paywall, I’ll post links to some of my other adjacent posts that give you a well rounded flavor of the interactions amongst other domains.

Silicon photonics is one of the hottest and fastest moving markets to address high speed data demands in co-packaged optics (CPO) in AI data centers.

In this post I’ll cover a few important high-level details of silicon photonic circuits. These include:

• An Overview of the Link Budget and important considerations:

  • Laser Power

  • Connector Losses

  • Receiver Figures of Merit

• 🔒Key Figures of Merit for eye quality

• 🔒Validation Challenges (wafer screening and thermal stability)

  Inside the Silicon Machine is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Link Budget

Understanding link budgeting is important to develop a system level understanding of a SiPh transceiver. Consider a typical electro-optical link with an external modulator:

At a high level, the job of a transmitter is to transmit the data at the specified data rate (224GPBS) with as low BER as possible in the given specification. Key system level performance specs include:

• Power efficiency: The ratio of energy consumed to the amount of data transmitted. This is usually measured in pJ/bit.

• BER: Bit error rate. Noise, loss and other effects degrade bit error rate. Requirements range between 1e-4 and 1e-6 depending on the standard.

A laser source (whether that be generated on board or coupled in externally) provides the power and is sent through a lossy modulator, connectors, and waveguides.

This power is then sent through the optical channel, converted back to a weak electrical signal through a photodetector, and amplified by a transimpedance amplifier. The signal at the RX need to be high enough compared to the noise in order to be able to detect the signal properly.

All of the sources of loss in between the TX and RX must be quantified for in a link budget in the optical chain. Lets break down each part.

Lasers

Lasers are either integrated within the PIC or coupled in externally. These external lasers are coupled into the chip through edge couplers or grating couplers.

There are a few important specifications:

• Power: The amount of coupled power is limited by two-photon absorption and other non ideal effects. Shown is a sweep of Pin vs Pout where Pout saturates and a high enough Pin can cause passivation melting:

  

• Relative intensity noise (RIN) - Rin refers to the relative intensity fluctuations of the optical signal of a laser source. This is a critically important parameter in PAM4 schemes because PAM4 has much lower noise margin than NRZ. A high RIN raises the noise floor causing a higher BER. RIN is given by:

Companies with lasers in the value chain include Lumentum, Broadcom, Coherent, Sivers, Scintil, Xscape, and Quintessent.

Losses in Connectors

There are several loss sources in the connectors themselves. These include:

• Modulator loss - due to intrinsic optical absorption and scattering

• Multipath Interference (MPI) - Optical reflections from multiple connectors / discontinuities

• Polarization Dependent Loss (PDL) - Measures loss due to change in signal polarization

Receiver

On the receive side, the signal is detected by a photodetector. This weak signal plus a noise signal is converted to a voltage through a transimpedance amplifier (TIA) that converts current to voltage.

A few key specs here are important:

• Responsivity - Measures how effective the photodetector converts photons to electrons. This is typically measured in A/W.

• Dark current - Measures the current flow when no light is present. Defects can cause high dark current that adds noise to RX and reduces sensitivity.

• Input-referred noise - assesses the noise floor of the TIA.

• Transimpedance Gain - measures the electrical strength of the receiver. TIA gain is very important because if TIA gain is too low, output voltage swing will not be high enough to compensate for noise. It is given by:

• Receiver sensitivity - the minimum optical power required to achieve a specific BER target. This is determined by plotting RX BER vs OMA. BER goes down when received optical power goes up.

• THD (Total Harmonic Distortion) - quantifies the non-linearity of the TIA. This affects PAM4 because it affects how even spaced out each of the four levels are after optical to electrical conversion.

Eye Quality

After the paywall I’ll go into key parameters that define the eye quality, practical challenges when scaling production of SiPh, and NVIDIA’s approach to thermal tuning design.

Share

If you’re curious to learn more, check out my other posts here:

Read more

However, leaders all suffer from human biases and tendencies, including:

• The tendency for linear thinking and templates in planning (SWOT, vision, mission, values)

• Tendency to reward people similar to themselves in business culture

• A disconnect between how high level business decisions affect people with boots on the ground

To establish a broad understanding on how engineers can lead, I’ve selected five books that cover a large breadth of challenges in the organizational space.

High Output Management

Written by the legendary former CEO of Intel, this is considered the “Bible” of Silicon Valley operations. Andy Grove treats management as a production process, using the “Breakfast Factory” as a metaphor for high-stakes manufacturing.

He establishes a few key responsibilities of managers in high-tech organizations:

• The Concept of Leverage: A manager’s output is the output of their organization plus the neighboring organizations they influence. A managers job is to focus on “High Leverage” activities—those where a small amount of their time produces a large amount of results.

• Task-Relevant Maturity (TRM): There is no one-size-fits-all management style. Depending on the TRM, new people might require hand on training, whereas more experienced people don’t need to be handheld as much. It is the managers job to ensure that people are adequately trained for the job they are in.

Grove observed that all large organizations with a common business purpose eventually become hybrid/matrix organizations with dual reporting to both functional and project managers. This structure allows a huge amount of flexibility to match talent to business needs, but this requires a strong corporate culture to manage dual reporting.

Good Strategy Bad Strategy

Rumelt argues that most “strategy” is actually just “bad strategy”—a collection of buzzwords, goals, and ambition, and the use of templates with vision, mission, and value statements. It is what leaders tend to revert to when they don’t want to deal with tricky fundamentals and risk tradeoffs.

Bad strategy can manifest at higher level positions because of things we tend to perceive and reward in business culture - sales ability, visibility, or pure business knowledge without much technical depth.

“Good strategy” is more intellectual in nature, and involves a “kernel” that consists of three components:

1. A Diagnosis: Defining the nature of the challenge.

2. A Guiding Policy: An overall approach to deal with the obstacles identified.

3. Coherent Actions: Steps coordinated with one another to support the guiding policy.

Good strategy involves viewing the organization as a “chain link” of interconnected functions that are hard to replace. An organization is only as strong as its weakest subunit. In semis, if your EDA tools are world-class but your packaging tech is mediocre, the entire product fails.

To increase performance, you must integrate components more tightly, not just improve them in isolation, so good strategy can involve strategic acquisitions that enhance this chain link.

Good strategy is hard because organizations are resource limited and it can involve choosing between a number of lucrative opportunities. To concentrate on one pivotal objective requires the courage to ignore a dozen other “important” things.

Innovators Solution

Speaking of bad strategy, there are no buzzwords more frequently abused than the words “disruptive” and “innovation” because everyone wants the benefit of these, but few people want to bear the risk involved.

If The Innovator’s Dilemma explained why large companies fail, The Innovator’s Solution explains how companies structure themselves to overcome this dilemma. This books highlights the key difference between conventional business processes which are deliberate, vs innovation processes which are emergent.

It advocated spitting business into into execution and R&D teams:

• Execution teams focus on core revenue generating business with low risk

• R&D teams incubating new ideas and technologies with high risk

On the product side, one core theme as it applies to complex engineering products is the concept of Modular vs. Interdependent Architectures: Early in a product’s life, when performance isn’t “good enough,” interdependent (proprietary) architectures win because they allow for performance optimization and control of all the components. Once performance exceeds market needs, the industry shifts to modular architectures that involves “outsourcing” components with industry-standard interfaces. This can lead to commoditization.

One major topic Christiansen advocates is avoiding the bias to favor “Right stuff Criteria” when evaluating managers for new growth businesses. These are mental heuristics used to evaluate people in corporate contexts among a large body of candidates, such as prestige of the company, the school they went to, whether they are a good communicator, results-oriented, decisive, and have good people skills. However, the downside of “right stuff criteria” is that managers can succeed for reasons not of their own making and the tendency to favor success in low-risk contexts. Instead, potential managers should be asked what problems they have wrestled with in the past.

Christiansen also talks about “good” and “bad” money involved between core revenue generating businesses and innovative businesses. This refers to the risk vs return expectations tied to the money that the shareholder or investor gives. Most corporate finance literature advocate for minimizing risk in the core business since shareholder value is largely tied to it. However, innovation requires risk distributed among a number of “bets”. When a core business sags, “good money” can turn into “bad money” when corporations demand that new ventures become “big, fast”. This “impatient capital” forces startups to bypass the learning phase, often leading to failure.

Thinking in Systems

Humans tend to feel a comfort in linear thinking - of thinking in terms of cause and effect. Frederick Taylor, one of the pioneers of scientific management, strongly advocated for breaking down complex tasks into smaller, specialized roles with rigorously defined processes for each.

However, this type of thinking can narrow ones view to complex interdependencies involved. Semiconductors are perhaps the world’s most complex system of feedback loops not only in the circuit themselves, but within the organization and supply chain.

Donella Meadows provides the tools to understand non-linear cause-and-effect with a few key concepts:

• Stocks, Flows, and Feedback: A system consists of elements (stocks), interconnections (flows), and a purpose. In our industry, the “stock” might be engineering talent or wafer inventory, and the “flow” is the rate of hiring or production.

• Feedback Delays: Because of long lead times in fabs, by the time a problem becomes apparent, it is often too late to solve easily. Leaders must “get the beat” of the system and look for “leverage points” where small shifts in structure cause big changes in behavior.

• The Trap of Sub optimization: When a subsystem (like a specific design team) optimizes for its own goals at the expense of the total system (the final chip’s power/performance), the whole organization suffers.

Reframing Organizations

“Reframing Organizations” by Bolman and Deal is one of the most influential leadership books that is commonly read in top MBA programs.

This book establishes four possible “frames” for leaders to view an organization through. These frames provide a framework for leaders to balance the coordination of the tasks involved with the psychological tendencies and needs of the humans that operate in organizations, including the desire for power and meaning. These four frames include:

1. The Structural Frame (The Factory): Focuses on goals, roles, and technology. It assumes the organization is a machine designed to achieve a specific task.

2. The Human Resource Frame (The Family): Focuses on the relationship between the organization and the people. It assumes that if people feel competent and supported, they are more willing to give more to the company.

3. The Political Frame (The Jungle): Sees the organization as an arena for competition. It recognizes that resources are scarce and power is the primary currency. Leaders here must be negotiators and coalition-builders.

4. The Symbolic Frame (The Theater): Focuses on meaning, ritual, and culture. It recognizes that in a world of ambiguity, people need a “vision” and symbols to give their work purpose.

Having knowledge of these frames allows leaders to be flexible in creating a culture that enables production. It also allow leaders to “reframing” to turn around failing organizations - that is, when organizations are facing issues in, say, structure and politics, leaders can emphasize the human resource and symbolic side to generate enough buy in amongst workers.

Conclusion: The Holistic Leader

Engineering leadership requires having a broad organizational knowledge of how peoples efforts are organized to achieve a goal. They must make decisions and set standards that balance the market needs and the psychology of the people within the organization.

All of the books I mentioned provide a foundational overview into the type of thinking leaders must face. Leaders might be concerned with how projects are being managed from within, the overall business strategy, how innovation is being handled, and whether all the parts of the organizations are being optimized properly.

Top leaders might start their morning in the structural frame (analyzing fab yields), move to the political frame (negotiating with a geopolitical partner), then the human resource frame (understand what motivates your workers to achieve), and end the day in the symbolic frame (rallying the team around a breakthrough).

To lead in this space, semiconductor marketers must possess a rare combination of skills: an understand of what key performance metrics customers are willing to pay a premium for, the ability to translate complex capabilities into business value, a time-to-market instinct, and a holistic grasp of the supply chain involved.

To understand why the industry is shaped the way it is, I have mapped out five foundational books and three key themes from each. Together, they form a cohesive foundation for understanding the move from disruptive innovation to the brutal realities of global economies-of-scale.

This post is designed for readers who want to gain a feel for to “why” the semi value chain is the way it is due to the complex interdependencies. I’ll keep a high level discussion here; refer to the books for more detail.

Innovators Dilemma

Innovation is a trap. Christensen’s core thesis is that “great” companies fail not because they are poorly managed, but because they are too responsive to their most profitable customers. He hits upon three key themes:

• Sustaining vs. Disruptive Innovation: Large incumbents excel at sustaining innovation—incremental improvements for high-margin tiers. However, they often ignore disruptive innovation—low-end, lower-performing tech that eventually scales to intersect and destroy the incumbent’s market.

• The Resource Allocation Trap: In semiconductors, where gross margins of 50%+ are the gold standard, executives are incentivized to “eat their way up” the value chain. This leaves the “bottom” of the market open for new entrants who can scale faster than the incumbents can pivot.

• The “Right Stuff” Bias: Organizations often use “heuristic shortcuts” for hiring and promotion, favoring managers who maximize short-term ROI. However, these are often the wrong people to lead a disruptive unit that needs to focus on learning rather than just execution.

Crossing the Chasm

If Christensen explains why firms fail to innovate, Geoffrey Moore explains why innovations fail to sell. The “Chasm” is the massive psychological gap between Early Adopters (who see the performance benefits) and the Early Majority (who want to follow the market leader). For products to cross this chasm, Moore proposes three key marketing strategies:

• The Beachhead Strategy: In a mainstream market, the early majority delays buying until they see a clear leader. To win, market share, companies must secure a “beachhead”—a specific, referenceable pragmatist customer base that allows them to “own” a niche before expanding.

• The Whole Product Concept: Marketing in semis often fails when engineers tout specs (FLOPs/TOPs) instead of solutions. Pragmatist customers buy the “whole product”—the silicon, the software stack, the documentation, and the support.

• Positioning as a Heuristic: Positioning isn’t about what the product is; it’s about creating a mental shortcut for the buyer that differentiates your product relative to the competition.

  • Example: NVIDIA’s early success wasn’t just about GPU architecture; it was about positioning themselves as the “only way” to play Doom or run professional CAD, creating a marketplace where they were the only reasonable choice.

Fabless: The Transformation of the Semiconductor Industry

This book tracks the “raw” history of how the industry moved from Integrated Device Manufacturers (IDMs) to the specialized, horizontal ecosystem we see today.

• The EDA & IP Revolution: The rise of Electronic Design Automation (Cadence, Synopsys) and reusable IP (ARM) decoupled design from manufacturing. This allowed “Fabless” companies to focus on architecture while outsourcing the capital-intensive fabrication.

• The Rise of “Coopetition”: As technology advances, no single company can do it all. We see increasing interdependencies: Samsung is Apple’s fiercest rival in handsets but its most critical partner in the foundry/supply chain.

• The System-in-a-Package (SiP) Era: We are moving past the “single chip” era. Modern high-end silicon (like FinFET-class chips with 3D memory stacks) requires a massive ecosystem of specialist companies to collaborate just to get a single product to market.

Chip War: The Fight for the world most critical technology

Strategy does not happen in a vacuum. Miller maps the industry through the lens of “Geopolitical Chokepoints”—specific nodes in the supply chain where a single entity holds disproportionate power.

• Mapping the Nodes: The supply chain is concentrated in specific geographies: EUV lithography in the Netherlands (ASML), manufacturing in Taiwan (TSMC), and EDA tools in the U.S.

• The Silicon Shield: Taiwan’s dominance in high-end manufacturing creates a “shield” where the global economy is too dependent on their output to allow for instability.

• Sovereign Silicon: We are seeing a shift where “marketing” a semiconductor company now involves navigating national security interests, as countries race to reverse-engineer, replicate, or replace foreign technology.

Apple in China: The Capture of the Worlds Greatest Company

Apple is the ultimate case study of all the previous concepts clashing in real-time. It is a masterclass in managing the “value network” across geopolitical lines while fighting the “innovator’s dilemma.”

• Zero-Defect Mentality: Apple’s success is built on a “zero-defect” mindset and a brutal Custom Silicon Management (CSM) process. They train suppliers to scale quality, then “pit” rivals against each other to drive down margins.

• The China Challenge: Apple highlights the tension of being a Western company deeply integrated into the Chinese ecosystem. While they benefit from the scale of partners like Foxconn, they face “reverse-engineering” threats from local giants like Huawei.

• Serialization as Control: To maintain their brand and deter bad actors, Apple serializes almost every component. This is a marketing and supply chain tactic to “own” the customer experience—even if it creates friction with right-to-repair advocates.

Conclusion

Understanding the semiconductor market isn’t just about knowing the “what” (the specs); it’s about knowing the “why” and how all the elements (marketing, geopolitics and technology) are interconnected. We see a map of an industry that is simultaneously the most globalized and yet the most protective on earth.

Leave a comment

Demis Hassabis is perhaps one of the most important influences in Modern AI that isn’t a household name.

Modern AI is not just a story of better models. It was shaped by the interaction between scientific ambition, commercial urgency, safety concerns, and the compute power available to model it.

Every AI enthusiast should read “The Infinity Machine” to better understand the development of modern AI. It tells the story of Demis Hassabis, CEO of Google DeepMind, from his origins, to his founding of DeepMind and its acquisition by Google.

I’ll highlight a brief history of AI through the formative years of Demis Hassabis, and cover the key technical developments and organizational challenges along the path to modern AI, particularly around AI safety.

I’ll also dive into the key papers addressed in the book that influence modern AI today such as deep reinforcement learning, transformer architecture, and chain-of-thought prompting that approximate human intuition.

Demis Hassabis

From his early days Demis Hassabis is a modern example of a modern day Polymath, with broad-ranging interests from sci-fi, philosophy, computer science, and neuroscience.

His entire career has been dedicated to developing artificial general intelligence (AGI), a hypothetical type of AI that matches or exceeds human cognitive abilities across a wide range of tasks. His dedication to this mission has him referred to as an “authentic entrepreneur” by VCs because he is genuinely dedicated to a cause, as opposed to entrepreneurs fishing for ideas to strike it rich.

We’ll see through his formative years in video game development about how he distinguishes between early machine vs human intelligence:

• Early machine intelligence uses deductive, first order reasoning to reason through problems

• Human intelligence generally uses induction or high level pattern recognition to identify shapes, sounds, and reason through sentences

Formative Years

Demis had a fairly modest upbringing in North London. When he was young he was highly influenced by classic Sci-Fi: “Enders Game”, “Foundation”, “Godel, Escher, Bach” and Ian Banks “Culture” series were some of his early influences. He was a chess genius and video game enthusiast in his early days, frequently ruminating on what algorithms are needed to program a machine to play chess.

He won admission at University of Cambridge to study CS, but he was too young at the age of 16. During this time gap, he worked on video game development at Bullfrog with Peter Molyneux to develop video games, including Theme Park.

After he graduated, he founded Elixir Studios in 1998, a London-based video games developer, and work on developing video games that focused on complex decisions on the user’s part, including Republic: The Revolution and Evil Genius.

Elixir Studios ultimately failed in April 2005, at which point he decided to pursue his PhD in neuroscience at University College London (UCL) from 2005 to 2009. His studies focused on the field of imagination, memory, and amnesia.

After he obtained his PhD, Demis Hassabis founded DeepMind Technologies in London in 2010, a ML AI startup with Shane Legg and Mustafa Suleyman. The goal is to combine insights from neuroscience with ML and computing hardware to create new algorithms to advance toward AGI. His company was successful at training a Deep Q-Network to play Atari games in 2013. [1]

In 2014, Google purchased DeepMind for $400 million where it remained a separate, siloed entity from Google. Since being purchased, it racked up a list of accomplishments from 2014 to 2020:

• It created a clinical alert system, Streams, for acute kidney injury in the UK. While well intended, it was controversial because of concerns over patient data governance.

• It created AlphaGo, a program that played Go. It defeated world champion Lee Sedol in 2016 and several other high level players with moves that humans had very little intuition for.

• It created AlphaFold and AlphaFold 2 to take on the CASP protein-structure prediction competitions. AlphaFold 2 achieved a breakthrough at CASP14 in 2020. In 2021, DeepMind and EMBL-EBI launched the AlphaFold Protein Structure Database, and by 2022 it included predictions for more than 200 million protein structures. Hassabis later won a Nobel Prize in Chemistry for his contribution, along with John Jumper and David Baker.

AI Safety: Commercial time-to-market vs Research Discovery

While DeepMind racked up an impressive list of accomplishments in its early years, it entered a ferocious competition to develop LLMs against OpenAI, led by Sam Altman.

Much of the story that follows revolves around two concepts:

• AI safety

• Research vs Commercial Environments

I’ll cover the groundwork for each concept, then dive into the high level narrative of the competition between OpenAI and DeepMind.

AI Safety

Long before AI safety became a technical field, Sci-Fi gave the public a meaningful way to think about machine intelligence and control. “I, Robot” by Isaac Asimov in 1950, proposed the “the Three Laws of Robotics” that dictate that “robots must never harm humans, must obey orders unless they conflict with the first law, and must protect their own existence”. Additionally, classic Sci-Fi highlights the potential dangers of misuse of AI such as Terminator, Blade Runner, and Cyberpunk genres.

These examples influenced the development of AI where humans ensure that AI is safe and aligned with human intentions. Humans need to ensure that AI doesn’t deceive users by providing harmful / misleading information.

One way to test for AI safety is through adversarial testing, or “red teaming”, where human testers act as adversaries to intentionally break, deceive, or trigger harmful outputs from AI systems. Red teams simulate real-world attacks to identify safety, security, and ethical flaws, such as prompt injections, bias, or data leakage, before malicious actors exploit them.

As AI models scaled in complexity, they became harder to interpret. Inputs could be fed in during training and outputs could be evaluated during inference, but the internal representations became increasingly difficult to explain. AI can produce human-aligned outputs 99.9% of the time, but identifying the 0.1% of cases where it acts against human intentions can be difficult.

Red teaming alone does not guarantee total safety, but is one tool in a broad range of strategies to ensure that AI is aligned with human intentions.

Research vs Commercial incentives

In large tech corporations such as Google, there are two main categories of groups that contribute to business revenues:

• Revenue-generating core business - Engineers who create products to take to market or maintain/update existing infrastructure. Here, work is considered to be low risk and execution on set priorities is important.

• Research and development - Researchers who work on high-risk, long-term, and ambitious ventures. In R&D environments, ideas are incubated to either be published or folded into products to take to market.

Organizations consist of groups that range from commercial readiness vs research focus that help sustain the revenue of the core business as well as bet on “moonshot” ideas that can generate revenue in the future.

In 2015, Google restructured under the umbrella holding company “Alphabet”, separating its core businesses from longer-term moonshot bets such as autonomous driving, fiber, and life sciences. This move positioned Google to overcome the “innovators dilemma” by incubating capabilities to enhance its core business or act as additional revenue generating streams in the future should outside disrupters arrive and threaten it.

DeepMind, HQed in London, was acquired by Google because its capabilities were potentially useful to the core business. However, it was heavily siloed from the main HQ in California at first. This siloing gave the researchers to environment to produce many of the key innovations but caused it to hesitate entering the market when commercial pressure arose.

OpenAI, on the other hand, is HQed in San Francisco, California, with a highly influential leader, Sam Altman, and the entire network and talent pool of the Valley nearby.

OpenAI - Fast to Market, with Safety Governance Under Pressure

OpenAI was founded in 2015 by Sam Altman, Elon Musk, Greg Brockman, and others as a non-profit with the primary goal of ensuring that artificial general intelligence (AGI) benefits all of humanity.

In 2020, OpenAI published “Language Models are Few-Shot Learners,” introducing GPT-3, a 175-billion-parameter autoregressive language model, building off of the “Attention is all you Need” paper published in 2017. This was one of the first truly major, large-scale Large Language Models (LLMs).

However, as models were becoming more complex, AI safety became a concern. Former OpenAI employees including Dario and Daniela Amodei defected and founded Anthropic in 2021 to prioritize AI safety, alignment, and research.

To improve AI safety, OpenAI released a paper in 2022, “Training language models to follow instructions with human feedback” which used Reinforcement Learning with Human Feedback (RLHF). It demonstrated that humans are able to “teach” AI outputs that aligns with humans intention so that the AI can be trained more effectively. This paper demonstrated improvements in how humans can “guide” AI to produce output that aligns with its intentions.

On November 30, 2022, ChatGPT was officially released by OpenAI. It was intended to be introduced as a “research preview” and fast tracked to not let Anthropic get ahead. It quickly went viral, reaching one million users within a few days of its launch, the fastest growing consumer application ever at the time (Threads would later surpass this).

About a year later, the board was concerned about Sam Altman’s handling of AI safety and other allegations. The board abruptly fired Altman on November 17, 2023, citing concerns about his candor. But the move triggered a rapid employee and investor backlash: nearly all OpenAI employees threatened to leave unless he was reinstated, and Altman returned as CEO on November 22, 2023 with a restructured board.

Google DeepMind - Prioritizing AI safety, but Caught in the Innovators Dilemma

Google saw commercial potential in AI when it first acquired DeepMind because it recognized the strategic importance of AI in its core business.

At first, DeepMind’s culture leaned toward research depth, scientific credibility, and safety caution. It saw too much existential risk to let commercial priorities drive AI’s deployment, which made it slower to convert frontier models into consumer products. Demis wanted a path to spinout to a semi-independent Alphabet company, but ultimately didn’t due to the legal effort involved.

When OpenAI released GPT-3 in 2020, DeepMind did their own work on LLMs and published a few key papers, including Flamingo, Gato, and Chinchilla in 2022. In Sept 2022, it worked on Sparrow, DeepMind’s version of ChatGPT, which used RLHF. However, it was ultimately not released to the general public.

When OpenAI released ChatGPT, Google was briefly trapped by the innovators dilemma where it would put its brand at risk if it released an AI model too early and it hallucinated. During this time, a lot of DeepMind researchers were being poached by OpenAI who were frustrated by lack of progress by Google releasing a product.

On April 20, 2023, Google Brain (aimed at applied machine learning for Google products like TensorFlow) and DeepMind merged to form a single unit, Google DeepMind, to become one comprehensive unit.

Google DeepMind ultimately launched Gemini in Dec 2023 and upgraded versions that followed. While it is widely used nowadays and offers many benefits over ChatGPT (such as much larger context window for large documents), it ultimately suffered from the first mover advantage OpenAI had.

Key Technical Developments in modern AI

With that institutional backdrop in place, the technical story becomes easier to understand.

Next I’ll dive deeper into the key technical developments along the way to modern AI referenced in the book that approximate human intuitions. This is by no means a complete list since this book does not cover the more recent “reasoning era” Agentic AI.

Deep Reinforcement Learning (2013-2015)

Reinforcement learning is a fairly intuitive, but math-heavy, ML method where an agent interacts with the world and periodically receives rewards / reinforcements that reflect how well its doing. Reinforcement learning is especially popular in environments where there are small number of training examples compared to the state space the agent exists in (like chess) that supervised learning would struggle in.

Each agent exists in an environment that resembles a Markov decision process, such as a maze, with other states that the agent can transition to. The agent has a policy that determines whether it will explore the environment or exploit an action with known outcomes. Each agent goes through several trials of decisions, determines which actions likely led to the desired outcomes, and updates its policies with what it learned.

One popular method of reinforcement learning is Q-learning (or Quality) where the agent estimates the max expected future reward for each action in each state. While Q learning is fairly primitive, it forms the basis for more sophisticated algorithms.

In a 2013 paper, “Playing Atari with Deep Reinforcement Learning”, Hassabis attempted to combine deep and reinforcement learning to create a single NN agent that is able to learn to play as many of the games as possible without knowing the rules in advance [1]. However, combining deep learning and reinforcement learning presented some challenges:

• Deep learning applications require large amounts of hand-labelled training data (at the time). It also assumes samples to be independent.

• RL algorithms, on the other hand, learn from a scalar reward signal that is frequently sparse, noisy and delayed. It also frequently encounters highly correlated states.

DeepMind used Deep Q-Networks (DQN) to play Atari games. They use a method of “experience replay” where the agent’s experiences are stored at each time step, then are sampled uniformly when performing updates. Randomizing samples helps break up correlations and allows the DNN to learn more effectively.

They demonstrate this approach on seven Atari Games (Beam Rider, Breakout, Enduro, Pong, Q*bert, Seaquest, and Space Invaders) and showed that it generalizes to a variety of games without specific information about each game.

Transformer (2017)

The Transformer architecture was introduced in the June 2017 seminal paper "Attention Is All You Need" by researchers from Google Brain and Google Research. It is widely considered one of the most important, consequential, and influential AI papers of the 21st century because it influences the architecture of Large Language Models we use today.

Prior to the transformer model, the standard approach for language training were Recurrent Neural Networks (RNNs) and sequence-to-sequence models with Long Short-Term Memory (LSTM). A RNN/LSTM processes words sequentially. This is useful in simple, condensed sentences structures such as subject-verb-object pairs. However, it suffers from modern sentences especially when meaning between words (both local and global context) are far away from each other in time.

The proposed “Attention” mechanism allows each word context into what other related words are doing. To get an idea for how the Attention mechanism works, lets consider real sentences.

Real sentences can have multiple parts of speech, with nouns (things), adjectives (modifiers to nouns), verbs (actions) and other parts of speech. Nouns can take on multiple meanings depending on the “context” is it in. Take the word “apple” in the following sentences:

• Apple will release an iPhone next year.

• I ate an apple and it was delicious.

In each sentence, its clear to us what “Apple” refers to: a company in the first sentence because of the world “IPhone” that provided context, and a fruit in the second because of the word, “delicious”. We humans used context clues to determine what “apple” means based on supporting words around it.

This is roughly what “Attention” enables mathematically in the Transformer architecture.

In the original Transformer, words are processed in parallel through attention layers rather than one step at a time as in RNNs. These words are broken down into several units called “tokens”.

Then, each token is converted to a high dimensional vector called an embedding that captures the meaning of each token. Think of this vector like the entries of a thesaurus where each tokens’ values corresponds to how strong it correlates to each entry in the thesaurus. A word like apple can map well on the the position that represents companies (such as Tesla, Exxon) and another position that specific fruits (oranges, pears). These embedding values are found during training on billions of words since each embedding has multiple “dimensions” it sits on.

Then, through matrix operations, every word in a sentence calculates how much “attention” it should pay to every other word in that same sentence. This allows it to figure out relationship between words based on context clues.

To accomplish this, each embedding vector is decomposed into three vectors : Query (Q), Key (K), and Value (V). The query vector for each word is then multiplied by the key vectors for all the other words to come up with an “Attention” matrix that represents how related words are to each other. This attention value is given by the following formula:

To predict the next word in the sequence, it uses a decoder to look at words in context to be able to generate the sequence of words that map well to the original meaning of the sentence.

AlphaFold 2 also showed how attention-like architectures could move beyond language: it used attention over sequence and pair representations to reason about relationships between amino acids, including residues far apart in the 1D sequence but close in 3D structure.

The transformer is significant because it is scalable along memory and compute:

• The context window represents the “short term memory” as the maximum number of tokens (words) the model can “see” in one single look.

• The main attention calculation is a matrix multiplication, so modern GPUs can parallelize these matrix multiplications to higher embedding dimensions.

Sentiment Neuron (2017)

The paper “Learning to Generate Reviews and Discovering Sentiment” was released by OpenAI on April 5, 2017. This paper was consequential because it became an early and memorable example of a neural network learning an internal representation that researchers did not explicitly program.

Originally, the researchers at OpenAI trained a specific type of RNN called an mLSTM (Multiplicative LSTM) on 82 million Amazon reviews with unsupervised language modeling. They found that one single neuron had essentially "volunteered" to track sentiment of whether parts of the review were positive, neutral, or negative.

This paper is considered highly influential to AI safety because it gave insight to researchers that certain neurons can be “inspected” to find big picture behavior and debug potentially dangerous behavior.

Few-shot and in-context learning in GPT-3 (2020)

Since the transformer paper was published, the standard approach for LLMs was to pretrain the models on a ton of data and fine tune the model for each specific task, such as reading comprehension and answering questions.

In Apr 2020, when OpenAI released GPT-3, it published a paper, “Language Models are Few-Shot Learners”. This paper presents “Few-shot learning” where a model learns to perform new tasks or recognize patterns simply by providing examples in the prompt, a capability known as in-context learning. It demonstrated that a generalized LLM model could be taught to solve new tasks using only a very small amount of training data. Other settings such as zero-shot and one-shot were shown with some accuracy loss compared to few-shot at the time.

Together, the Transformer architecture and GPT-3 helped make scaling a central paradigm in AI: as model size, data, and compute increased, capabilities improved in increasingly predictable ways.

Reinforcement Learning with Human Feedback (2022)

When LLMs began scaling in complexity, they sometimes generated outputs that were not aligned with their users, such as toxic or biased statements. The paper “Training language models to follow instructions with human feedback” used human-guided Reinforcement learning to “teach” the model what outputs are acceptable to humans.

OpenAI hired 40 contractors and trained a model, InstructGPT, with human feedback. They evaluated its responses vs GPT-3 and found that labelers significantly prefer InstructGPT outputs over outputs from GPT-3.

This procedure showed improvements in truthfulness and reducing toxicity. However, RLHF is hard to scale because human-provided feedback on AI is harder as AI generated more sophisticated outputs.

Chain-of-thought Prompting (2022)

The concept of Chain-of-thought prompting was introduced in the 2022 paper, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. CoT was a major demonstration that providing intermediate reasoning examples to pretrained LLMs can improve performance on arithmetic, commonsense, and symbolic reasoning tasks in sufficiently large models.

CoT also made model outputs more interpretable at the surface level, because humans could inspect the generated intermediate steps, though those steps should not be treated as a guaranteed faithful transcript of the model’s internal computation.

This approach is similar to how humans learn by analogy/example. It is typical for humans to decompose the problem into intermediate steps and solve each before giving the final answer. Most of us were taught to “show your work” when solving complex problems and were rewarded partial credit even though our answer might be wrong, since the line of reasoning can be valid up to a certain point.

Anthropic - Interpretability (2021-)

Anthropic was founded in 2021 by former OpenAI executives Dario and Daniela Amodei and other colleagues. It was established as a public benefit corporation (PBC) dedicated to AI safety, focusing on creating reliable, interpretable, and safe AI systems (like Claude) following disagreements over the rapid commercialization and safety approach at OpenAI.

Anthropic’s safety research pushed hard on mechanistic interpretability: trying to identify features, circuits, and internal representations that help explain why a model produced a given output.

This matters because interpretability is one possible path toward moving AI safety beyond trial-and-error behavior testing. Red teaming can reveal when a model fails from the outside; interpretability tries to understand why the model behaves that way from the inside.

Conclusion

Modern AI did not emerge from a single breakthrough, company, or personality. It came from the convergence of several technical ideas: deep reinforcement learning, transformers, scaling laws, RLHF, chain-of-thought prompting, and interpretability — with a set of organizational tensions that shaped how quickly those ideas moved from research labs into the hands of the public.

Demis Hassabis and Sam Altman represent two different archetypes in this story.

• Hassabis reflects the research-driven, long-horizon pursuit of intelligence as a scientific problem.

• Altman reflects the commercial, political, and go-to-market instincts required to turn frontier research into a mass-market product.

Both approaches have strengths and failure modes, and that tension is the real story of modern AI.

The next era of AI will likely be shaped not only by better models, but by how well institutions can balance those forces while building systems that are powerful, useful, and aligned with human intent.

If you want to continue expanding your knowledge on the HW side of AI, check out my other post:

References

[1] Mnih, V, et al. “Playing Atari with Deep Reinforcement Learning” https://arxiv.org/pdf/1312.5602

[2] Sutskever, I, Vinyals, O, Le, Q. “Sequence to Sequence Learning with Neural Networks” https://arxiv.org/pdf/1409.3215

[3] Vaswani, A, et al. “Attention Is All You Need” https://arxiv.org/pdf/1706.03762

[4] Radford, A, Jozefowicz, R, Sutskever, I. “Learning to Generate Reviews and Discovering Sentiment” https://arxiv.org/pdf/1704.01444

[5] Brown, T, et al. “Language Models are Few-Shot Learners” https://arxiv.org/pdf/2005.14165

[6] Ouyang, L, et al. “Training language models to follow instructions with human feedback” https://arxiv.org/pdf/2203.02155

[7] Wei, J, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” https://arxiv.org/pdf/2201.11903

(Edit: I revised this article on 5/20/26 with sections and much richer insights, including the generational career gap, the compartmentalization in the Manhattan project and why we mistake curiosity for consumption)

I attended ISSCC in 2026, which is the “Olympics” of chip design and one of the most highly specialized conferences in the world. During a plenary talk, Hope Giles, VP HW PM at Apple, emphasized that system thinking and cross domain awareness will become critical skills as semiconductors and technology become more complex.

And I absolutely agree.

However, the advanced nature of technology work nowadays makes it difficult to develop this type of thinking in practice.

I’m writing this post because I notice a huge gap amongst the way job roles and defined nowadays, and the type of cross-domain thinking that is actually a premium as systems become more tightly integrated.

The Generational Career Gap

I’d argue that there is a perspective gap amongst leaders/more senior engineers and junior/mid level engineers nowadays. As companies have gotten bigger and more integrated, the former group have scaled their experience along with the development of the technology. This allowed them to get a much broader perspective of entire systems earlier in their career when they were less complex, and build their career capital through incremental ownership of complex systems.

Now, as systems have become more complex, most job descriptions are specifically tailored toward specific subsystem ownership of more complex systems, not broad ownership of less complex systems. This is causing gaps understanding amongst engineers where they have trouble describing the impacts their role had on the bigger picture.

Why Cross Domain Thinking is Easy to Say but Difficult in Practice

Due to structural and psychological factors, cross-functional awareness is hard to develop in complex technology organizations because it tends to reward specialized subsystem ownership with handoffs at interfaces.

To coordinate peoples efforts in designing a complex product, organizations are set up to flow down requirements to subsystem executors, and later integrated at the end. This is an unfortunate consequence of how we must coordinate work to be able to create complex products that meet performance demands of the market.

Some organizations are especially notorious for keeping information under tightly guarded wraps on a need-to-know basis. For some organizations, this is to prevent IP leaks, while for others, its for national security reasons.

The consequence of this is that big picture assumptions often do not make it to subsystem executors, and can lead to inefficient work being done as information is thrown over the wall. There have been a few times where I’ve asked “can I better understand why you need this?” or “can I sit in a meeting to better understand customer needs” and either told no or gotten spoon-fed information.

• For instance, the Manhattan Project was one of the most heavily compartmentalized efforts in American history. Approximately 130k people worked on the project not knowing they were building a bomb. Only about 10 people knew of the projects purpose. This causes quite a stir amongst physicists like Richard Feynman, who thought it would make people “stupid and incompetent”. Compartmentalization even caused serious risks when workers handed Uranium without knowing what is was intended for. When these workers were read on and understood the purpose, they were able to better handle the uranium.

Cross domain thinking is also difficult psychologically because it requires you to admit to yourself what you don’t know. When you are an expert in a specific area, it often feels like a blow to the ego to not understand material in adjacent domains and to ask someone in that field to explain something simple to them. This requires a huge amount of psychological safety amongst members of an organization, which may not always be present. Not all organizations and teams really emphasize psychological safety or rather treat it as “fluff”.

Of course, businesses need to make money, execution on set priorities is important and information does need to be controlled to some extent.

How to develop Cross Domain Awareness

It can be very difficult and frustrating to develop cross-functional awareness in such environments that rely on sole handoffs at interfaces and deadline pressure. Trust me, I’ve had my experiences.

However, the two most powerful ways to develop cross-functional awareness within such organizations is by cultivating your own curiosity and by teaching others in small ways. Let me explain.

In my experience, which I teach something, whether as a TA in undergrad, mentoring others, or writing dense technical Substack articles, it forces me to consider what the learner doesn’t know. There’s also a risk of me being wrong when I put information out there. This forces me to do a ton of research and filter out complex information to convey what is important for them. In the process, it helps me sharpen my own mental models.

Teaching is more about the process of synthesizing information on your own so that both you and the learner mutually benefit from the information exchange. This is the reasoning behind the old adage, “You remember 90% of the things you teach others”.

Why do we mistake curiosity for consumption?

However, when it comes to tech, the real risk is mistaking curiosity with consumption of information. What do I mean by this?

• Consumption is passive and mindless. When we consume information, such as YouTube videos or Substack articles, we tend to do so because the algorithm effectively lowers the resistance to seek out such information. You might read a nice article describing a complex system and thinking you gained something of value, and you might have indeed if you really focused. But unless you successfully explain to someone like they are 5, you haven’t really learned it.

• Real curiosity, however, is deliberate and requires focus. It forces you to admit to yourself what you don’t know, and force you to seek information to satisfy that curiosity.

When out jobs make us stressed, we tend to resort to consumption and escape at the end of the work day. This is understandable.

However, to develop a broad base of knowledge, we need to actively carve out small chunks of time as “deep work” times to let new material soak in.

How to Cultivate Curiosity

There are several little ways to cultivate your own curiosity. These include:

• Taking advantage of organizational resources.

• Going to the local/university library and look for technical and business resources. Read small units of chapters at a time.

• Attending internal seminars and asking the speaker questions afterwards

• Attending external conferences and reporting back on what you learned

• Asking questions of more senior people like “How did you get to your position”. (In my experience, people enjoy imparting their wisdom, but only if you ask them for it.)

• Reach out to people in other domains in the company and ask them what they do.

One thing that actually helped me cultivate real curiosity outside of work is Brazilian Jiu Jitsu. BJJ is a sport that humbles even the strongest people because in most cases, success is more about leverage and technique than pure strength. As a white belt you might be learning something new every class but not able to use it right away against more skilled opponents. Rolls feel frustrating. However, by continually showing up, you slowly accumulate knowledge little by little and a few months later, you know more than the new guy who just came in.

I personally BJJ is a perfect analogy of how real insight is developed: not through chasing quick trends, but slowly accumulating hard-won insight that compounds over time. When new people show up, you have a platform to share those insights.

Another useful thing that helps is integrating a mindfulness practice in your daily routine to help regulate your nervous system. Even if it helps a bit, it can open up small windows that all you to develop the curiosity needed to let new material soak in.

How to Teach Others

Once you’ve gathered enough insight, there are small ways you can teach others in corporate environments:

• Find ways to offer interesting insight to people during hallway conversations and handoffs at interfaces

• Propose a seminar in your organization about a past problem or conference you’ve been in

• Create internal “lessons learned” and forum posts on things you think would benefit others in the organization

• Find impressionable people, such as early career engineers, and offer to mentor them

You don’t have to go out of your way; just find little moments here and there to add value to others. Run little micro-experiments with people, see what works and doesn’t.

Eventually, these little efforts now can lead to much bigger efforts down the road such as organizing seminars and writing that you and other greatly benefit from. These also lead to visibility that helps during promotion decisions.

In some cases, teaching others may not improve your visibility within the organization. Even in these cases, you still benefit from the preparation that went into knowledge accumulation that can’t be taken away, unlike titles and status. It also helps future proofs your skillset in case the industry can and will take a turn. Trust me, you will want to ensure your skillset is sharp because you never know what will happen.

Conclusion

In my opinion, cultivating curiosity and teaching others around you is more beneficial for both your career and others around you in the longer run. Organizational execution and gaining visibility for promotions is also important, but should not be the sole focus for career success, even when it seems like everyone else is doing it.

Share

Editor’s Note (6/1/26): Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

Before each paywall, I’ll post links to some of my other adjacent posts that give you a well rounded flavor of the interactions amongst other domains.

In this post I will cover entire optical communications space:

• I will introduce the conventional IM-DD optical approach with VCSELs used for short reach

• I will then highlight challenges with scaling conventional IM-DD to higher data rates

• Then I’ll explain techniques to improve data bandwidth:

  • 🔒 Co-packaged optics

  • 🔒 Silicon photonics

  • 🔒 External modulation

  • 🔒 Coherent optics

High speed optical interconnects have become one of the hottest markets in AI data centers. As bandwidth demand rises, the challenge is no longer just computing on data faster, but moving it across packages, boards, racks, and clusters efficiently.

The optical space has exploded over the past few years with many companies and startups proposing solutions to tackle data demands.

But I want to step back and ask ourselves: what are the first principles challenges with scaling high-speed optical communications?

The workhorse for short reach - IM-DD & Pluggable Optical transceiver

The workhorse for short reach optical communications today are IM-DD with pluggable optical transceivers. Lets take a looks at what each consists of.

Intensity Modulation - Direct Detection

IM-DD is the workhorse of short reach optics. IM-DD encodes information in optical intensity (whether that be 0/1 or higher levels in PAM4) and recovers it through direct detection.

This intensity modulation can be performed in one of two ways:

• Direct modulation - the laser is turned up/down by the electrical signal

• External modulation - a driver modulates a steady optical carrier

Pluggable Optical Transceivers

These IM-DD are typically packaged into pluggable optical transceiver modules that sit far away from the chip. These are are match-box sized modules that take in electrical (client) side data and convert it to optical (line side). Each block consists of following:

• Serdes + DSP - For serializing / desterilizing the data from/to host and perform equalization

• EIC - Electrical integrated circuit - consists of the electrical circuits fabricated in electrical processes (drivers, mixed signal conditioning, etc)

• PIC - Photonic integrated Circuit - consists of the optical circuits fabricated in optical processes (Si Photonics, VCSELs, etc)

Optical transceiver modules are a very mature technology that has scaled in data throughput for many years. However, with AI workloads demanding 224+ GPBS throughput capability, the performance of the optical transceiver itself needs to adapt, as well as the signal integrity between host die and optical engine.

System architecture breakdown of a IM-DD

Next I will break down the architecture of an IM-DD in each of its components:

• Transmitter

• Fiber

• Receiver

I will discuss several options for each and important design considerations.

TX Driver - Direct Modulation

Laser (Light amplification by stimulated emission of radiation)

Lasers are powered by a large pump current stimulating emission. Photons with energy close to the material bandgap interact with electron - hole pairs and cause identical photons to be emitted. This causes an “optical gain” of photons within the active region.

This photons is then confined using two mirrors: one opaque, and other that is semitransparent to allow the light to shine through.

Lasing occurs when the pump current is high enough such that optical gain > optical losses.

Then, either a cavity structure or frequency selective bragg reflectors selects which optical modes are sustained.

In a Distributed Feedback laser (DFB) laser, a periodic grating of Bragg reflectors provides wavelength - selective optical feedback, favoring a narrow lasing wavelength. These Bragg Reflectors are highly tunable depending on the index of refractions and the lengths and # of sections.

VCSEL (Vertical-Cavity Surface-Emitting Laser)

VCSELS are lasers that emit light vertically in a circular cross section. The active region consists of multiple quantum wells that enhances optical gain. Distributed bragg reflectors are stacked on the top and bottom of the active region with alternating refractive indexes. These layers are fabricated using epitaxial growth on a substrate.

The light is then confined with an oxide and current is pumped in through pads. The light is shown vertically above and redirected toward the optical fiber with optics.

The Pout/I of VCSELs almost resembles a linear diode. The VCSEL begins lasing above a particular threshold current and exhibit a linear relationship between Pout / Iin over a useful operating region. Lasers require a DC bias to set the device in the linear region.

One important parameter is the extinction ratio, which is the ratio between P1 and P0. A higher extinction ratio is preferred as it improves BER margin to detect teh signal at the other end, but higher extinction ratios require higher currents, and therefore power, to drive the device.

The two major materials used in lasers are GaAs or InP:

• 850 nm GaAs VCSELs are common for short-reach multimode links.

• 1310/1550 nm InP-based lasers/modulators are common where single-mode fiber, longer reach, or telecom/datacom bands matter.

VCSELs are very popular in short reach communications for a numbers of reasons:

• Low cost manufacturability - they are easy to test on-wafer across multiple samples

• Scalability - they can be formed in “arrays” to drive multiple optical lanes

However, VCSELs do have some drawbacks:

• Reliability - issues due to oxide confinement and high currents degradating mean-time-to-failure

• Signal switching nonidealities, including:

  • Turn on delay- a delay in the turn on of the signal that adds to latency

  • Jitter - uncertainty in the signal edge that degrades BER

  • Relaxation oscillation - an “underdamped” response in the turn-on characteristic that can affect BER and signal quality

The main tradeoff in VCSELs is between BW and reliability. Higher current densities allow for higher BWs, but it also decreases the mean time to failure, a common metric used when evaluating reliability.

Driving the Laser

Here are a few DC coupled, common anode circuits used to to drive the laser directly using either fully differential or pseudodifferential to convert a voltage to a current to drive the VCSEL. There are other methods of driving such as common-cathode, BJTs, and push-pull implementations depending on the requirements of the system.

In general, the more VCSEL current that is needed, the slower the driver will be due to increasing output capacitance.

Fibers

Multimode fiber optic cables are primarily constructed from silica glass (silicon dioxide). There are two major categories of fiber for high speed communications:

• Multimode - Uses a larger core (50–62.5 um) to transmit multiple light paths

  • There are five classifications, OM1-5, that grade the quality of the fiber.

  • Pros: Cost-effective for short-distance, high-speed applications

  • Cons: Limited distance due to higher signal modal dispersion

• Single Mode - Uses a smaller core (8-10um) to transmit a single mode of light

  • Pros: Low signal loss, avoids modal dispersion

  • Cons: Higher cost of laser components and more complex installation requirements

Generally speaking, multi-mode fibers are used for short reach interconnect due to its low cost and complexity. However, single mode fiber is being evaluated to “future proof” the data communications since it can support the higher BWs, though it adds to cost and integration complexity.

There are also four wavelength bands used in the fiber:

• 850nm band - used for short reach links

• O-Band - 1310nm

• C-band - 1550nm

• L-Band - Extends C band to allow 2x network capacity

There are four major impairments:

• Loss. Losses cause pulse attenuation after propagation from a fiber. These are caused by intrinsic sources, including

  • Rayleigh Scattering

  • Absorption

  • Manufacturing Stress

• Chromatic Dispersion - pulse spreading after propagation over the fiber. This is caused by fiber refractive index dependence on wavelength

• Modal dispersion - distortion due to different propagation delays of different optical modes. Modal dispersion matters in multimode fiber.

  

• Polarization Mode dispersion (PMD) - dispersion cause by pulse spreading of different polarizations due to asymmetries in the fiber core.

Receiver - Photodetector

A photodetector consists of a Reverse biased PIN junction that generates current proportional to the detected light. The transimpedance amplifier then amplifies this current to an output voltage at the other end.

There are some important specs of this system:

• PIN characteristics: Responsivity, capacitance, noise, dark current

• Receiver Q-factor: The received signal level relative to total receiver noise

• BER: BER can be derived from this Q factor: BER = 1/2*erfc(Q/sqrt(2))

A higher extinction ratio is desired to ease the burden on the receiver, but it does come with a higher power usage.

Why Scaling IM-DD is hard

Scaling IM-DD to higher data throughputs is hard due to several unfavorable tradeoffs between BW, reach, noise, power, SNR, and cost. These include effects in both optical and electrical domain themselves:

Optical

• Chromatic dispersion

• Polarization Mode dispersion (PMD)

• Nonlinear and interference effects

Electrical

• Noise penalties at Higher BW

• Tighter jitter, skew, and ISI

• BW scaling limits in front ends

You can have the fastest optical transceiver, fiber, but its pointless if the signal can’t get to it fast enough. Not only the optical receiver itself needs to improve, but the signal integrity from the electrical interconnect is running into challenges shuttling high speed data back and forth. Solutions need to ensure that speed “bottlenecks” within the system scale with the system demands.

This leads to four major trends that generally increase complexity:

• Co-packaged optics - embedding optical components within the package itself

• Silicon Photonics - routing light through silicon waveguides and modulating it

• External modulation - modulating a continuously running laser for higher BW

• Coherent Optics - modulation amplitude and phase in the optical domain

After the paywall I’ll discuss these in further detail.

Share

If you want to learn more about how optical communications interfaces with adjacent domains, consider checking out these other posts:

Read more

Editor’s Note (5/26/26): Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

Before each paywall, I’ll post links to some of my other adjacent posts that give you a well rounded flavor of the interactions amongst other domains.

In this post I cover the following topics:

• Core Electrical Constraints of AI Loads

• Architectural Goals of AI Power Delivery

• Converting below 48V: The many possibilities

• 🔒Possibilities of 800V to 0.8V Conversion

• 🔒The Conventional PoL Approach: Lateral Multiphase Buck Converters

• 🔒Why PDN Constraints Push Regulation Closer to the Chip

• 🔒Why Vertical Power Delivery Is Emerging as the Endgame

AI compute is scaling so quickly that power delivery is becoming a first-order system bottleneck. Traditional 12V server racks were not designed for AI accelerators that operate at extremely low voltage, draw enormous current, and swing their load profile rapidly.

Over the past few years, AI power architecture has evolved quickly, with power demands projected to exceed 1 MW per rack and companies proposing solutions across every layer of the stack. Among those solutions, one direction stands out: vertical power delivery.

Vertical power delivery is emerging as the natural endpoint of a system that must deliver massive current into increasingly power-dense silicon. But that raises a few interesting questions:

• How did the architecture arrive there?

• What tradeoffs in power distribution make vertical power delivery an increasingly attractive solution at the point of load?

In this post, I build a holistic overview of AI data-center power architecture from first principles to explain why it is evolving in this direction. My aim is to keep the discussion high level enough to provide a map of how the different subsystems influence one another.

Much of the architectures in AI data centers derive themselves from the principles of buck converters. Buck converters are step down DC-DC converters and are the workhorse of most power electronics to bridge the gap between distribution voltage and the voltage that electronics require.

Core Electrical Constraints of AI Loads

Modern AI accelerators are among the most challenging electrical loads because they combine three properties that make power delivery unusually difficult: low supply voltage, high current density, and extremely fast transients.

High power density

GPUs consume massive amount of power due to low supply voltage and very high current:

• GPUs operate at very low supply voltage, typically around 0.6 V to 1.1 V. The supply voltage wants to be as low as possible because these devices switch an enormous number of transistors at high frequency, and a lower voltage reduces dynamic switching loss through the familiar P=CV^2*f relationship. Advanced CMOS devices also impose tighter voltage limits because thinner oxides leave less reliability margin. The downside is that at such low voltages, even a small amount of droop becomes significant.

• At the same time, GPUs can draw on the order of 1000’s of amps to support massive parallel computation. As compute density increases, the current required to support it rises as well. Since conduction losses scale with I^2*R, every bit of resistance in the path adds significantly more to loss.

Extremely fast transients (up to 2000A/us)

AI loads also exhibit extremely fast transients, sometimes on the order of 2000 A/µs. AI loads can change more quickly than a distant regulator can respond because much of AI computation is performed on-demand through large bursts of multiply-accumulate computations. That forces the regulator and the PDN to work together across a broad bandwidth, often with additional system-level support mechanisms.

Architectural Goals of AI Power Delivery

These electrical constraints force the AI power delivery architecture to optimize around two goals: efficiency and reliability.

Efficiency

Efficiency matters because even a 1% improvement can translate into major savings in heat, electricity, and cooling when multiplied across racks, clusters, and full data centers. That means losses must be evaluated across the entire chain, not just at the converter level.

• The converters contribute loss from conduction loss, switching loss, diode-related loss, and CV^2 loss

• The PDN contributes additional loss through planes, traces, vias, connectors, package parasitics, and capacitor ESR.

The interconnectedness of the power delivery system is what makes apples-to-apples comparisons across vendors and research papers so difficult. A converter may look excellent on paper, but that does not mean it is the best system-level solution once power density, component count, and PDN burden are considered.

Reliability

Reliability matters just as much. Data centers make money when they stay online and productive, so the power system must be designed with fault tolerance, serviceability, and safety in mind. As higher distribution voltages such as 800 V enter the architecture, maintainability and safe access become even more important.

Design Goals for the Architecture

The rack architecture for 1 MW-class systems exists to manage current over distance. At a high level, AI power delivery is trying to satisfy two goals at once:

• Transport energy efficiently from the power source to the compute load

• Regulate it precisely at the point of load.

This is why AI power delivery is fundamentally a distribution-and-distance problem. A data center contains large amounts of compute spread across multiple racks and shelves, and the goal is to move power to each GPU with minimal loss.

That naturally leads to a hierarchical architecture: distribute power at high voltage across longer distances, then step it down progressively as the delivery path shortens and the burden shifts toward precise near-load regulation.

These goals have led to the conventional modular stack: 800V → 48V → 12V → 1V

• High voltage minimizes current during long-distance transport amongst racks

• 48 V provides a practical intermediate distribution rail and minimizes PDN losses within the rack

• Lower voltages are introduced only as the distance to the load gets shorter

In short, the architecture tries to keep very high current confined to the shortest possible path near the load itself.

Converting below 48V: The many possibilities

Below 48 V, the design space opens up dramatically because there are many possible ways to reach the roughly 0.6 V to 1.1 V rail required by the GPU.

Broadly speaking, these approaches fall into either single-stage or two-stage architectures.

• In a single-stage approach, the converter attempts to step 48 V directly down to the load voltage.

• In a two-stage architecture, an intermediate bus converter first steps 48 V down to a lower bus voltage, which is then converted to the final load voltage by a PoL regulator.

The intermediate bus architecture is typically employed because it separates the conversion problem into two more manageable stages in the IBC and PoL. The IBC can focus on efficient transformation, while the PoL focuses on tight local regulation.

Within this architecture there are also commonly employed topologies for both the IBC and PoL and options for the bus voltage:

• Convert from 48V to 12V and then from 12V to 0.8V. 12V has a very robust server power infrastructure that can be leveraged, so that most engineering effort can be put in the 48V to 12V IBC conversion. This way, researchers can focus on just optimizing on either the 48V → 12V conversion or the 12V→ 0.8V conversion.

  

  

• Lower the bus voltage even further from 12V to 6/5/3V. Lowering the bus voltage eases the burden on the PoL since the Vout/Vin ratio is smaller. However, a lower bus voltage also increases the current between the first and second stage, and therefore, the PDN loss between those two stages. This solution pushes the IBC closer to the GPU to minimize PDN resistance, and this in turn pushes the PoL as close to the GPU as possible.

  

• Convert directly from 48V to 1V. Many research proposals attempt to directly convert 48V to 1V conversion, which typically employ two stage hybrid topologies. This option is very attractive and has a lot of potential, but current solutions don’t always scale to the require current densities and transient responses required.

Possibilities of 800V to 0.8V Conversion

After the paywall I’ll describe the following:

• A high level chart possibilities of 800V to 0.8V conversion and PoL delivery options with efficiency parameters

• The Conventional PoL Approach: Lateral Multiphase Buck Converters and TLVRs

• Why PDN Constraints Push Regulation Closer to the Chip

• Why Vertical Power Delivery Is Emerging as the Endgame

Share

If you want to continue expanding your knowledge of the subsystems that underlie power systems, I have a post on the tradeoffs inherent in PMIC design using the buck converter as a design case study.

Read more

Editor’s Note (5/30/26): Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth of the synthesis and combination of topics involved, the deepest technical layers of each of my guides will now be reserved for paid subscribers.

Before each paywall, I’ll post links to some of my other adjacent posts that give you a well rounded flavor of the interactions amongst other domains.

As data speeds climb from 112Gbps to 224Gbps and 448Gbps, data starting running into physics constraints and transceivers become more complex to compensate for these effects.

In this post I’ll be doing a system architecture breakdown of a modern DSP-based transceiver. I’ll explain the underlying physics constraints and the major blocks data passes through. Then, I’ll do a deep dive into the high-speed building blocks (shown in yellow) on the TX and RX side. I’ll be covering the following:

• The real problem: increasing data throughput under physical limits

• System Datapath overview

• TX Architecture:

  • Serializer / Mux tree

  • DAC - 6-7 Bit current steering DAC

• RX Side - Time-Interleaved ADC

• 🔒Clocking Architecture

  • 🔒Clock Recovery: Bang Bang Phase Detector

  • 🔒Clock Generation: LC-VCO Based PLL

  • 🔒Multi-phase clock generation: The phase Interpolator

  • 🔒Clock Distribution: Repeater, transmission line, and resonator-based

Most of the figures in these slides are from the ISSCC tutorials as well as the papers in the references.

The real problem: increasing data throughput under physical limits

As data rates climb, we need to ask ourselves: what fundamentally constraints the data speed from first principles?

Data speed is now limited by physics and device parameters on the chip, which include, but not limited to, the following:

• Channel Loss: Loss scales with signal frequency, and high frequency attenuation rounds out the sharp edges of digital signals

• High impedance nodes in analog circuits: Active loads have a high input resistance, which is good for gain, but also creates dominant poles that lower overall BW.

• On-Chip Capacitance: Bigger transistors have bigger C that can affect rise/fall times of signals

• Noise / Jitter: The randomness in clock / data edges affect how fast those signals can be relative to the clock period. This impacts reliability of high speed digital circuits and time-interleaved switches

What tricks do designers have to increase speed? I contend there are two major paradigm shifts:

• Current steering circuits: These circuits steer constant currents through switches to a resistive load. The main advance is that it minimizes charging/discharging time of parasitic capacitors. However, current steering circuits tend to consume more power than equivalent charged-based topologies.

• Time interleaving: Instead of making a single component faster, place many in parallel and distribute data through “ranks” of S/H and T/H switches. Time interleaving introduces added complexity in clocks to coordinate parallel operations

System Datapath overview

Next lets look at the major blocks in the datapath.

The function of a transceiver is quite simple: transmit a large amount of data from the TX and RX with minimal errors across a medium as fast as possible.

For short reach, low speed operation, the architecture relied mostly on analog blocks. However, due to recent data demands, one of the big paradigm shifts is a move to digital for the processing. Digital is great at performing parallel processing of data at a moderate clock (1-10’s GHz) frequency.

However, when it comes to data transmission, edge density becomes the constraint. You can only pack so much circuitry to drive the parallel channels on the edge of a chip and copper cables in the system.

As a result, data transmission favors serialized data at a high speed to pack as much data density as possible and transmit on a single cable.

These constraints influence the architecture of modern wireline transceivers.

On the TX Side, the main high speed blocks include:

• Forward Error Encoding (FEC) - Encodes the data with parity bits that correct for channel error at the RX side

• Feed Forward Equalization (FFE) - Pre-distorts the signal using a FIR filter to compensate for channel losses

• Serializer - Serializes the digital data from multiple channels and raises the effective output rate. 112 Gb/s PAM4 corresponds to a 56 GBd symbol rate, though practical serializers often use half-rate or quarter-rate architectures that are time-interleaved

• High-speed DAC - turns digital symbols into an analog waveform

• Driver - amplitudes the signal before it is transmitted on the channel

In the Channel:

• Matching network - matches the TX and RX to the characteristic impedance to avoid reflections and transmission line effects

• Channel - attenuates and distorts the transmitted signal

In the RX:

• Receiver front-end - conditions the signal by amplifying it (variable gain amplifier) and providing equalization (CTLE)

• TI-ADC - converts the analog value to digital values

• Deserializer - converts the digitized data from the transmission rate to the digital frequency

• DSP - processes and recovers data

As a result of the added digital complexity, high speed and parallel processing blocks require a robust clock architecture to synchronize the data properly.

In the following sections, we’ll step through each of the high speed blocks, with the principle of operation, the key impairments, and how they affect the system level.

TX Architecture

Serializer / Mux tree

The mux tree is used for serializing / deserializing data. Here, stages of 2-1 muxes are arranged in ranks. This ensure each interconnect between muxes aren’t heavily loaded that can slow down operation.

The final rank operates with the highest-rate clock, while earlier ranks use divided versions of that clock so that each stage runs at a rate compatible with its local data rate.

A common high-speed mux cell is the current steering mux cell. This cell uses two differential pairs for each of the inputs. To select each input, the SEL signal, synchronized to the CLK, steers the current between the two input branches.

Impairments

• Kickback noise from clock feedthrough. Clock feedthrough and switching transients can disturb internal common-mode nodes and degrade timing margin, especially in the final high-speed stages.

• Skew between SEL and clk.

• Skew between stages in the MUX tree. The accumulated clock delay associated with the driver and the mux itself can cause ranks to fail to clock the data fast enough.

DAC - 6-7 Bit current steering DAC

The high speed DAC commonly employs a segmented current steering architecture. This architecture uses scaled current sources representing the “weight” of the binary bits. The steered currents are summed and converted to an output voltage across a load impedance.

The current steering cell consist of two main branches and a selector with a flip flop and driver. This current source is generated from the bottom two transistors. The driver cell “steers” this current between two branches: the left branch to enable the bit and right branch to disable it. The switch control is retimed so current steering occurs with precise timing at the symbol rate.

Current continuity is often preferred for high speed because it minimizes voltage swings from constantly charging / discharging capacitive nodes. However, it also consumes static power even when the bit is disabled.

This current cell is then scaled to a segmented architecture, which uses both unary and binary DACs. Unary (or thermometer code) uses unit-sized current sources arrayed in a 2x2 grid in layout to achieve better matching of current sources.

In many wireline transmitters, DAC resolution is kept modest, often around 6–7 bits, because channel loss, jitter, and front-end noise dominate before very fine quantization does.

Impairments

There are several impairments that cause both timing and amplitude errors:

• Time delay

• Voltage glitches

• Clock Skew

• Finite Input impedance

• EMI at all sources

  

These impairments introduce timing and amplitude error at the system level. They translates to jitter in the x domain and noise in the y dimension. These effects directly affect the size of the “eye” at the RX side, so it is important to account for these and spec these appropriately.

RX Side - Time-Interleaved ADC

A common modern RX approach uses time-interleaved asynchronous SAR ADCs operating in a self-clocked manner.

In normal asynchronous SARs, it takes N clock cycles to complete an N bit conversion. In asynchronous circuits, self-clocking allows the completion of one comparison to automatically trigger the next.

Self-clocking removes the need for a very high external clock, but it makes the conversion timing dependent on timing delays and handshake robustness.

Asynchronous SARs are commonly used due to low comparator count, simple digital logic, and capacitive DACs for energy efficient operation at typical 6-7 bit resolution.

These SAR ADCs are time interleaved. The high speed data is “fanned out” to each of the sub-ADCs in order for their analog values to be digitalized. This is accomplished through a two-rank NxM interleaving scheme. This architecture performs sampling in two ranks:

• Rank 1 - Track and hold. The first rank tracks and holds the signal. The performance of these switches are critical as they sample the full BW signal.

• Rank 2 - Sample and hold buffer. These buffer the T/H signals to drive the parallel ADCs.

In modern PAM-4 112Gbps, the interleave factors range from 36-128 sub-ADCs depending on the layout area, with 64 being common. [1]

Impairments: There are several impairments:

• Skew and Jitter of clock

• Mismatches between sub-ADCs

These impairments translate timing mismatches into sampled voltage errors, often in a signal-dependent way, and can create both noise-like degradation and spurious tones.

Clocking Architecture

Perhaps the most important part of a high speed chip is the one everyone takes for granted: The clock.

The clock distribution becomes an increasingly important constraint as dimensions and speeds have scaled down. There are several challenges the clock faces:

• The clock needs to be recovered from the asynchronous RX data to ensure that the clock is in “sync” with the data to clock it appropriately

• The VCO needs to generate a reference clock with low jitter

• The time-interleaved blocks need some form of multi-phase clock to synchronize operations across parallel processing elements cleanly

• The parallel clock distribution needs minimal skew to minimize mismatches

After the pay wall I’ll describe the clocking architecture in more detail.

Share

If you want to get a flavor for the adjacent domains involved, I have several other posts to check out as well:

Read more

Editor’s Note (6/1/26): Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth and combination of the synthesis, the deepest technical layers of my guides will now be reserved for paid subscribers.

In this post I cover the following topics:

• SerDes - The Workhorse of chip to chip communication

• The Root Causes of Signal-Integrity Degradation at High Speed

• How Channel loss and Jitter degrade Bit Error Rate

• Major System Trends:

  • PAM-4

  • DSP-based Architectures

• 🔒Error Reduction Techniques

  • 🔒Forward Error Correction

  • 🔒Equalization

    • 🔒TX - Pre-emphasis / FFE

    • 🔒RX - CTLE and DFE

• 🔒How to Achieve Higher Data Rate

Everyone talks about the importance of chip-to-chip communication, but few people appreciate the complexity hiding underneath it.

Signal integrity and SerDes architecture are often discussed as if they belong to different worlds: one in the lab and on the board, the other inside the chip. In reality, they are deeply coupled.

My goal in this post is to show how signal-integrity constraints at the channel and board level—loss, jitter—affect performance through BER and ultimately drive complexity back into the transceiver itself, shaping choices in equalization, clocking, and data conversion. Here I’ll focus on the signal-integrity foundations from first principles, then build on that in a follow-up post on chip architecture.

Much of the content of this post is material consolidated from a Tutorial given by Dr. Mike Peng Li from DesignCon 2026, the premier high-speed communications conference, as well as from ISSCC, academic papers and SI books. Figures are reproduced for commentary and educational purposes.

SerDes - The Workhorse of chip to chip communication

A serializer-deserializer (SerDes) converts parallel digital data into a high-speed serial stream for transmission, then reconstructs it at the receiver. This allows systems to move large amounts of data between chips and systems without requiring a huge number of parallel wires.

SerDes and optical links are the “workhorses” of communication within data centers in “scale up” and “scale out” networks:

• Chip to chip

• Modular top chip copper links on a PCN and an optical module

• Between a CPU and network interface

• Backplane and midplane links over backplane PCBs or passive copper cables

SerDes is what enables parallel computing across many CPUs and GPUs as a means of data distribution.

Historically, many SerDes architectures have been analog-based. SerDes speeds have scaled with PAM-2 (NRZ) signals over time to meet the growing needs of traditional computing.

Then, AI and chiplets came in and started demanding more data.

These two technological trends put a lot of pressure on chip-to-chip communication to get faster and reach farther due to the massive amounts of data movement:

• AI workloads involve massive data movement: repeated memory access, movement of intermediate results, and parallel processing across many GPUs in a rack.

• Chiplets disaggregate a system into multiple smaller dies, which must then be tied back together through high-speed chip-to-chip interfaces.

However, as data demands push speeds higher and links longer, fundamental physics begins constraining performance:

• Channel loss due to higher attenuation at higher frequencies

• Jitter budgets getting tighter

• Transmission line effects that cause distortions

• Noise that can cause errors

These add complexity to the SerDes to correct for. Unfortunately, the added complexity makes it more challenging to understand how one part of the system affects the other, such as:

• How jitter in the VCO eats into the timing margin needed to meet the target BER

• How receiver clock skew and TI-ADC area constraints affect sampling margin, data throughput, and calibration burden

In the following sections I’ll be focusing on signal integrity fundamentals from first principles. The goal is to ask:

• What are the fundamental physics constraints limiting performance?

• What complexity in the chip architecture is needed to scale speeds beyond 112.5Gbps to 224Gbps and eventually 448Gbps?

The Root Causes of Signal-Integrity Degradation at High Speed

In this section, I’ll focus on describing two primary causes of signal-integrity degradation at high speeds: channel loss, and jitter. I’ll then show how they affect BER.

Channel loss in Copper interconnects

Copper is widely used as the interconnect of choice to transmit signals between chips due to its high electrical conductivity and durability.

Copper has been used for many years at low data rates where the losses were much less. However, the loss increases at higher signal frequencies. There are two major loss mechanisms:

Dielectric loss. Dielectric loss occurs due to heat energy dissipation of dipoles at high frequencies. A quickly alternating electric field in the dielectric causes polarization in dipoles in the dielectric (i.e. the positive and negative parts of the atom move apart) and results in energy dissipation through heat.

Dielectric loss scales with frequency and is generally the dominant loss mechanism at the data rates that modern SerDes nowadays run at.

Skin effect. Skin effect occurs when time-varying electromagnetic fields push current toward the edges of the conductor. This results in an effective lowering of the cross-sectional area that the signal actually “sees”.

Skin-effect loss scales roughly with the square root of frequency and has historically been a major source of channel loss until dielectric loss becomes more dominant at higher frequencies.

In both cases, signal attenuation increases with frequency. This means that signals with multiple frequency components like square waves will have higher order frequencies attenuated more strongly. This will result in a “rounding” of the signal at the other end with degraded amplitude.

Modern 112G PAM4 systems typically target 25 dB to 35 dB of channel loss.

Jitter

Jitter is defined as the time variation in when a signal edge actually transitions compared to its ideal transition point. Jitter is a significant limiter of performance in high-speed applications because it affects how much of a time ”window” you have to accurately sample the digital data.

Jitter is broadly broken down into two major categories: Random and Deterministic:

Random Jitter comes from noise-like mechanisms that are fundamentally statistical in nature, such as thermal noise.

This is always uncorrelated and typically modelled as a Gaussian distribution due to the central limit theorem that states that summing up a variety of PDFs from uncorrelated sources tend toward a Gaussian distribution.

Deterministic Jitter is jitter from repeatable structure in the system such as data movement and reflections.

This jitter is typically bounded with a max and min timing shift - the edge shifts in a repeatable way. It can also be correlated or uncorrelated. Examples include:

• Data dependent jitter (ISI) - Symbols from previous transitions spill into the next time window

• Periodic Jitter

• Bounded uncorrelated jitter

As link complexity has grown, many other jitter sources must be accounted for during design and measured in silicon.

How Channel loss and Jitter degrade Bit Error Rate

Both losses distort the signal in complementary ways:

• Channel loss distorts the waveform itself by attenuating and smearing its higher-frequency content

• Jitter introduces uncertainty in when transitions occur.

Together, these (and other effects such as reflections) reduce eye opening and increase the Bit error rate (BER) at your sampling edge.

Bit error rate is not easy to measure directly because of the amount of time it takes to observe enough samples (sometimes trillions) to get a statically significant measurement of BER. It is usually easy to measure a BER of 10^-6 which is 1 error in 1 million bits with a BERT (Bit error rate tester), but some systems require BERs of 10^-12 which is one error in one-trillion bits!

Often, lower BERs are often extrapolated from higher BER curves. The Dual-Dirac model is a widely accepted “worst case model” used to extrapolate this. To show how BER is driven from jitter, consider the two major classes of jitter:

• RJ spreads the signal edge according to a Gaussian distribution

• DJ shifts the signal edge with two “bounds”. In this model, the worst cases are represented as two worst case impulse ”spikes” at either bound.

  (Note that this is not the actual PDF of the DJ as it varies in uniformity within the bounds. However, this works fine because we only care about the “tails” of the RJ curve convolved at the bounds)

These two sources combine in a “convolution” format, so the gaussians of the RJ map onto the bounds of the DJ to obtain the Total Jitter (TJ). The BER is then estimated from the tails of the total-jitter distribution: as the distribution spreads further into the sampling margin, the probability of error increases.

You can then use this to map contours of constant BER in the center of the eye that decrease the more the signal is sampled at the center of the eye:

Major System Trends: PAM-4 and DSP-based Architectures

PAM-4: 4 level pulse amplitude modulation

PAM-4 uses four levels of modulation instead of two. This allows you to transmit two bits per symbol while halving the required symbol rate for a given bit rate.

However, added signal levels make PAM-4 more sensitive to noise, nonlinearity, ISI, and timing jitter. It also requires DACs to generate the four analog levels. There are also more transition combinations to account for.

DSP-Based Transceivers

When data speeds were slower, most of the processing came from analog circuits that were sufficient to perform the signal conditioning needed to achieve good BERs. Figure 9 shows a typical short-reach, analog-based transceiver with common building blocks.

As speed and throughput increased, most architectures added a lot of DSP along with the analog blocks. Digital allows a lot more programmability to handle data processing with FIR/IIR filters. However, digital also adds a large amount of peripheral complexity to handle the speed it needs, including:

• The ADC/DACs needed to convert between the analog and digital domains

• The clocking architecture to clock the circuits

Error Reduction Techniques

After the paywall, I’ll cover key error reduction techniques, including equalization, FEC, CTLE, and DFE.

If you want to continue expanding your knowledge related to signal integrity, including why modern SerDes is so complex, check out my other posts:

Read more

Editors note (6/1/26) - Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

I am removing …

Read more

After traveling to four flagship semiconductor conferences (ISSCC, DesignCon, APEC, and ECTC), its clear that the intersection of high speed communications, power, packaging, and thermals where the real architectural bottlenecks are happening.

Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. I added links to additional posts I wrote that interface with Buck converters after I originally wrote this one.

To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

However, I’ll leave this post free to get a flavor for the type of depth I go into.

If you want the full bird’s-eye view of how these systems fit together, consider upgrading to a paid subscription.

Buck converters are where the money is, especially in AI data centers, because most electronic systems need some form of step down conversion.

Buck converters are a DC-DC converter responsible for stepping down voltage from a higher voltage to a lower one. They are often times the chips at the “Point of load” (PoL) after AC rectification to an intermediate voltage. In most electronic systems, distribution voltage tends to be high (like 2-4S Li-ion batteries and 48V busses) and required chip voltages are low, (especially in advanced process nodes that can run anywhere from 0.5V-1V).

Buck converters are becoming increasingly challenging to design as distribution voltage gets higher and process voltage gets lower. High efficiency, precise control of control loops, and component area become increasing important as PMICs tend to be highly tailored to the specific application it is powering.

In this post, I want to provide a look into the fundamental operation of the buck converter and how nonidealities affect performance. I’ll cover continuous conduction mode power stage fundamentals. Control-loop design, DCM modes, EMI compliance, and advanced architectures (3L, Multiphase) are intentionally out of scope and left for a future post.

Much of this information is synthesized from the book “Design of Power Management Integrated Circuits” along with my original insight; refer to the book for more details.

The buck converter in CCM: principle of operation

To begin, we look at the inductor current during two time periods: t_on, and t_off

• t_on: Energy is charged through the inductor:

• t_off: Energy is discharged from the inductor:

During steady state operation, we need the voltage across the inductor to average out to 0 over time. Therefore, we need the change in current during t_on equal the current going out on t_off. Setting delta_Il equal to each other and arrange terms we get:

Rearranging terms to get the voltage conversion ratio:

There are two important quantities here:

• Current Ripple

  The current through the inductor will ripple as it is switched on and off. The delta in the current ripple is given by:

The variables should be selected to make the current ripple reasonable. As a general rule of thumb, current ripple is typically +- 20%. Note that this is not the same value as the voltage/load current ripple since the current ripple gets “filtered” out to be much smaller.

Voltage Ripple

• Due to the switching and filtering operation inherent to DC-DC converters, the output will experience a small amount of voltage ripple:

The capacitor value and switching frequency should be selected to ensure the voltage ripple is within the desired specs.

How Nonidealities and Sources of Loss affect efficiency

All electronic components have important nonidealities that contribute to loss. MOSFETs have a few important nonidealities worth pointing out:

• Gate Charge. Charge accumulates on the gate of the MOSFET to be able to turn this on, typically in the nC range. Note that gate charge and energy dissipated are usually specified rather than power, because actual power loss depends on the switching frequency selected by the designer.

• R_dson. The resistance of the MOSFET in the ON state. Lower is better.

• Turn on / off Time. The time it takes for the switching nodes to transistors to ON/OFF.

• Parasitic Inductance. The parasitic inductance of the MOSFET due to packaged leads. Parasitic inductance can have an impact on ringing/EMI/overshoot that can be mitigated by making loops tight.

• Input / Output Capacitance. The terminals of a MOSFET have capacitance that direct affect how fast the nodes can be switches on and off.

These nonidealities lead to a few major sources of loss that direct affect efficiency: Conduction, switching, diode, CV², gate driver, and supply-related:

Conduction (I²R) losses

Conduction loss occurs due to the IR drop across the MOSFETs Rdson during steady state operation. For a single transistor switching, if Iload is fairly flat, the conduction loss is equal to.

Where D represents the duty cycle, representing the % time the signal is high and the switch is on:

However, large current ripples increase conduction loss because this loss scales with I², so the moments where I is higher than average causes it to burn more power that when the current is lower than average. This is accounted for as a delta_Il term:

Switching losses

Switching losses occur when switching between ON and OFF as there is a region in which Vds is decreasing as Id is increasing. The loss of this region is given by:

There are two notes here:

• This loss scales with switching frequency. As you switch at a higher frequency, the switching loss increases since there are more triangle regions where energy is dissipated.

• This waveform applies to the “hard” switch case where the voltage across the switch is not 0 when it is turned on. “Soft” switching technique attempt to reduce this voltage through techniques like resonant circuits to switch at Zero voltage (ZVS) or zero current (ZCS).

CV² Losses in the switch nodes

There is power dissipation due to charges when the switch node is capacitively charged up through the input voltage:

Charge Losses in the Gate driver. This is caused by the charge required to drive the gate of the MOSFET high to be in the “ON” state:

Diode - related losses.

The diode in figure 7 has two losses associated with it.

• Diode Reverse recovery. Reverse recovery is an effect in pn-junctions in diodes (and MOSFETS as well) where charge Qrr is required to bring diode D from ON to OFF. This charge also rushes out and is seen as a tiny “blip” in the voltage waveform. This effect has a finite amount of time, and its loss is given by:

• Diode Conduction loss. When the current conducts through a diode, it experiences a forward voltage drop Vf and the loss is given by:

This loss is important to consider when dead time is discussed.

Supply Current related losses

The housekeeping and control circuits on chip itself will consume power to maintain operation:

This current draw can change depending on the state of the device (shutdown, standby, full operation, light load operation) and should be properly quantified on a datasheet to ensure there are no unexpected power leaks in the chip.

Sidebar - The Miller Plateau - Switching a single transistor

One note about switching is that is NOT a linear ramp, but rather there is a plateau that the gate voltage sits that directly affects rise / fall times.

In the resistive load case, there are three phases of switching:

• T1: Precharge phase. Enough charge accumulates on the gate to turn the transistor on when Vth is reached

• T2: Miller phase.

  • At the input, Vgs rises slowly due to the miller effect

  • At the output, Id begins to rise and Vds falls

• T3: End of Miller phase. Id has reaches its peak value to supply Rload, so VGS is able to increase to its maximum value

Lets extend these concepts to inductive loads, which is what most buck converters switch into. Note that this switching diagram assumes the inductor current is at steady state at time T0.

In the inductive load case, there are three phases of switching:

• T1: Beginning of Miller phase. Id ramps up quickly to take the inductor current As the switch is turned on, Vds begins dropping

• T2: Miller Plateau. Vgs stays in the miller “plateau” until Vds continues dropping until it hits 0

• T3: End of Miller phase . Vgs continues to reach its maximum value.

Tradeoffs

The buck converter looks simple, but in reality, buck converter design is an art due to the sheer number of interactions and tradeoffs that designers must make. Lets take a look at a few:

Switching frequency vs component size / frequency dependent losses / EMI

Increasing the switching frequency is desirable for a few reasons:

• Current ripple decreases

• Lower component values can be used since there is less on and off time

However, there are a few undesirable effects:

• Frequency-dependent losses increase (Switching, CV²*f)

• EMI increases

Therefore, an preliminarily loss across frequency analysis needs to be performed to select an appropriate switching frequency that balances performance and losses.

Dead time vs diode loss

Both switches must avoid being turned on at the same time at all costs for obvious reasons. To prevent this, a feature called “dead time” intentionally adds in a small amount of “off time” between switching cycles to guarantee that both switches will not be on at the same time.

However, during this dead time, the current has to flow through the diode of the low side FET to maintain continuous current through the inductor. This adds loss.

A small amount of deadtime is desired, but should be not too large to make the conduction loss through the diode a major loss source.

Trace and Package Parasitics

Parasitics have important implications during layout because your circuit can look like unintended LRC circuits if parasitics aren’t well controlled. Uncontrolled parasitics can result in a bunch of nasty interactions and effects such as ringing that can cause overvoltage conditions as well as diodes forward biasing. Parasitics include the following:

• Parasitic Inductance. Large current area loops have a lot of inductance. Bond wires in discrete transistors have inductance due to the area underneath the bond wire where magnetic field flows that resists changes in current.

• Parasitic Capacitance. Virtually every node has a capacitance where the voltage on a node is maintained with charge with a ground reference.

• Parasitic Resistance. Parasitic resistance shows up current carrying metal that can affect IR drop, as well as the Equivalent Series resistance (ESR) of the output capacitor

Parasitics should be added as early in the modelling process and simulated with PEX at the end to ensure proper circuit functionality.

Conclusion

Buck converters look simple on paper—two switches, an inductor, and a capacitor. But in reality, buck converter design is an art. Hardware, efficiency and stability are negotiated between switching losses, ripple, dead time, and parasitics. And that doesn’t even include the complex control schemes and advanced architectures that performance-defining products need.

That’s why PMICs aren’t one-size-fits-all: the “best” buck is the one customized to its system’s voltage range, load dynamics, noise tolerance, and thermal limits. As a result, PMICs tend to be a very competitive industry with marketing managers claiming they have the best “bang for your buck”. No pun intended :)

If you want to learn more about buck converters used in the power system of AI data centers, especially multi-phase and TLVR variants, check out my other post on “AI Data Center Power System: A Complete Overview”

Editors note (6/1/26) - Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

Before each paywall, I add links to additional posts I wrote so you can get a wholistic sense of how systems that interface with PLLs work.

This post covers the following topics:

• High level design principles

• An Intuitive Look into Type I and Type II Phase Locked loops

  • Type 1 PLL: XOR and RC LPF

  • Type 2 PLL: Charge Pump

  • Bode Plots Comparison for Type I and II PLLs

• The Transfer Function of the type II PLL

• 🔒Common transistor level implementations of type II PLLs

  • 🔒Phase-Frequency detector

  • 🔒Charge Pump

  • 🔒Voltage Controlled Oscillator

    • 🔒LC Oscillator

    • 🔒Current Starved Ring oscillator

• 🔒Important considerations: Phase Noise and Jitter

• 🔒A high-level look into the Fractional-N Synthesizer

A modern chip is like a finely tuned orchestra. Each member of the orchestra knows their role and what notes to play, but needs to know when to play them. The conductors role is to align musicians to a shared interpretation of tempo.

Imagine if there was no conductor; musicians don’t know when to play, so the orchestra gets out of sync really quickly as each person plays to their own tempo.

This is the role PLLs play in circuits: as the “conductor” that produces a clock to keep blocks synchronized to a common reference. PLLs help shape and distribute timing under real noise and delays.

The PLL is a beautifully complex and common circuit in ICs that reward high-level analog-design intuition. That is, begin at the architecture level, make sure your design is feasible, and work your way down to transistor level implementation.

At its core, phase locked loops generate clocks from a reference source. This sounds simple, but actually finds use in a wide variety of applications:

• Frequency Synthesis: Generate a range of frequency multiples for sampling, DSP, SerDes, RF LO, etc…

• Clock Recovery: Recover data from a bit stream of asynchronous data without a clock reference

• RF Channel Selection: Select channels across specific frequency channels in transceivers

The question is, why do you need a PLL anyways? Why can’t you just use a reference clock?

Well there are a number of reasons:

• Generating internal clocks from a reference- different parts of a system need different internal clock frequencies at different multiples.

• Frequency tuning - The PLL frequency can be tuned by controlling parameters

• Shaping jitter / phase noise - A PLL “reshapes” noise from a reference source and can be optimized in a way to minimize it. Inside the loop bandwidth, the reference/PFD-path noise tends to dominate; outside that bandwidth, the VCO noise dominates.

• Clock skew compensation - In systems with long signal distribution, A PLL can adjust for signal delays across the chip by adjusting its “phase” to be in sync with the original clock.

I want to give a survey level overview of the major design considerations of a PLL at different abstraction levels, from system level, to transistor level, to noise analysis. More detailed information can be found in the book “Design of CMOS Phase-Locked Loops” by B. Razavi [2].

High level design principles

The idea with system architecture is to detect problems at the highest possible level of abstraction.

Typically, at this stage, global stability is owned by the architecture. The architect models the transfer function of the PLL architecture through gain, integrator, and filter blocks.

At this stage there are several things modelled:

• Speed - how high are the frequencies needed?

• Loop bandwidth and phase margin - how fast does the loop need to be to meet the lock time / overshoot requirements?

• Tuning Range - What range of frequencies need to be supported? What VCO gain is required to support this frequency range?

• Capture range - What range can the PLL pull-in from unlocked?

• Lock/hold range - What is the frequency range over which the PLL stays locked once locked?

• Sensitivity - What are the worst case corners with respect to global / random variations? What are critical mismatch sensitivities?

• Noise - Where are the major contributors of noise to the output jitter / phase noise?

• Spurs - where are reference spurs and fractional spurs?

These issues are investigated in a high level modelling language like VerilogAMS or Simulink.

More local stability issues can be left to transistor level designers like op amp phase/gain margin and gain-bandwidth tradeoffs.

The worst thing you want is to jump to transistor level design with a process and find you can’t implement the frequency or obtain the desired phase noise with the given process!

An Intuitive Look into Type I and Type II Phase Locked loops

The basic idea behind a PLL is that it is a closed-loop system designed to synchronize the output clock to the input reference, whether that be a:

• Crystal oscillator

• External system clock

• Data stream

It does this with the following blocks:

• Phase-Frequency detector: Responds to both phase and frequency differences and generates a stream of pulses.

  • Note that “Phase detector” is used in type I PLLs like in the XOR gate, where the frequencies are already close.

• Loop Filter: Filters the input to provide a control voltage Vctrl into the voltage controlled oscillator

• Voltage Controlled Oscillator: Produces an output frequency as a proportion of the input control voltage with a loop gain:

  

• Divide-by-N Counter: Divides the output frequency down by integers of N, allowing the output frequency to be a multiple of the input frequency and the resulting divided clock to be compared against the reference.

The PLL “captures” the frequency from an initial value and adjusts itself until the reference frequency tracks the input frequency. Below is an example of a PLL locking to 7.5MHz in 80us from startup:

Ideally, when a PLL is in “lock”, the average frequency matches and the phase error settles to a small constant (nearly 0). The voltage on the VCO is a DC value with periodic “ripples” to micro adjust the frequency.

The dynamics of a PLL are similar to damped oscillator systems, with a few different “types” of PLLs:

Type 1 PLL: XOR and RC LPF

In a simple implementation, one can simply implement the above with an XOR and and RC Low pass filter.

It is type I because it only has 1 integrator:

• The VCO

So it has one pole at the origin, and another pole at the RC corner frequency.

However, there are two drawbacks:

• Poor lock acquisition - A type 1 PLL often needs the VCO to be initialized close to the target, or else acquisition can be slow or fail without an initialization mechanism

• Non-zero static phase error in lock - In lock, a Type I loop typically requires a constant phase offset to sustain the DC control voltage.

Type 2 PLL: Charge Pump

The Charge Pump is a common block used that sources / sinks current into the filter.

Charge Pump PLLs are considered Type II because this system has 2 integrators:

• The VCO

• The Charge pump current pulses + loop filter capacitor that integrates current to voltage

There are two poles at the origin, and a zero formed by the R.

C2 is added as a high frequency bypass to reduce voltage ripples across Vctrl. C2 is usually smaller, about 1/10 or 2/10 of C1.

Type II integrators can drive steady state error to 0 even with a constant frequency offset, allowing for better tracking and acquisition.

Bode Plots Comparison for Type I and II PLLs

The open loops gains are different for type I and II. Notice how in the type II case, the R introduces a zero that adds phase lead (+90 deg), improving phase margin and stability.

The Transfer Function of the type II PLL

To design a PLL, one must first understand its dynamics through a transfer function. Note that C2 is neglected from this analysis because its value is small compared to C1.

In this derivation, the loop variable is phase, and gains are expressed in the usual PLL units (PFD/CP gain and VCO gain).

The open loop transfer function is given by:

To obtain a closed loop transfer function from an open loop form H(s) and Feedback component G(s):

Substituting H(s) into the above:

There’s a lot of terms here, but the key thing is that this is a second order transfer function of the form:

With damping factor:

And natural frequency:

If you recall second order transfer functions, the output “response” can be over or underdamped.

Now that we’ve seen the blocks and built some intuition, we can add additional layers of understanding on top of it. After the paywall we’ll define the common blocks that are used, define where noise and jitter show up, and give a high-level introduction to fractional-N PLLs.

Share

In the mean time, check out my other posts to see the systems that PLLs commonly interface with, as well as how to learn more efficiently:

Read more

Editors note (6/1/26) - Moving forward, I am transitioning this Substack into a deep-dive resource on System Architecture that covers the packaging, thermal, power, and signal integrity stack. I added links to additional posts I wrote that interface with Bandgaps after I originally wrote this one. To reflect the depth and combination of the synthesis involved, the deepest technical layers of my guides will now be reserved for paid subscribers.

However, I’ll leave this post free to get a flavor for the type of depth I go into.

A bandgap reference generates a stable voltage with low sensitivity to temperature, supply, and process (often with trimming).

Most IC datasheets specify that the IC will work within a specific temperature range (such as -55 ̊ C to 125 ̊ C). Variations in temperature can shift bias points and thresholds enough to break specs without a stable reference and proper matching.

Bandgap references are found in virtually all analog circuits as stable references for high-precision analog functions, such as ADCs, DACs, and comparators. They are also used to bias current mirrors for setting bias points so that the resulting circuit is robust to PVT variations. Because of their ubiquity, they tend to be reused and modified depending on the fab process and the level of precision required.

Bandgaps look simple, but are a delicate balance of a few fundamental analog principles, including:

• Matching: resistor ratios, BJT area ratios, amplifier offsets

• Feedback: Forcing ΔV_BE and setting I_PTAT robustly

• Temperature variation: How BJT and resistor characteristics vary over temperature, sometimes nonlinearly

• Trim & Calibration: Output accuracy, temp trim, and curvature correction

Below is a self-biased bandgap design example I built that illustrates these principles. I also have a fun side story at the end about how a group of young designers “competed” on who can make the best bandgap.

Circuit Analysis

Consider the Brokaw Bandgap reference in Fig. 1. The reference voltage is the output of the op amp. This circuit sums a PTAT voltage and a CTAT voltage:

• A PTAT voltage (Proportional to absolute temperature) means that the voltage increases with increasing temperature.

  • The PTAT voltage is generated by the BJT base - emitter voltage difference ΔV_BE = Vt*ln(n) by forcing two BJTs to operate at different current densities (often via an emitter-area ratio n). This voltage appears across R3 in the above example.

• A CTAT voltage (Complementary to absolute temperature) means that the voltage decreases with increasing temperature.

  • The CTAT voltage is generated across the base-emitter of the BJT which behaves like a diode. The reason behind this is related to the fact that the bandgap energy of silicon decreases with increasing temperature.

Additionally, the op-amp sets the PTAT current by driving the loop so its input nodes match (in the ideal case).

When both PTAT and CTAT voltages are summed with the right scaling, the first-order temperature dependence largely cancels, leaving a much flatter reference over temperature.

Derivation of Output Voltage

The derivation of the output voltage is as follows. Starting from KVL:

The current through the circuit flows through R2, where the voltage across it is the difference of the base-emitter voltages of Q and Q1:

Is is the saturation current of the diode. Substituting all of the terms:

The op amp uses negative feedback to ensure that I1=I2, so both terms can be cancelled. However, Is1 and Is2 are proportional to the cross sectional area of the BJT. Therefore,

where n is the number of BJTs in parallel for Q₀. The output voltage is given by:

The output voltage is the sum of a PTAT voltage (Vt) times a gain term and a CTAT voltage Vbe1. However, the temperature variations are not the same. The temperature variations of each term are:

The values in the term R3/R2*log(n) are selected such that the dV/dt of both terms cancel each other:

• n = 8 is common because 8 unit devices plus 1 unit device can be arranged symmetrically (e.g., a 3×3 array) to improve matching.

• The resistor ratio R3/R2 is selected to be equal to 5.5.

Using a bias current of 55 uA and Is1 = 6.734 fA from the SPICE model, and substituting all of these values in the equation above:

• Vbe1 = .59V.

• V_ref at 27 °C = 1.18V.

Circuit Schematic and Simulation Results

The circuit consists of the bandgap core which generates the PTAT and CTAT voltages. Both collector voltages are fed into an op amp which uses a PMOS differential pair as the first stage. This is done to avoid the VMIN problem an NMOS based op amp would experience since the common mode input voltage is approximately 1.18V.

The output of the first stage is fed into a second stage where the output is referenced to the VDD rail to control the PMOS current sources. The op amp is biased with a PMOS based cascode current mirror of 20uA. All of the DC voltages and currents are shown at T = 27 °C.

The simulation results of the bandgap reference are shown below.

Output voltage across temperature. The graph above shows the DC value of Vref as temperature swept from -55 °C to 125°C in 1°C increments. Ideally, the voltage should be flat across temperature, but the simulation shows the voltage follows a parabolic shape. The maximum voltage is 1.1885V which occurs at approximately 30 °C. The voltage drops for lower and higher temperature, where the minimum voltage is approximately 1.1869V. When the voltage is confined to this range, the maximum voltage difference is 1.6 mV, which is a variation of 0.14% over temp.

Line Regulation. This simulation sweeps the value of the voltage supply for three different temperatures: -55 °C (left curve), 27°C (middle curve), and 125 °C (right curve). The reference voltage stabilizes when the power supply has reached a certain threshold. This threshold is different for all three temperatures and is shown in Table 1. Once the power supply voltage has reached its threshold, the reference voltage is fairly stable.

Power supply rejection ratio, or PSRR. The PSRR is the ability for the circuit to reject noise from the power supply. It is defined as:

To perform this simulation, an AC signal of 1 mV was places in series with the power supply voltage. An AC sweep was performed for three temperature values from 100Hz to 100MHz and the magnitude of the output voltage was observed. The above formula was then applied to obtain the plots shown above, with results shown in table 1.

Mismatch in R10 Resistor. The resistance was swept parametrically from 5kΩ and 6kΩ in 100Ω increments and the output voltage is plotted. This simulation shows that for low resistor values, the CTAT voltage of the BJT dominates the characteristic, and the curve drops over temperature. When R10 is high, the PTAT voltage dominates, and the reference curves increase over temperature.

The following chart summarizes the simulation results.

Additional Features

There are additional features that bandgaps need:

Trim Range. The degree to how precise a bandgap has to be depends on the precision required across PVT. Monte carlo simulations are run to determine the untrimmed range over which the output voltage can vary, and trim features are added in appropriate places to ensure the output voltage can be turned to within spec across the temperature range.

Curvature Correction. Parts that operate in wide temperature ranges with tight specs can require additional “curvature correction” to improve the error across temperature. First order cancellation like resistor-only tuning can give residual curvature because parameters can have nonlinearities across temperature (such as BJTs) that affect the output.

Startup Circuitry. Because the core is self-biased, it has a zero-current equilibrium point, so startup circuitry is required to guarantee the intended operating point.

Story time: A Friendly Bandgap Competition

At an Analog Design internship, I once saw a group of young engineers have a friendly completion against each over who can design a bandgap with the highest accuracy across temperature. Work was to be done after 2-3pm on Friday so as to not interfere with day to day work.

Most of the “tricks” they experimented with involved:

• curvature correction techniques, like how to bias a transistor to turn on as a specific temperature, and where to place in the circuit

• NPN / PNP topologies

• value ratios (n, resistors)

• what current mirror to use (cascode, wide swing, source degenerated, wilson)

• what amplifier topology (NMOS, PMOS, gain stages, active loads) maximize gain and minimize offset

Each participant presented their results weekly and learned insights from each other to improve their day job, since analog is an art after all.

Conclusion

I feel like designing bandgaps are a really good introduction to analog circuit design because designing them develops a lot of the fundamentals used in more complex analog circuits.

Share

If you’re curious about how bandgap references are used in higher level systems, check out my other posts here:

References

[1] Timm, S. Wickmann, A., “A Trimmable Precision Bandgap Reference on 180nm CMOS”, Semiconductor Conference Dresden-Grenoble (ISCDG), 2013 International, 26-27 Dept. 2013, http://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=6656292

[2] Hugo. “Bandgap voltage Reference” [Online]. Available: http://www.onmyphd.com/?p=bandgap.reference

[3] P. Miller and D. Moore. (2005). “Precision voltage references” [Online] . Available: http://www.ti.com/lit/an/slyt183/slyt183.pdf\

[4] A. Paul Brokaw. (2011). “How to make a bandgap reference in one easy lesson” [Online]. Available: https://www.idt.com/document/whp/how-make-bandgap-voltage-reference-one-easy-lesson-paul-brokaw

This primer focuses on AI accelerators: hardware designed to run deep learning efficiently. We’ll build intuition from the ground up (MACs → DNN/CNN structure → memory), explain the Von Neumann bottleneck, introduce the key performance metrics, and then ground everything in a classic example: Eyeriss.

This post synthesizes ideas primarily from “Efficient Processing of Deep Neural Networks” and “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks.” Figures are adapted for educational commentary with attribution in captions; please see the original papers for full detail.

The compute: The Multiply- accumulate

The Multiply - accumulation operation is the heart of most deep learning applications.

In simplest form,

then applies a nonlinearity (often ReLU). At the output, sigmoid (binary) or softmax (multi-class) can convert logits into probabilities.

There are few intuitions I use the thinking about Neural networks:

• MAC operations are analogous to biological neurons: many inputs influence an output, and the weights determine what patterns the unit responds to. (It’s an analogy—not a literal model of the brain—but it’s a useful intuition.)

• A neuron (dot product + nonlinearity) behaves like a pattern detector where the output is high in response to a input pattern that correlates with the combination of weights.

This multiple accumulate operation can be found across many range of deep learning applications.

Feedforward Neural Networks

In feedforward neural networks, many of these neurons are assembled in parallel in a “pipelined” fashion. The inputs all connect to each “neuron”, each with its own unique set of weights to detect specific patterns, and these neurons then pass that information to later layers or the output. Additional layers can be added to be able to fine tune the pattern detection.

Adding “hidden” layers between the input and output allows the network to build up from simple features (edges/textures) to more complex features (higher-level shapes and objects).

There are two major operations performed on neural networks:

• Training = learning the weights by passing images through, computing the error, and tuning the weights through “backpropagation”

• Inference = running a trained model to produce outputs

Inference is commonly run on edge devices locally; training is usually done on large GPU/accelerator systems. Both training and inference can be run on GPUs or specialized accelerators optimized for these workloads.

Convolutional Neural networks

Convolutional neural networks (CNNs) are widely used for image recognition and classification. Instead of connecting every pixel to every neuron (which would explode parameter count and compute), CNNs use small filters that slide across the input in strides.

Each layer of the Convolutional Neural Networks operates in the following way:

• Small filters represented as matrices can be customized to detect specific features, but the simplest case are “edge detectors” that detects vertical lines, horizontal lines, and diagonal dimensions.

• These filters are “convolved” with each each input feature map (ifmaps), which is image pixel that is most commonly represented by a tensor of RGB values.

• The output feature map (ofmaps) is calculated by sliding the filter across the input and accumulating partial sums (psums) along the way.

• This feature map is condensed through normalization and “pooling” operations.

Each output feature map corresponds to a filter where the particular pattern is present (such as horizontal/vertical/diagonal line segments). As layers get deeper, higher order constructs are formed from simpler ones and spatial resolution shrinks. When dimensions are small enough, the CNN is connected to a fully connected layer that generates an output.

There are computational observations in both DNNs and CNNs as it relates to hardware:

• Filter weights can be reused in computation without needing to retrieve it from memory every time.

• The tensor shapes and data dependencies are known ahead of time, so you can pre-plan tiling and data movement to minimize expensive memory traffic.

These two points will become important later when we discuss Eyeriss.

The Von Neumann bottleneck

You have the best compute in the world, but it doesn’t matter if memory can’t keep up.

Most computing nowadays uses a Von Neumann architecture, where the compute and memory are split into two entities. Data stored in memory is retrieved and stored locally into CPU memory to perform operations on. When the result is computed, it is written back to the memory.

This architecture has worked as compute and memory are scaled. However, there are challenges being faced today:

• Compute throughput has scaled faster than memory bandwidth/latency improvements

• There is a limit transfer rate and a high energy cost per memory retrieval

This results in a bottleneck, where compute is limited by bandwidth + high energy per byte moved between memory and compute.

The question is, why don’t you put compute and memory on the same chip? You could, but its often not cost effective. Fab process for compute and memory optimize different things:

• Logic (CPU/GPUs) are optimized for speed of transistor switching and ease of routing interconnects through stacking metal layers

• Memory requires high density storage and capacity

One method classical computer architecture employs to mitigate this issue is arranging memory in a hierarchy through caches. Here, data is retrieved from the main memory only as “needed”. This exploits a phenomenon called “spatial locality” where you are likely to retrieve data close to the one you retrieved. As a result, you can write a whole data “chunk” of memory to a cache to increases the chances that you “hit” data in the cache when going to retrieve it.

AI Accelerators exploit both spatial locality and data reuse. In order to take advantage of both, we need to understand two mature memory technologies used across the hierarchy: DRAM, and SRAM.

The Memory: DRAM and SRAM

In a naïve implementation, each MAC operation requires three memory reads and one memory write:

• Read: weight

• Read: activation

• Read: accumulated partial sum

• Write: updated partial sum

To store data into memory, there are two mature memory technologies available:

1. DRAM - Dynamic Random access memory

   

   DRAM consists of capacitors that store charge and are read / written to word lines through a transistor. These memory cells are are typically arrayed in a 2D grid of cells with vertical row lines and horizontal bit lines. Memory cells are places at the intersection of word lines and bit lines where each ones is accessed by a combination of specific word line and bit line.

   Pros: DRAM is cheap and dense.

   Cons: DRAM is volatile and requires periodic refresh because capacitor charge leaks. It also requires constant electrical power to hold data.

2. SRAM - Static Random access memory

   

   SRAM consists of cross coupled inverters in a latch (or positive feedback) configuration. The most common configuration is a 6 transistor cross coupled inverters to hold data. To read and store data, bitlines are charged and the wordline (WL) is activated depending on the values being read/written from/to the cell. The bitline needs to be strong (i.e. generate a lot of current) during write operations to break the feedback loop of the latches in order to flip the state of the memory cell. As with DRAM, these cells are arranged in a grid with these cells at the intersections.

   Pros SRAM is faster and lower energy per access than DRAM. SRAM can also be easier to integrate on compute processes.

   Cons: SRAM is much less dense and more expensive per bit.

Cost of memory: energy and bandwidth

There are two issues with DRAM:

• The energy cost of DRAM is 1-2 orders of magnitude more expensive to access than on chip memory [1]. There are several factors that contribute to this, one of the largest being constant refresh cycles needed when non-ideal capacitors discharge through leakage paths.

• The data bandwidth of DRAM is finite. In modern computer designs, DRAM is separate from the compute chips (CPU, GPU) and connected via busses with finite latency and throughput capabilities. In addition, DRAM itself has high latency due to steps of row activation, command delays, and precharging.

Due to the high energy cost, it is not desirable to read / write from DRAM continuously. Its better to read/write a “chunk” of information at a time and store the data locally on the chip to be accessed at memory technologies with lower energy cost.

There are a few trends that have been going on improve performance without significant changes to the Von-Neumann paradigm:

• High bandwidth memory (HBM) - stacked DRAM integrated closely with the compute package to increase bandwidth.

• Optimizing memory usage on chip into a memory “hierarchy”. When data is know in advance, it is possible to structure the data in a way that optimizes the memory cost and schedule data as such.

• “Compute in/near memory” which uses more efficient memory technologies designed to handle this type of computation, or architectures design to do the computation by storing weights in conductance of resistors. There are many intriguing niche approaches here [2] but are beyond the scope of this post.

Performance metrics for DNN Models

“Performance” in a deep learning system consists of how performance is defined on both the model side and the Hardware side. Both are important and closely intertwined.

Metrics for DNN Models include:

• Accuracy and robustness - How accurate is the model on common “benchmark” datasets

• Network architecture - The number of layers, filters, and sizes

• Number of weights - Affects storage requirements

• Number of MAC operations

Often times, performance is compared in terms of normalized energy per MAC to give an apples to apples comparison of performance across different models.

Performance metrics for AI Accelerators

Hardware optimized for AI computation has its own set of performance metrics that that are optimized depending on the application:

• Energy and Power is important for most applications, including cloud data centers with stringent power ceilings, as well as portable devices with limited battery

• Low latency is important in real time interactive applications

• High throughput is important for big data analytics is important when action needs to be taken based on the data

• Hardware cost is important for most applications, when tightly integrating an AI chip on a consumer device to run inference, as well as integrating into a blade on a rack in a data center

Bridging Compute and Memory: Eyeriss [3]

Eyeriss is a classic AI accelerator that optimizes for the deterministic nature of AI calculations. If you know the size and shapes of the filters, ifmaps, etc. you can optimize your flow to a given hardware set based on the size of the array. By doing so, you minimize the energy cost of moving data around.

Eyeriss has a few key building blocks:

• Processing Engine (PE)

  

  Each processing engine consists of a 16b two stage pipelined multiplier and adder, along with scratch pad memory to temporarily store the weights.

• Data parallelism through Network on chip (NoC)

  • There are 168 processing elements arranged in a Spatial architecture of 12x14.

  • Each PE can communicate with neighboring PEs or the GLB through an NoC

  • The global input network handles the process of distributing the computations to each of the PEs though busses

  • Processing in each PE can be run independently from each other with data movement coordinated by the NoC

• Four-level Memory Hierarchy

  The memory hierarchy is arranged into four-level hierarchy in order of decreasing energy:

  • Off Chip DRAM

  • 108kB global buffer

  • NoC reuse paths (neighbor PE communication)

  • Scratch pads local to PE (520 bytes each)