AI will not replace chip designers anytime soon. In fact, AI has made chip design even harder in many ways. This post will illustrate why from first principles by covering the following topics:

• High Level Overview of VLSI Design Process

• Vibe Coding RTL – Systolic Array example

  • Benefits and Risks

• Why Chip Design Takes So Long: Cost of Error due to Fab Bottlenecks

• Key Bottlenecks that Limit AI use in Chip Design to Quality-of-Life Enhancements

  • 🔒Functional Verification

  • 🔒Physical Co-design: Modularity and Thermal/PDN/SI/Packaging Constraints

  • 🔒AI-Driven Design Space Exploration

  • 🔒The Hidden “Memory Wall” - Compute Resource Bottlenecks

I went to the floor of the Design Automation Conference on July 26-29 in Long Beach, CA. This is conference #5 of a broader conference tour this year (ISSCC, DesignCon, APEC, and ECTC) where I’ve worked my way up in abstraction levels to better understand the complete AI datacenter stack and educate you on it.

I’ve noticed a lot of prominent writers who don’t know much about chip design speculate about how Agentic AI will impact it. Most people assume that AI will have the same disruptive impact in the SW and chip industries since software and RTL code can easily be generated with LLMs, creating a lot of FUD.

I’ll highlight the benefits of AI with a vibe coding tutorial of RTL and how chip designers can benefit. Then I’ll show the critical flow differences in chip design that need to be scrutinized and cause veteran engineers to not completely trust AI.

Please note that I have more experience in analog / mixed signal design, not digital design. I have interfaced with digital / verification people and written my share of VerilogAMS to have a working understanding of the VLSI design flow. I reference the book “VLSI Physical Design: From Graph Partitioning to Timing Closure” for more detailed flow information.

As always, if you’re an expert in this area and notice a mistake, please reach out to me so I can have it promptly corrected.

High Level Overview of VLSI Design Flow

Chips have become much more complex over the years due to a well known phenomenon of Moore’s law. Moore’s law enables more transistors to be placed and doubling compute density about every two years. Very Large Scale Integration (VLSI) is the umbrella term used to refer to the tight integration density of the various classes of chips (SoC, GPU, ASIC, FPGA).

Automation is necessary for chip design because of the sheer design complexity and errors in chip designs are much harder and costly to fix that board-level errors. As a result, the entire EDA industry revolves around robust automation tools to perform a lot of complicated steps automatically. Only low-level JP and customized analog blocks tend to be hand drawn and instantiated by the tool at higher lever assemblies.

At a DesignCon keynote “Agentic AI for Chip Design”, Mark Ren, founder and CEO of AGENTRYS, notes how EDA tools flow raised the abstraction level that humans operate at. AI enables an even higher level of abstraction by allowing engineers to specify design intent in plain language rather than a specific programming language.

The goal of the VLSI design process is to produce a fabricable, error free mask that implements the desired system specification within the constraints of the fabrication Process Development Kit (PDK). The VLSI design process evolved over time to incorporate the following major steps:

• System Specification. The overall high level goals of the system are defined collectively amongst chip architects, product marketers, and ASIC engineers

• Architectural Design. An architecture must be defined to achieve the high level goals of the system. These include:

  • Analog/Mixed-Signal Blocks - ADCs/DACs, regulators

  • Digital Blocks - DSPs, and types of computational blocks

  • Interfaces - I2C, UCIe, PCIe

  • Hard / Soft IP

  • Packaging

  • Clocking Architecture

  • Process Node

• Functional Design and Logic Design - Logic design is performed at the RTL level using a HDL such as Verilog or VHDL.

• Custom Circuit Design - A handful of critical blocks such as ESD protection, I/O, analog circuits, and RAM blocks have to be hand-designed.

  • Companies can develop these IP in house with library / IP teams across major projects or license IP from specific vendors.

  • In some cases of one-time jobs, such as customized analog blocks, companies can utilize contract labor to complete the work if there is not a steady workload for a W2 employee.

• Physical Design - The RTL is synthesized and billions of design components get instantiated with their geometric representations using highly sophisticated Place-and-Route Tools. Physical design is highly iterative and includes the following steps:

  • Partitioning

  • Floorplanning

  • Placement

  • Clock tree Synthesis

  • Routing

  • Timing / Power Closure

• Physical Verification and Signoff. The layout needs to be checked against different design rules defined by the fab (DRC, LVS)

  • The design needs to be officially “signed off” by specific tools to keep a paper trail should errors arise down the line. These tools often have valuable information in the logfiles that can pinpoint sources of errors. Design waivers can be made to in special circumstances to overcome tool constraints and are agreed upon in design reviews, but are generally avoided at all costs.

• Fabrication - The final layout is streamed out as a GDSII Stream format and is sent to a fab.

• Packaging and Testing - The chips are packaged using appropriate package types such as DIPs, PGA, BPA, flip chip, or WLCSP.

The design flow is iterative, with engineers making design choices at each step based on simulation feedback before handing off the work to the next step.

Chip design has historically been siloed with well-defined specialties and tool flows developed within each specialty to maintain consistent flows along process node improvements. This approach has worked to scale chip performance for a number of years, but the slowing down of Moore’s law and multiphysics constraints had led to a paradigm shift toward cross-domain engineering. This approach requires a shift in engineering mindset that is not always easy to transition to due to entrenched organizational incentives. This substack aims to facilitate that mindset shift by making cross-domain information easier to access.

Vibe Coding RTL – Systolic Array example

Before I talk about the challenges of AI for chip design, I’ll highlight a positive example of how AI can benefit chip designers: code generation of RTL.

At the beginning of DAC, Steven Truitt, Nan Wu, and Liu Liu gave a very good tutorial “From Vibe Coding to Silicon: Engineering Agentic Systems for Hardware Design” using Microsoft Discovery. Their GitHub is public at the following link that you can check out:

https://github.com/stevetru1/DAC2026_Vibes

https://github.com/ORCA-lab/DAC26_VibeCoding_Runs

What is Vibe Engineering

Vibe engineering is the hottest trend in SW engineering that enables non-SW engineers to code SW and offers productivity enhancements for junior engineer-level tasks. Vibe engineering can also apply to the SW portions of chip design, including RTL coding.

Vibe engineering involves thinking an abstraction level higher than traditional engineering. You rigorously describe your intent in terms of the desired outcome, rather than the programming language, for your agent to perform. The design is iterated on through conversation and feedback loops based on what the agent produces.

Here is an example prompt template where the desired outcomes are rigorously specified for the agent.

One important consideration in vibe engineering is defining the definition of done. Task organization up front is extremely important to properly decompose, organize tasks, and build the harness flow for the agent to execute in. The test and evaluation suite is also very important to evaluate agent performance properly.

Vibe engineering places more emphasis on judgement in execution rather than execution expertise. The challenge is figuring out the right thing to build, instead of building faster. As a result, the human layer carries more weight.

Example: Systolic Array

One key block in AI accelerated ML applications is the systolic array. Systolic arrays are often used for AI acceleration and are typically co-designed to the specific data needs. I’ve written on Systolic arrays previously in my other posts.

A description a vibe coder might give an agent can look something like this:

Now I’ll admit: I hate writing RTL code, even though I have taken a Verilog and VHDL class, so I find this vibe engineering approach such a breath of fresh air.

More detailed information can be found in the slide decks and tutorials; I’ll leave it up to you to check those out.

Why Chip Design Takes So Long: Cost of Error due to Fab Bottlenecks

Chip Design is one of the most brutally unforgiving industries because chip design is very expensive, time consuming, and carries a high cost of error.

In board-level design, any design errors can be easily fixed after fabrication because the device is more interpretable and technicians can workaround issues such as adding jumpers and lifting off pins. As a result, there is less scrutiny during board design because components with datasheets can be hooked up with passives in between and simulated in SPICE at a high level.

Chip design has way more components than boards and does not offer that deep of a debug capability to probe deep inside the circuit. Test modes and circuits for DFT must be inserted such as scan-chains and test mode registers. It takes 3 - 4 months for a chip to be fabricated because each individual fabrication step can take days due to manufacturing queue times and tool bottlenecks.

First Pass Silicon Success

Generally speaking, there is a low pass rate of first pass success, around 14% nowadays. This might come at a surprise for outsiders, but not unexpected for engineers as schedules typically account for this. For most practical purposes, the first silicon revision is used to identify bugs that weren't caught early on, and the second revision is used to fix them. Two revisions are generally considered standard, with some CEOs like Intel’s Lip-Bu Tan threatening to fire engineers who exceed this.

There are a few ways to minimize time delays and errors:

• Pre-Silicon Emulation. The design is loaded on an FPGA or hardware array to make sure the design is bug-free. Emulation is much faster than computation-heavy simulation and typically run at lower clock speeds to test specific test cases and interactions within blocks.

• Pre-Silicon Prototyping. Not only does the chip has to work as designed, it also has to work as intended in the broader system context. Prototyping serves an important function to ensure you build the system you intended, not the bug free one you built.

• IP Reuse. Almost all new chip designs have a starting point with known good IP blocks. Low risk IP can be incorporated in a new design, and new IP to implement the performance requirements needs to be modified or created out of scratch. The overall design risk is assessed based on the risk of existing and new IP blocks themselves.

• Wafer Staging. During fabrication, several wafers are “staged” where only the time-consuming FEOL layers are formed. In case of bugs identified early that can be fixed in metal-only, this shortens the fab time to get wafers back without having to go through the entire PnR flow all over again.

  • Companies generally name revisions A0, A1, B0, where A0 is the first revision, A1 represents a metal change, and B0 represents a whole new silicon revision with a new mask set.

  • The fabrication cost and time is largely dictated by the # of masks since each tool loads a specific mask.

First pass silicon success is really only feasible when either the chip is simple enough, there is enough existing IP that is leveraged, or there are heavy emulation/prototyping efforts. Some companies who want to specifically integrate specific circuits want to see a test chip done first, key data taken, and risk evaluated before committing the chip to be integrated in their product in the next revision. General market products can be a bit more lax.

Scar Tissue

The long fab time and cost of error leads to a less discussed phenomenon in the chip industry: scar tissue, which is a form of lessons learned painfully through delays and expenses.

Scar tissue is one big reason why experienced silicon veterans often push back against claims that AI will “autonomously design chips” because it doesn’t carry accountability. This lack of accountability for AI tools means that it won’t replace human designers anytime soon.

Key Bottlenecks That Limit AI use in Chip Design

I’ll discuss four key bottlenecks that several panelists at DAC raised and promising solutions within them.

Functional Verification

Most people outside of the semiconductor industry don’t know that verification, not design, has historically been the most labor and time intensive task. Let me give a bit of context to what verification should be, and how its implemented in practice.

Share

Read more

Editors Note: (7/22/26): I trimmed the paywall to only paywall the scaling challenges with Analog CiM. This way, you can come away with a mental framework for the components of the Analog CiM.

Compute-in-memory has emerged as a promising technology to overcome the Von-Neumann bottlenecks of conventional digital computing. Many venture-backed startups have entered this space with promising solutions to enhance compute efficiency.

In this post I’ll cover a broad overview of the AI accelerator landscape and do a deep dive into the challenges with scaling Analog Compute-in-memory. The outline of this post is as follows:

• A Survey of the AI Accelerator landscape

  • Application Spaces - from VLP to Data Center Systems

  • Scaling Trends - Co-Design and Specialized Architecture

  • Hardware Classifications of Compute in/near memory AI Accelerators

    • Parallel Threads (NVIDIA GPU)

    • Tensor Array (Google TPU, Groq)

    • Manycore Accelerators (Cerebras, Tenstorrent)

  • Digital Compute-in-Memory

• The Building Blocks of Analog Compute-in-Memory

  • SRAM Arrays

  • Input Encoding

  • Charge / Current Domain Devices

• 🔒Scaling Challenges with Analog CiM

  • 🔒Array Nonidealities that degrade SNR

  • 🔒Area Inefficiency

  • 🔒Modality Conversion Overhead

  • 🔒Example CiM startup: Encharge AI

• 🔒BONUS: The Challenges of Optical Compute for AI

Back in February 2026 at ISSCC in San Francisco, CA, I personally sat in the tutorial “Fundamentals of Compute-in-Memory: From Circuits to Systems” by Prof. Jaydeep P. Kulkarni at the beginning and an all-day forum “Analog for AI and AI for Analog” at the end. Several heavyweights in the analog community such as Prof. Naveen Verma and EDA community such as Prof. David Pan were present to discuss trends in Analog CiM and Automation for AMS/RFIC Signal Design, the latter of which I will eventually cover after DAC.

Personally, analog compute-in-memory is one of my favorite topics because it almost resembled the way the brain processes data. Analog CiM is perhaps the most interdisciplinary topics among semiconductor specialties because it combines knowledge from digital, mixed signal, logic fabrication, and memory domains. I think the circuits are inherently power efficient for repetitive computations.

However, the CiM space is filled with a lot of dense academic research and marketing claims of VC-backed companies. During those ISSCC sessions and while researching CiM papers, I initially found CiM difficult to digest because there aren’t universally agreed on standards to classify related architectures. The terminology is also inconsistent; some authors use compute-in memory, others use processing-in-memory, and other add subcategories of their own. Moreover, most papers tend to jump right into the block diagrams of specific applications known among academics in that space. I’m sure many of you also feel the same way trying to understand the space.

In my opinion, CiM is a space where broad system benefits can be lost in the vast chasm between marketing claims and actual technical challenges. I believe that researchers are candid about the real engineering challenges. However, claims of performance benefits tend to narrowly focus on the energy savings of ONE part of the whole computational energy budget. These claims tend to get picked up and hyped up by non-technical people without their proper nuances, which, for some companies, is a marketing strategy all in itself.

Having sat in those rooms and felt the confusion myself, I aim to bring clarity to the space. Here are some of my previous posts I’ve written that lay a foundation for this material from both the software and hardware side:

A Survey of the AI Accelerator Landscape

GenAI and Agentic AI models have exploded in the past few years with a combination of gigantic training sets and powerful models to handle a variety of AI applications. This creates a huge demand for tokens for Agentic AI that is primarily satisfied by high power, general-purpose data center GPUs in data centers.

The underlying hardware is becoming more powerful to satisfy this demand. However, the logic is scaling much faster than the memory and interconnect by approximately ~2x. This means that the compute bottleneck is not in the compute itself, but the data movement. As models become more powerful, memory transfer operations are consuming more and more energy and latency.

As a result in the explosion of compute power, co-packaged optics and vertical power delivery are emerging and widely discussed trends to satisfy these demands. I have written extensively on these topics on my website.

One of the main arguments for CiM is to minimize the power overhead in compute-memory transfers. Here is a prominent slide given in 2014 with ballpark energy consumption of various computational operations and memory accesses. Notice the order of magnitude difference between the energy used in cache and DRAM accesses, and the higher energy consumption for higher precision operations. This chart has driven a trend toward lower precision computations in the digital domain and memory efficient architectures.

Application Space - from VLP to Data Center Systems

AI accelerators are not just limited to centralized data center GPUs, but also embedded applications with many different form factors and use cases.

Here we notice a range of publicly announced accelerators that range in compute efficiency from 0.1 - 100 TOPs/W. Notice how there is a cluster primarily centered around 1 TeraOps / W for general purpose GPUs data centers. Also notice how at lower power, accelerators specialized for embedded applications tend to have higher TOPs/W because they are tailored for the specific power / performance / form factor requirements of the system they are in. In those cases, it doesn’t make sense to stream data to a centralized data center to run inference on.

General purpose GPUs are built to support both training and inference across a broad range of AI models targeted for different applications, depending on the tasks you throw at it. Two common applications include:

• CNNs: Uses deep neural networks to extract lower level features and are used for image / speech classification

• LLMs: Uses attention layers to calculate relations between words using Query, Key, and Value vectors and require large KV cache storage for long sequences

Scaling Trends - Co-design and Specialized Architecture

Massive scale-up of AI clusters are driving a huge need for compute efficiency. Large networks of parallel general-purpose GPUs move a lot of data around, consuming vast amounts of power and generating a lot of heat in the process.

To address this need for more compute efficiency, the industry is shifting from general purpose to hardware-software Co-Design (or co-optimization) of both DNN Models and Hardware.

This involves customizing the HW to be as efficient at computing its specific task. Some optimization examples include:

• Memory: Higher bandwidth and more tightly integrated memory

• On the SW Side:

  • More aggressive quantization / lower precision architectures

  • Sparse weights

• HW architecture: Stationary architectures that “fix” one value in an array and stream in the other (weight, output, and input)

In general, all else being equal, there is a tradeoff between compute efficiency and how specialized the hardware is. Specialized hardware can augment general purpose GPUs but run the risk of being considered “one trick ponies” if data needs change in the future.

I personally think what most people describe as “co-design” should extend beyond the HW-SW interface into the broader high speed communication, power, and packaging subsystem. That is what my website aims to achieve.

A Taxonomy of the Compute-in-Memory Landscape

In classical computer architectures, main memory is stored on a huge chunk of off-chip DRAM (HBM) and chunks of data are read onto on-chip caches (typically SRAM) when requested. When MAC operations occur, intermediate results needs to be stored in registers until the final result is computed on.

Depending on how much you read/write data to/from memory, up to 30-70% (or even higher) of the compute energy can from memory transfers, not the compute itself. This makes efforts to optimize / trim data transfers between compute and memory such an attractive option to scale compute efficiency along with performance.

There are three major architectural classifications that classify how close the memory is to the bit cell:

• Compute with large memory

• Compute near memory

• Compute-in-memory

We’ll go through examples within each classification.

Compute with Large Memory

Compute with large memory is the easiest solution to scale in existing paradigms with a large GPU and HBM side by side.

There is a wide range of chip architectures within this compute fabric that vary in specificity and programmability. Most AI accelerators fall in between two conventional categories:

• CPUs: Very flexible compute-wise but not optimized for highly parallel inference and training computations.

• FPGAs: Used off-the-shelf and easily repurposable for specific embedded applications. FPGAs are preferred for faster time-to-market over custom ASICs since they can be modified with RTL. However, for large models AI, FPGAs are not “programmable” in the SW sense because the hardware effectively changes every time the FPGA is programmed.

The bulk of compute-efficient AI accelerator products lie in-between these two categories to enable software, hardware, and algorithm co-optimization. Two such categories in between fall in the “compute with large memory” paradigm:

• Parallel thread accelerators

  • Use a temporal architecture with traditional ALUs computed in a SIMT (Single Instruction, Multiple Threads / Data ) paradigm

    • Typically, one CUDA-thread processes one element of the target data set

    • All threads in a process share the same memory heap

  • Key Features

    • Warp Scheduler - controls a large number of ALU’s by broadcasting a single instruction to all of them to execute in lockstep

    • Memory Coalescing - combines multiple memory requests from parallel threads into a single, highly efficient memory transaction from the HBM

      • Memory coalescing is critically important for making data transfer efficient from the HBM

  • Advantages

    • Quick to program

    • The most flexible to handle a wide range of AI applications

    • Massive parallelism

  • Disadvantages

    • Low compute efficiency

    • Needs sophisticated control circuitry to coordinately data movement across ALUs and memory

      

  • Examples

    • NVIDIA H100

    • AMD Instinct MI250X

    • Intel Arc 770

    Generally speaking, powerful GPUs are used for heavier, general purpose tasks that require large complex models.

  

• Tensor Array

  • Tensors are mathematical objects that generalizes scalars, vectors, and matrices to higher dimensions

  • Tensor Arrays use Systolic Arrays or static spatial pipelines. A systolic array is a 2D grid of tightly integrated Processing Elements (PEs) which are a more specialized form of ALUs tailored for MAC operations.

  • A classic example of one is Eyeriss which I described in more detail in my previous post.

  • Key Features

    • Static scheduling - preload model parameters into accelerator before executing inference/training on them

  • Advantages

    • Data reuse - data doesn’t have to be stored in registers and can flow between ALUs that perform either Multiply or Accumulate operations

    • Dense Compute Density

    • Latency is deterministic - no complexity overhead from unpredictable scheduling

  • Disadvantages

    • Rigid, inflexible structure. The array size must be tailored to the specific application

    • Inefficient at handling sparsity. The sparse data still has to flow through the array

    • SW compiler complexity

  • Examples

    • Google TPU - showcased the first TPU in 2017

    • Groq LPU

    • Intel Gaudi

    In short, Tensor arrays have the potential to be computationally efficient, but only for specific array sizes.

Compute Near Memory - Manycore

Manycore is an example of compute-near-memory. This architecture decentralizes both compute and memory arrays them along a grid. This shortens the data path from the local SRAM memory to the compute units.

Prominent examples of manycore include:

• Cerebras

  • Perhaps the most well-known example of manycore with cores arranged on a wafer scale engine (WSE) with 850k cores

  • The memory is fully distributed, with the SRAM using 50% of the die area that provides full memory bandwidth to all data paths at full performance, removing the need for data reuse

  • Their WSE is efficient for sparse matrix operations

• Tenstorrent

  • Tenstorrent has a very similar architecture to Cerebras, which fuses a NoC interface to move data to local SRAM to be computed on in each Tensix core.

Advantages

• Memory path is short between compute and memory

Disadvantages

• Additional software overhead to coordinate data movement to individual cores

Other accelerator options

There are other accelerator options available:

• Neural Processing Unit (NPU): NPUs are built specifically to accelerate AI tasks directly on devices like smartphones and laptops

• RISC-V processors: RISC-V is a open source ISA widely used among academia and emerging startups. Its modular nature allows easier integration capabilities with embedded applications.

Within the CiM space there are two major categories: Digital, and Analog.

Digital Compute-in-Memory

In standard digital PE-based accelerators, the partial sums of the matrix operations are stored in registers and accumulated with PEs. In Digital CiM, these intermediate registers are eliminated and uses digital adder trees after the digital multiply to perform the accumulation of partial sums.

Digital CiM offers the potential to improve energy efficiency MAC operation through bit serial / parallel operations as well as sophisticated control circuits to better control data movement.

Most efficiency analysis of DCIM is done by examining the intricacies of floating point operations and accounting for the energy consumption of individual operations, which is beyond the scope of this post. In general, with data accumulation trees, the floating point operations need to be synchronized, with a few optimization choices including alignment-first and multiply-first.

D-Matrix is one of the most prominent examples that leverages their in-SRAM computing techniques. Here they optimize data movement by building SRAM directly into the compute units and use block floating-point math to assign a single shared exponent to a group of numbers.

Share

One issue with DCIM is that logic is incredibly bloated compared to memory. In the example above, 70% of the area is taken up by the MAC and 11% taken up by the memory.

When compared against a standard, state-of-the-art digital processing element (PE) on the exact same technology node, we see that there appears to be a big efficiency gain. However, most of the area efficiency gain comes from custom DCIM hard macros that can be generally made much denser, but adds to physical design time.

As a result, some people amongst the analog CiM community argue that DCIM reverts to standard digital PE acceleration with little efficiency gain when compared apples-to-apples. Here we see another slide with a side by side comparison of TOPs performance of digital accelerators vs digital IMC. Claims and counterclaims like this is partly why analyzing the performance benefits of IMC is so confusing because there are a lot of nuances.

So, yes, in theory, DCIM can be potentially compute dense if you hire an army of custom layout engineers to route the wires, but most companies don’t do that due to the time-to-market implications.

So in short, digital CiM offers marginal efficiency advantage for the effort involved, but it hasn’t stopped startups such as D-matrix from trying.

The Building Blocks of Analog Compute in Memory

SRAM Array Skeleton

Most work done in Analog CIM involves modifying a conventional SRAM array with either modulated inputs or alternative devices. In an conventional SRAM array, SRAM unit cells are arranged in rows of wordlines and columns of bit lines.

When SRAM arrays are modified for Analog CiM, peripheral circuits are modified to handle parallel analog operations. These include the following components:

• Input

  • Decoder - asserts multiple wordlines simultaneously with modulated input values. You can’t normally do this in SRAM memories, but this is essential for parallel matrix operations.

• Output

  • Bit Line Processing - Performs SIMD analog processing for each column (such as a S/H or integrator)

  • Cross bitline processor - Connects processing elements across different columns. This block performs analog averaging to clean up array non-idealities and common mode noise to enhance SNR

  • Residual digital unit - The ADC that converts the result to digital domain

Within this SRAM array the compute operation itself requires a few modifications.

Input Encoding

The input can be either an analog voltage or pulse width based, with the duty cycle representing the effective analog value to be either integrated or accumulated on.

Weight Encoding.

There are two primary classifications of devices used to encode the weights that this input computes on.

• Charge Domain

  • Charge Domains devices perform parallel MAC operations by injecting charge across a target capacitance through PWM, PAM, or a direct analog voltage. The total accumulated charge creates a voltage shift that maps to the inner product output.

  

  • Examples of charge domain devices include:

    • Conventional SRAM - uses either pulse amplitude or width modulation to charge the bitline voltage and reading the accumulated charged at fixed intervals

      • This bit line must be pre-charged to a high voltage before every compute cycle

      • Then, a wordline pulse allows the bitline to discharge, lowering the voltage on the line.

      • Note that additional switches are typically added to make the 6T SRAM either 8T or 10T

    • DRAM - relies on charge sharing between the internal cell capacitance and the bitline capacitance

      • Like with conventional DRAM, this can be potentially compute-dense, but required some modification to read out values non-destructively

    • Flash - Accumulating charge along a single bit line through non-volatile cells, effectively computing the entire parallel MAC operation natively.

      • This is perhaps the most “brain like” structure, but is very sensitive to device variation and array size as line parasitics matter more.

      • Note that flash looks like a current domain device below, but can be made charge domain by turning on the transistor with a highly precise, strict time window with one of the inputs I described.

• Current Domain

  

• Also know as resistance based memory, this memory exploits the resistance characteristics of non volatile memory to act as analog variable resistor.

• The resistance are programmed using sophisticated methods well beyond the scope of this post.

• When a voltage is applied to these lines, a current accumulates at the output, which is then processed and converted to the digital domain.

• Embedded non volatile memory can potentially be a high density option, but the memory devices themselves are subject to inconsistent resistance across devices that affect SNR, as well as typical memory nonidealities such as write endurance and speed.

  

• Examples of such devices include:

  

  • RRAM - consists of a a top and bottom electrode with a thin oxide layer between. This devices exploits oxygen vacancies to create conductive filaments.

    

  • Phase Change Memories (PCM) - One of the most extensively researched technologies, PCMs stores data by passing precise electrical current pulses through a chalcogenide material (almost like a flexible glass) to alter its atomic structure between a Crystalline Phase and Amorphous Phase.

    

  • MRAM - Magnetic RAM consists of two ferromagnetic layers with a fixed and a free layer. The orientation between the two layers affects the resistance.

Scaling Challenges with Analog CiM

Read more

Demis Hassabis is perhaps one of the most important influences in Modern AI that isn’t a household name.

Modern AI is not just a story of better models. It was shaped by the interaction between scientific ambition, commercial urgency, safety concerns, and the compute power available to model it.

Every AI enthusiast should read “The Infinity Machine” to better understand the development of modern AI. It tells the story of Demis Hassabis, CEO of Google DeepMind, from his origins, to his founding of DeepMind and its acquisition by Google.

I’ll highlight a brief history of AI through the formative years of Demis Hassabis, and cover the key technical developments and organizational challenges along the path to modern AI, particularly around AI safety.

I’ll also dive into the key papers addressed in the book that influence modern AI today such as deep reinforcement learning, transformer architecture, and chain-of-thought prompting that approximate human intuition.

Demis Hassabis

From his early days Demis Hassabis is a modern example of a modern day Polymath, with broad-ranging interests from sci-fi, philosophy, computer science, and neuroscience.

His entire career has been dedicated to developing artificial general intelligence (AGI), a hypothetical type of AI that matches or exceeds human cognitive abilities across a wide range of tasks. His dedication to this mission has him referred to as an “authentic entrepreneur” by VCs because he is genuinely dedicated to a cause, as opposed to entrepreneurs fishing for ideas to strike it rich.

We’ll see through his formative years in video game development about how he distinguishes between early machine vs human intelligence:

• Early machine intelligence uses deductive, first order reasoning to reason through problems

• Human intelligence generally uses induction or high level pattern recognition to identify shapes, sounds, and reason through sentences

Formative Years

Demis had a fairly modest upbringing in North London. When he was young he was highly influenced by classic Sci-Fi: “Enders Game”, “Foundation”, “Godel, Escher, Bach” and Ian Banks “Culture” series were some of his early influences. He was a chess genius and video game enthusiast in his early days, frequently ruminating on what algorithms are needed to program a machine to play chess.

He won admission at University of Cambridge to study CS, but he was too young at the age of 16. During this time gap, he worked on video game development at Bullfrog with Peter Molyneux to develop video games, including Theme Park.

After he graduated, he founded Elixir Studios in 1998, a London-based video games developer, and work on developing video games that focused on complex decisions on the user’s part, including Republic: The Revolution and Evil Genius.

Elixir Studios ultimately failed in April 2005, at which point he decided to pursue his PhD in neuroscience at University College London (UCL) from 2005 to 2009. His studies focused on the field of imagination, memory, and amnesia.

After he obtained his PhD, Demis Hassabis founded DeepMind Technologies in London in 2010, a ML AI startup with Shane Legg and Mustafa Suleyman. The goal is to combine insights from neuroscience with ML and computing hardware to create new algorithms to advance toward AGI. His company was successful at training a Deep Q-Network to play Atari games in 2013. [1]

In 2014, Google purchased DeepMind for $400 million where it remained a separate, siloed entity from Google. Since being purchased, it racked up a list of accomplishments from 2014 to 2020:

• It created a clinical alert system, Streams, for acute kidney injury in the UK. While well intended, it was controversial because of concerns over patient data governance.

• It created AlphaGo, a program that played Go. It defeated world champion Lee Sedol in 2016 and several other high level players with moves that humans had very little intuition for.

• It created AlphaFold and AlphaFold 2 to take on the CASP protein-structure prediction competitions. AlphaFold 2 achieved a breakthrough at CASP14 in 2020. In 2021, DeepMind and EMBL-EBI launched the AlphaFold Protein Structure Database, and by 2022 it included predictions for more than 200 million protein structures. Hassabis later won a Nobel Prize in Chemistry for his contribution, along with John Jumper and David Baker.

AI Safety: Commercial time-to-market vs Research Discovery

While DeepMind racked up an impressive list of accomplishments in its early years, it entered a ferocious competition to develop LLMs against OpenAI, led by Sam Altman.

Much of the story that follows revolves around two concepts:

• AI safety

• Research vs Commercial Environments

I’ll cover the groundwork for each concept, then dive into the high level narrative of the competition between OpenAI and DeepMind.

AI Safety

Long before AI safety became a technical field, Sci-Fi gave the public a meaningful way to think about machine intelligence and control. “I, Robot” by Isaac Asimov in 1950, proposed the “the Three Laws of Robotics” that dictate that “robots must never harm humans, must obey orders unless they conflict with the first law, and must protect their own existence”. Additionally, classic Sci-Fi highlights the potential dangers of misuse of AI such as Terminator, Blade Runner, and Cyberpunk genres.

These examples influenced the development of AI where humans ensure that AI is safe and aligned with human intentions. Humans need to ensure that AI doesn’t deceive users by providing harmful / misleading information.

One way to test for AI safety is through adversarial testing, or “red teaming”, where human testers act as adversaries to intentionally break, deceive, or trigger harmful outputs from AI systems. Red teams simulate real-world attacks to identify safety, security, and ethical flaws, such as prompt injections, bias, or data leakage, before malicious actors exploit them.

As AI models scaled in complexity, they became harder to interpret. Inputs could be fed in during training and outputs could be evaluated during inference, but the internal representations became increasingly difficult to explain. AI can produce human-aligned outputs 99.9% of the time, but identifying the 0.1% of cases where it acts against human intentions can be difficult.

Red teaming alone does not guarantee total safety, but is one tool in a broad range of strategies to ensure that AI is aligned with human intentions.

Research vs Commercial incentives

In large tech corporations such as Google, there are two main categories of groups that contribute to business revenues:

• Revenue-generating core business - Engineers who create products to take to market or maintain/update existing infrastructure. Here, work is considered to be low risk and execution on set priorities is important.

• Research and development - Researchers who work on high-risk, long-term, and ambitious ventures. In R&D environments, ideas are incubated to either be published or folded into products to take to market.

Organizations consist of groups that range from commercial readiness vs research focus that help sustain the revenue of the core business as well as bet on “moonshot” ideas that can generate revenue in the future.

In 2015, Google restructured under the umbrella holding company “Alphabet”, separating its core businesses from longer-term moonshot bets such as autonomous driving, fiber, and life sciences. This move positioned Google to overcome the “innovators dilemma” by incubating capabilities to enhance its core business or act as additional revenue generating streams in the future should outside disrupters arrive and threaten it.

DeepMind, HQed in London, was acquired by Google because its capabilities were potentially useful to the core business. However, it was heavily siloed from the main HQ in California at first. This siloing gave the researchers to environment to produce many of the key innovations but caused it to hesitate entering the market when commercial pressure arose.

OpenAI, on the other hand, is HQed in San Francisco, California, with a highly influential leader, Sam Altman, and the entire network and talent pool of the Valley nearby.

OpenAI - Fast to Market, with Safety Governance Under Pressure

OpenAI was founded in 2015 by Sam Altman, Elon Musk, Greg Brockman, and others as a non-profit with the primary goal of ensuring that artificial general intelligence (AGI) benefits all of humanity.

In 2020, OpenAI published “Language Models are Few-Shot Learners,” introducing GPT-3, a 175-billion-parameter autoregressive language model, building off of the “Attention is all you Need” paper published in 2017. This was one of the first truly major, large-scale Large Language Models (LLMs).

However, as models were becoming more complex, AI safety became a concern. Former OpenAI employees including Dario and Daniela Amodei defected and founded Anthropic in 2021 to prioritize AI safety, alignment, and research.

To improve AI safety, OpenAI released a paper in 2022, “Training language models to follow instructions with human feedback” which used Reinforcement Learning with Human Feedback (RLHF). It demonstrated that humans are able to “teach” AI outputs that aligns with humans intention so that the AI can be trained more effectively. This paper demonstrated improvements in how humans can “guide” AI to produce output that aligns with its intentions.

On November 30, 2022, ChatGPT was officially released by OpenAI. It was intended to be introduced as a “research preview” and fast tracked to not let Anthropic get ahead. It quickly went viral, reaching one million users within a few days of its launch, the fastest growing consumer application ever at the time (Threads would later surpass this).

About a year later, the board was concerned about Sam Altman’s handling of AI safety and other allegations. The board abruptly fired Altman on November 17, 2023, citing concerns about his candor. But the move triggered a rapid employee and investor backlash: nearly all OpenAI employees threatened to leave unless he was reinstated, and Altman returned as CEO on November 22, 2023 with a restructured board.

Google DeepMind - Prioritizing AI safety, but Caught in the Innovators Dilemma

Google saw commercial potential in AI when it first acquired DeepMind because it recognized the strategic importance of AI in its core business.

At first, DeepMind’s culture leaned toward research depth, scientific credibility, and safety caution. It saw too much existential risk to let commercial priorities drive AI’s deployment, which made it slower to convert frontier models into consumer products. Demis wanted a path to spinout to a semi-independent Alphabet company, but ultimately didn’t due to the legal effort involved.

When OpenAI released GPT-3 in 2020, DeepMind did their own work on LLMs and published a few key papers, including Flamingo, Gato, and Chinchilla in 2022. In Sept 2022, it worked on Sparrow, DeepMind’s version of ChatGPT, which used RLHF. However, it was ultimately not released to the general public.

When OpenAI released ChatGPT, Google was briefly trapped by the innovators dilemma where it would put its brand at risk if it released an AI model too early and it hallucinated. During this time, a lot of DeepMind researchers were being poached by OpenAI who were frustrated by lack of progress by Google releasing a product.

On April 20, 2023, Google Brain (aimed at applied machine learning for Google products like TensorFlow) and DeepMind merged to form a single unit, Google DeepMind, to become one comprehensive unit.

Google DeepMind ultimately launched Gemini in Dec 2023 and upgraded versions that followed. While it is widely used nowadays and offers many benefits over ChatGPT (such as much larger context window for large documents), it ultimately suffered from the first mover advantage OpenAI had.

Key Technical Developments in modern AI

With that institutional backdrop in place, the technical story becomes easier to understand.

Next I’ll dive deeper into the key technical developments along the way to modern AI referenced in the book that approximate human intuitions. This is by no means a complete list since this book does not cover the more recent “reasoning era” Agentic AI.

Deep Reinforcement Learning (2013-2015)

Reinforcement learning is a fairly intuitive, but math-heavy, ML method where an agent interacts with the world and periodically receives rewards / reinforcements that reflect how well its doing. Reinforcement learning is especially popular in environments where there are small number of training examples compared to the state space the agent exists in (like chess) that supervised learning would struggle in.

Each agent exists in an environment that resembles a Markov decision process, such as a maze, with other states that the agent can transition to. The agent has a policy that determines whether it will explore the environment or exploit an action with known outcomes. Each agent goes through several trials of decisions, determines which actions likely led to the desired outcomes, and updates its policies with what it learned.

One popular method of reinforcement learning is Q-learning (or Quality) where the agent estimates the max expected future reward for each action in each state. While Q learning is fairly primitive, it forms the basis for more sophisticated algorithms.

In a 2013 paper, “Playing Atari with Deep Reinforcement Learning”, Hassabis attempted to combine deep and reinforcement learning to create a single NN agent that is able to learn to play as many of the games as possible without knowing the rules in advance [1]. However, combining deep learning and reinforcement learning presented some challenges:

• Deep learning applications require large amounts of hand-labelled training data (at the time). It also assumes samples to be independent.

• RL algorithms, on the other hand, learn from a scalar reward signal that is frequently sparse, noisy and delayed. It also frequently encounters highly correlated states.

DeepMind used Deep Q-Networks (DQN) to play Atari games. They use a method of “experience replay” where the agent’s experiences are stored at each time step, then are sampled uniformly when performing updates. Randomizing samples helps break up correlations and allows the DNN to learn more effectively.

They demonstrate this approach on seven Atari Games (Beam Rider, Breakout, Enduro, Pong, Q*bert, Seaquest, and Space Invaders) and showed that it generalizes to a variety of games without specific information about each game.

Transformer (2017)

The Transformer architecture was introduced in the June 2017 seminal paper "Attention Is All You Need" by researchers from Google Brain and Google Research. It is widely considered one of the most important, consequential, and influential AI papers of the 21st century because it influences the architecture of Large Language Models we use today.

Prior to the transformer model, the standard approach for language training were Recurrent Neural Networks (RNNs) and sequence-to-sequence models with Long Short-Term Memory (LSTM). A RNN/LSTM processes words sequentially. This is useful in simple, condensed sentences structures such as subject-verb-object pairs. However, it suffers from modern sentences especially when meaning between words (both local and global context) are far away from each other in time.

The proposed “Attention” mechanism allows each word context into what other related words are doing. To get an idea for how the Attention mechanism works, lets consider real sentences.

Real sentences can have multiple parts of speech, with nouns (things), adjectives (modifiers to nouns), verbs (actions) and other parts of speech. Nouns can take on multiple meanings depending on the “context” is it in. Take the word “apple” in the following sentences:

• Apple will release an iPhone next year.

• I ate an apple and it was delicious.

In each sentence, its clear to us what “Apple” refers to: a company in the first sentence because of the world “IPhone” that provided context, and a fruit in the second because of the word, “delicious”. We humans used context clues to determine what “apple” means based on supporting words around it.

This is roughly what “Attention” enables mathematically in the Transformer architecture.

In the original Transformer, words are processed in parallel through attention layers rather than one step at a time as in RNNs. These words are broken down into several units called “tokens”.

Then, each token is converted to a high dimensional vector called an embedding that captures the meaning of each token. Think of this vector like the entries of a thesaurus where each tokens’ values corresponds to how strong it correlates to each entry in the thesaurus. A word like apple can map well on the the position that represents companies (such as Tesla, Exxon) and another position that specific fruits (oranges, pears). These embedding values are found during training on billions of words since each embedding has multiple “dimensions” it sits on.

Then, through matrix operations, every word in a sentence calculates how much “attention” it should pay to every other word in that same sentence. This allows it to figure out relationship between words based on context clues.

To accomplish this, each embedding vector is decomposed into three vectors : Query (Q), Key (K), and Value (V). The query vector for each word is then multiplied by the key vectors for all the other words to come up with an “Attention” matrix that represents how related words are to each other. This attention value is given by the following formula:

To predict the next word in the sequence, it uses a decoder to look at words in context to be able to generate the sequence of words that map well to the original meaning of the sentence.

AlphaFold 2 also showed how attention-like architectures could move beyond language: it used attention over sequence and pair representations to reason about relationships between amino acids, including residues far apart in the 1D sequence but close in 3D structure.

The transformer is significant because it is scalable along memory and compute:

• The context window represents the “short term memory” as the maximum number of tokens (words) the model can “see” in one single look.

• The main attention calculation is a matrix multiplication, so modern GPUs can parallelize these matrix multiplications to higher embedding dimensions.

Sentiment Neuron (2017)

The paper “Learning to Generate Reviews and Discovering Sentiment” was released by OpenAI on April 5, 2017. This paper was consequential because it became an early and memorable example of a neural network learning an internal representation that researchers did not explicitly program.

Originally, the researchers at OpenAI trained a specific type of RNN called an mLSTM (Multiplicative LSTM) on 82 million Amazon reviews with unsupervised language modeling. They found that one single neuron had essentially "volunteered" to track sentiment of whether parts of the review were positive, neutral, or negative.

This paper is considered highly influential to AI safety because it gave insight to researchers that certain neurons can be “inspected” to find big picture behavior and debug potentially dangerous behavior.

Few-shot and in-context learning in GPT-3 (2020)

Since the transformer paper was published, the standard approach for LLMs was to pretrain the models on a ton of data and fine tune the model for each specific task, such as reading comprehension and answering questions.

In Apr 2020, when OpenAI released GPT-3, it published a paper, “Language Models are Few-Shot Learners”. This paper presents “Few-shot learning” where a model learns to perform new tasks or recognize patterns simply by providing examples in the prompt, a capability known as in-context learning. It demonstrated that a generalized LLM model could be taught to solve new tasks using only a very small amount of training data. Other settings such as zero-shot and one-shot were shown with some accuracy loss compared to few-shot at the time.

Together, the Transformer architecture and GPT-3 helped make scaling a central paradigm in AI: as model size, data, and compute increased, capabilities improved in increasingly predictable ways.

Reinforcement Learning with Human Feedback (2022)

When LLMs began scaling in complexity, they sometimes generated outputs that were not aligned with their users, such as toxic or biased statements. The paper “Training language models to follow instructions with human feedback” used human-guided Reinforcement learning to “teach” the model what outputs are acceptable to humans.

OpenAI hired 40 contractors and trained a model, InstructGPT, with human feedback. They evaluated its responses vs GPT-3 and found that labelers significantly prefer InstructGPT outputs over outputs from GPT-3.

This procedure showed improvements in truthfulness and reducing toxicity. However, RLHF is hard to scale because human-provided feedback on AI is harder as AI generated more sophisticated outputs.

Chain-of-thought Prompting (2022)

The concept of Chain-of-thought prompting was introduced in the 2022 paper, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. CoT was a major demonstration that providing intermediate reasoning examples to pretrained LLMs can improve performance on arithmetic, commonsense, and symbolic reasoning tasks in sufficiently large models.

CoT also made model outputs more interpretable at the surface level, because humans could inspect the generated intermediate steps, though those steps should not be treated as a guaranteed faithful transcript of the model’s internal computation.

This approach is similar to how humans learn by analogy/example. It is typical for humans to decompose the problem into intermediate steps and solve each before giving the final answer. Most of us were taught to “show your work” when solving complex problems and were rewarded partial credit even though our answer might be wrong, since the line of reasoning can be valid up to a certain point.

Anthropic - Interpretability (2021-)

Anthropic was founded in 2021 by former OpenAI executives Dario and Daniela Amodei and other colleagues. It was established as a public benefit corporation (PBC) dedicated to AI safety, focusing on creating reliable, interpretable, and safe AI systems (like Claude) following disagreements over the rapid commercialization and safety approach at OpenAI.

Anthropic’s safety research pushed hard on mechanistic interpretability: trying to identify features, circuits, and internal representations that help explain why a model produced a given output.

This matters because interpretability is one possible path toward moving AI safety beyond trial-and-error behavior testing. Red teaming can reveal when a model fails from the outside; interpretability tries to understand why the model behaves that way from the inside.

Conclusion

Modern AI did not emerge from a single breakthrough, company, or personality. It came from the convergence of several technical ideas: deep reinforcement learning, transformers, scaling laws, RLHF, chain-of-thought prompting, and interpretability — with a set of organizational tensions that shaped how quickly those ideas moved from research labs into the hands of the public.

Demis Hassabis and Sam Altman represent two different archetypes in this story.

• Hassabis reflects the research-driven, long-horizon pursuit of intelligence as a scientific problem.

• Altman reflects the commercial, political, and go-to-market instincts required to turn frontier research into a mass-market product.

Both approaches have strengths and failure modes, and that tension is the real story of modern AI.

The next era of AI will likely be shaped not only by better models, but by how well institutions can balance those forces while building systems that are powerful, useful, and aligned with human intent.

If you want to continue expanding your knowledge on the HW side of AI, check out my other post:

References

[1] Mnih, V, et al. “Playing Atari with Deep Reinforcement Learning” https://arxiv.org/pdf/1312.5602

[2] Sutskever, I, Vinyals, O, Le, Q. “Sequence to Sequence Learning with Neural Networks” https://arxiv.org/pdf/1409.3215

[3] Vaswani, A, et al. “Attention Is All You Need” https://arxiv.org/pdf/1706.03762

[4] Radford, A, Jozefowicz, R, Sutskever, I. “Learning to Generate Reviews and Discovering Sentiment” https://arxiv.org/pdf/1704.01444

[5] Brown, T, et al. “Language Models are Few-Shot Learners” https://arxiv.org/pdf/2005.14165

[6] Ouyang, L, et al. “Training language models to follow instructions with human feedback” https://arxiv.org/pdf/2203.02155

[7] Wei, J, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” https://arxiv.org/pdf/2201.11903

This primer focuses on AI accelerators: hardware designed to run deep learning efficiently. We’ll build intuition from the ground up (MACs → DNN/CNN structure → memory), explain the Von Neumann bottleneck, introduce the key performance metrics, and then ground everything in a classic example: Eyeriss.

This post synthesizes ideas primarily from “Efficient Processing of Deep Neural Networks” and “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks.” Figures are adapted for educational commentary with attribution in captions; please see the original papers for full detail.

The compute: The Multiply- accumulate

The Multiply - accumulation operation is the heart of most deep learning applications.

In simplest form,

then applies a nonlinearity (often ReLU). At the output, sigmoid (binary) or softmax (multi-class) can convert logits into probabilities.

There are few intuitions I use the thinking about Neural networks:

• MAC operations are analogous to biological neurons: many inputs influence an output, and the weights determine what patterns the unit responds to. (It’s an analogy—not a literal model of the brain—but it’s a useful intuition.)

• A neuron (dot product + nonlinearity) behaves like a pattern detector where the output is high in response to a input pattern that correlates with the combination of weights.

This multiple accumulate operation can be found across many range of deep learning applications.

Feedforward Neural Networks

In feedforward neural networks, many of these neurons are assembled in parallel in a “pipelined” fashion. The inputs all connect to each “neuron”, each with its own unique set of weights to detect specific patterns, and these neurons then pass that information to later layers or the output. Additional layers can be added to be able to fine tune the pattern detection.

Adding “hidden” layers between the input and output allows the network to build up from simple features (edges/textures) to more complex features (higher-level shapes and objects).

There are two major operations performed on neural networks:

• Training = learning the weights by passing images through, computing the error, and tuning the weights through “backpropagation”

• Inference = running a trained model to produce outputs

Inference is commonly run on edge devices locally; training is usually done on large GPU/accelerator systems. Both training and inference can be run on GPUs or specialized accelerators optimized for these workloads.

Convolutional Neural networks

Convolutional neural networks (CNNs) are widely used for image recognition and classification. Instead of connecting every pixel to every neuron (which would explode parameter count and compute), CNNs use small filters that slide across the input in strides.

Each layer of the Convolutional Neural Networks operates in the following way:

• Small filters represented as matrices can be customized to detect specific features, but the simplest case are “edge detectors” that detects vertical lines, horizontal lines, and diagonal dimensions.

• These filters are “convolved” with each each input feature map (ifmaps), which is image pixel that is most commonly represented by a tensor of RGB values.

• The output feature map (ofmaps) is calculated by sliding the filter across the input and accumulating partial sums (psums) along the way.

• This feature map is condensed through normalization and “pooling” operations.

Each output feature map corresponds to a filter where the particular pattern is present (such as horizontal/vertical/diagonal line segments). As layers get deeper, higher order constructs are formed from simpler ones and spatial resolution shrinks. When dimensions are small enough, the CNN is connected to a fully connected layer that generates an output.

There are computational observations in both DNNs and CNNs as it relates to hardware:

• Filter weights can be reused in computation without needing to retrieve it from memory every time.

• The tensor shapes and data dependencies are known ahead of time, so you can pre-plan tiling and data movement to minimize expensive memory traffic.

These two points will become important later when we discuss Eyeriss.

The Von Neumann bottleneck

You have the best compute in the world, but it doesn’t matter if memory can’t keep up.

Most computing nowadays uses a Von Neumann architecture, where the compute and memory are split into two entities. Data stored in memory is retrieved and stored locally into CPU memory to perform operations on. When the result is computed, it is written back to the memory.

This architecture has worked as compute and memory are scaled. However, there are challenges being faced today:

• Compute throughput has scaled faster than memory bandwidth/latency improvements

• There is a limit transfer rate and a high energy cost per memory retrieval

This results in a bottleneck, where compute is limited by bandwidth + high energy per byte moved between memory and compute.

The question is, why don’t you put compute and memory on the same chip? You could, but its often not cost effective. Fab process for compute and memory optimize different things:

• Logic (CPU/GPUs) are optimized for speed of transistor switching and ease of routing interconnects through stacking metal layers

• Memory requires high density storage and capacity

One method classical computer architecture employs to mitigate this issue is arranging memory in a hierarchy through caches. Here, data is retrieved from the main memory only as “needed”. This exploits a phenomenon called “spatial locality” where you are likely to retrieve data close to the one you retrieved. As a result, you can write a whole data “chunk” of memory to a cache to increases the chances that you “hit” data in the cache when going to retrieve it.

AI Accelerators exploit both spatial locality and data reuse. In order to take advantage of both, we need to understand two mature memory technologies used across the hierarchy: DRAM, and SRAM.

The Memory: DRAM and SRAM

In a naïve implementation, each MAC operation requires three memory reads and one memory write:

• Read: weight

• Read: activation

• Read: accumulated partial sum

• Write: updated partial sum

To store data into memory, there are two mature memory technologies available:

1. DRAM - Dynamic Random access memory

   

   DRAM consists of capacitors that store charge and are read / written to word lines through a transistor. These memory cells are are typically arrayed in a 2D grid of cells with vertical row lines and horizontal bit lines. Memory cells are places at the intersection of word lines and bit lines where each ones is accessed by a combination of specific word line and bit line.

   Pros: DRAM is cheap and dense.

   Cons: DRAM is volatile and requires periodic refresh because capacitor charge leaks. It also requires constant electrical power to hold data.

2. SRAM - Static Random access memory

   

   SRAM consists of cross coupled inverters in a latch (or positive feedback) configuration. The most common configuration is a 6 transistor cross coupled inverters to hold data. To read and store data, bitlines are charged and the wordline (WL) is activated depending on the values being read/written from/to the cell. The bitline needs to be strong (i.e. generate a lot of current) during write operations to break the feedback loop of the latches in order to flip the state of the memory cell. As with DRAM, these cells are arranged in a grid with these cells at the intersections.

   Pros SRAM is faster and lower energy per access than DRAM. SRAM can also be easier to integrate on compute processes.

   Cons: SRAM is much less dense and more expensive per bit.

Cost of memory: energy and bandwidth

There are two issues with DRAM:

• The energy cost of DRAM is 1-2 orders of magnitude more expensive to access than on chip memory [1]. There are several factors that contribute to this, one of the largest being constant refresh cycles needed when non-ideal capacitors discharge through leakage paths.

• The data bandwidth of DRAM is finite. In modern computer designs, DRAM is separate from the compute chips (CPU, GPU) and connected via busses with finite latency and throughput capabilities. In addition, DRAM itself has high latency due to steps of row activation, command delays, and precharging.

Due to the high energy cost, it is not desirable to read / write from DRAM continuously. Its better to read/write a “chunk” of information at a time and store the data locally on the chip to be accessed at memory technologies with lower energy cost.

There are a few trends that have been going on improve performance without significant changes to the Von-Neumann paradigm:

• High bandwidth memory (HBM) - stacked DRAM integrated closely with the compute package to increase bandwidth.

• Optimizing memory usage on chip into a memory “hierarchy”. When data is know in advance, it is possible to structure the data in a way that optimizes the memory cost and schedule data as such.

• “Compute in/near memory” which uses more efficient memory technologies designed to handle this type of computation, or architectures design to do the computation by storing weights in conductance of resistors. There are many intriguing niche approaches here [2] but are beyond the scope of this post.

Performance metrics for DNN Models

“Performance” in a deep learning system consists of how performance is defined on both the model side and the Hardware side. Both are important and closely intertwined.

Metrics for DNN Models include:

• Accuracy and robustness - How accurate is the model on common “benchmark” datasets

• Network architecture - The number of layers, filters, and sizes

• Number of weights - Affects storage requirements

• Number of MAC operations

Often times, performance is compared in terms of normalized energy per MAC to give an apples to apples comparison of performance across different models.

Performance metrics for AI Accelerators

Hardware optimized for AI computation has its own set of performance metrics that that are optimized depending on the application:

• Energy and Power is important for most applications, including cloud data centers with stringent power ceilings, as well as portable devices with limited battery

• Low latency is important in real time interactive applications

• High throughput is important for big data analytics is important when action needs to be taken based on the data

• Hardware cost is important for most applications, when tightly integrating an AI chip on a consumer device to run inference, as well as integrating into a blade on a rack in a data center

Bridging Compute and Memory: Eyeriss [3]

Eyeriss is a classic AI accelerator that optimizes for the deterministic nature of AI calculations. If you know the size and shapes of the filters, ifmaps, etc. you can optimize your flow to a given hardware set based on the size of the array. By doing so, you minimize the energy cost of moving data around.

Eyeriss has a few key building blocks:

• Processing Engine (PE)

  

  Each processing engine consists of a 16b two stage pipelined multiplier and adder, along with scratch pad memory to temporarily store the weights.

• Data parallelism through Network on chip (NoC)

  • There are 168 processing elements arranged in a Spatial architecture of 12x14.

  • Each PE can communicate with neighboring PEs or the GLB through an NoC

  • The global input network handles the process of distributing the computations to each of the PEs though busses

  • Processing in each PE can be run independently from each other with data movement coordinated by the NoC

• Four-level Memory Hierarchy

  The memory hierarchy is arranged into four-level hierarchy in order of decreasing energy:

  • Off Chip DRAM

  • 108kB global buffer

  • NoC reuse paths (neighbor PE communication)

  • Scratch pads local to PE (520 bytes each)

The goal of this chip is simple:

To minimize high cost DRAM and GLB access by reusing data from low cost spads and inter-PE communication as much as possible.

Eyeriss accomplishes this through a methodology called “Row stationary” that divides the matrix multiplication in a series of 1D convolution primitives that form 2D convolutional sets. :

• 1-D Convolution primitive in a PE

  

  • A 1D convolution primitive can be best thought of as a “sliding window” approach because it ‘fixes” the filter weights in memory while the ifmaps are “slid” across it.

  • This allows weights/activations are reused many times before going back to DRAM and/or the GLB.

  • Accumulate psums from different primitives to generate the ofmap values

• 2D Convolution PE Set

  

  • 2D convolution sets are formed from multiple 1D convolution primitives

  • In a 2D convolution set, the weight are written to spads in rows and the weights are “slid across” diagonally. It works in the following order:

    • Reuse each filter row horizontally

    • Reuse each of ifmap diagonally

    • Accumulate rows of psums vertically / features

  • The challenge is to “map” PE sets onto the PE array cleanly due to different sizes of layer shapes and PE array dimensions. Any computation primitive that exceed the size of the PE array itself or width / height can incur a large energy cost and needs further optimization from RS dataflow.

Other optimizations. These optimizations are also used and influential.

• Run length compression - compress runs of 0’s

• When distributing data from the GLB, only activate busses that correspond to the tag to save energy

A large fraction of the chip area goes to the memory, including the global buffer and on-chip storage close to the MACs.

Conclusion

In this primer I built up intuition of how AI accelerators optimize energy of AI computations:

• MACs are the core primitive in most DNN workloads

• DNN layers are repeated dot products with heavy weight/activation reuse potential

• Performance is often limited by memory movement (energy + bandwidth), not arithmetic

• Accelerators win by choosing dataflows + memory hierarchies that maximize reuse and minimize DRAM traffic

There is a lot of research done to optimizing AI accelerators including related compute near memory architecture that leverage traditional compute architectures as much as possible, as well as intriguing compute in memory approaches.

If you want to learn more about the technical developments of AI on the SW modelling side, check out my other post, based heavily on the book “Infinity Machine”:

References

[1] V. Sze, Y.-H. Chen, T.-J. Yang, and J. Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” arXiv:1703.09039v2, 2017.

[2] A. A. Khan, J. P. C. de Lima, H. Farzaneh, and J. Castrillón, “The Landscape of Compute-near-memory and Compute-in-memory: A Research and Commercial Overview,” arXiv preprint arXiv:2401.14428, 2024.

[3] Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks,” IEEE Journal of Solid-State Circuits, vol. 52, no. 1, pp. 127–138, Jan. 2017, doi: 10.1109/JSSC.2016.2616357.