A Comprehensive Overview of the AI Accelerator Landscape: From Digital to Analog Compute-in-Memory Architectures
A survey of major AI accelerator categories, and a deep dive into the scaling challenges with Analog Compute-in-memory
Editors Note: (7/22/26): I trimmed the paywall to only paywall the scaling challenges with Analog CiM. This way, you can come away with a mental framework for the components of the Analog CiM.
Compute-in-memory has emerged as a promising technology to overcome the Von-Neumann bottlenecks of conventional digital computing. Many venture-backed startups have entered this space with promising solutions to enhance compute efficiency.
In this post I’ll cover a broad overview of the AI accelerator landscape and do a deep dive into the challenges with scaling Analog Compute-in-memory. The outline of this post is as follows:
A Survey of the AI Accelerator landscape
Application Spaces - from VLP to Data Center Systems
Scaling Trends - Co-Design and Specialized Architecture
Hardware Classifications of Compute in/near memory AI Accelerators
Parallel Threads (NVIDIA GPU)
Tensor Array (Google TPU, Groq)
Manycore Accelerators (Cerebras, Tenstorrent)
Digital Compute-in-Memory
The Building Blocks of Analog Compute-in-Memory
SRAM Arrays
Input Encoding
Charge / Current Domain Devices
🔒Scaling Challenges with Analog CiM
🔒Array Nonidealities that degrade SNR
🔒Area Inefficiency
🔒Modality Conversion Overhead
🔒Example CiM startup: Encharge AI
🔒BONUS: The Challenges of Optical Compute for AI
Back in February 2026 at ISSCC in San Francisco, CA, I personally sat in the tutorial “Fundamentals of Compute-in-Memory: From Circuits to Systems” by Prof. Jaydeep P. Kulkarni at the beginning and an all-day forum “Analog for AI and AI for Analog” at the end. Several heavyweights in the analog community such as Prof. Naveen Verma and EDA community such as Prof. David Pan were present to discuss trends in Analog CiM and Automation for AMS/RFIC Signal Design, the latter of which I will eventually cover after DAC.
Personally, analog compute-in-memory is one of my favorite topics because it almost resembled the way the brain processes data. Analog CiM is perhaps the most interdisciplinary topics among semiconductor specialties because it combines knowledge from digital, mixed signal, logic fabrication, and memory domains. I think the circuits are inherently power efficient for repetitive computations.
However, the CiM space is filled with a lot of dense academic research and marketing claims of VC-backed companies. During those ISSCC sessions and while researching CiM papers, I initially found CiM difficult to digest because there aren’t universally agreed on standards to classify related architectures. The terminology is also inconsistent; some authors use compute-in memory, others use processing-in-memory, and other add subcategories of their own. Moreover, most papers tend to jump right into the block diagrams of specific applications known among academics in that space. I’m sure many of you also feel the same way trying to understand the space.
In my opinion, CiM is a space where broad system benefits can be lost in the vast chasm between marketing claims and actual technical challenges. I believe that researchers are candid about the real engineering challenges. However, claims of performance benefits tend to narrowly focus on the energy savings of ONE part of the whole computational energy budget. These claims tend to get picked up and hyped up by non-technical people without their proper nuances, which, for some companies, is a marketing strategy all in itself.
Having sat in those rooms and felt the confusion myself, I aim to bring clarity to the space. Here are some of my previous posts I’ve written that lay a foundation for this material from both the software and hardware side:
A Survey of the AI Accelerator Landscape
GenAI and Agentic AI models have exploded in the past few years with a combination of gigantic training sets and powerful models to handle a variety of AI applications. This creates a huge demand for tokens for Agentic AI that is primarily satisfied by high power, general-purpose data center GPUs in data centers.
The underlying hardware is becoming more powerful to satisfy this demand. However, the logic is scaling much faster than the memory and interconnect by approximately ~2x. This means that the compute bottleneck is not in the compute itself, but the data movement. As models become more powerful, memory transfer operations are consuming more and more energy and latency.
As a result in the explosion of compute power, co-packaged optics and vertical power delivery are emerging and widely discussed trends to satisfy these demands. I have written extensively on these topics on my website.
One of the main arguments for CiM is to minimize the power overhead in compute-memory transfers. Here is a prominent slide given in 2014 with ballpark energy consumption of various computational operations and memory accesses. Notice the order of magnitude difference between the energy used in cache and DRAM accesses, and the higher energy consumption for higher precision operations. This chart has driven a trend toward lower precision computations in the digital domain and memory efficient architectures.
Application Space - from VLP to Data Center Systems
AI accelerators are not just limited to centralized data center GPUs, but also embedded applications with many different form factors and use cases.
Here we notice a range of publicly announced accelerators that range in compute efficiency from 0.1 - 100 TOPs/W. Notice how there is a cluster primarily centered around 1 TeraOps / W for general purpose GPUs data centers. Also notice how at lower power, accelerators specialized for embedded applications tend to have higher TOPs/W because they are tailored for the specific power / performance / form factor requirements of the system they are in. In those cases, it doesn’t make sense to stream data to a centralized data center to run inference on.
General purpose GPUs are built to support both training and inference across a broad range of AI models targeted for different applications, depending on the tasks you throw at it. Two common applications include:
CNNs: Uses deep neural networks to extract lower level features and are used for image / speech classification
LLMs: Uses attention layers to calculate relations between words using Query, Key, and Value vectors and require large KV cache storage for long sequences
Scaling Trends - Co-design and Specialized Architecture
Massive scale-up of AI clusters are driving a huge need for compute efficiency. Large networks of parallel general-purpose GPUs move a lot of data around, consuming vast amounts of power and generating a lot of heat in the process.
To address this need for more compute efficiency, the industry is shifting from general purpose to hardware-software Co-Design (or co-optimization) of both DNN Models and Hardware.
This involves customizing the HW to be as efficient at computing its specific task. Some optimization examples include:
Memory: Higher bandwidth and more tightly integrated memory
On the SW Side:
More aggressive quantization / lower precision architectures
Sparse weights
HW architecture: Stationary architectures that “fix” one value in an array and stream in the other (weight, output, and input)
In general, all else being equal, there is a tradeoff between compute efficiency and how specialized the hardware is. Specialized hardware can augment general purpose GPUs but run the risk of being considered “one trick ponies” if data needs change in the future.
I personally think what most people describe as “co-design” should extend beyond the HW-SW interface into the broader high speed communication, power, and packaging subsystem. That is what my website aims to achieve.
A Taxonomy of the Compute-in-Memory Landscape
In classical computer architectures, main memory is stored on a huge chunk of off-chip DRAM (HBM) and chunks of data are read onto on-chip caches (typically SRAM) when requested. When MAC operations occur, intermediate results needs to be stored in registers until the final result is computed on.
Depending on how much you read/write data to/from memory, up to 30-70% (or even higher) of the compute energy can from memory transfers, not the compute itself. This makes efforts to optimize / trim data transfers between compute and memory such an attractive option to scale compute efficiency along with performance.
There are three major architectural classifications that classify how close the memory is to the bit cell:
Compute with large memory
Compute near memory
Compute-in-memory
We’ll go through examples within each classification.
Compute with Large Memory
Compute with large memory is the easiest solution to scale in existing paradigms with a large GPU and HBM side by side.
There is a wide range of chip architectures within this compute fabric that vary in specificity and programmability. Most AI accelerators fall in between two conventional categories:
CPUs: Very flexible compute-wise but not optimized for highly parallel inference and training computations.
FPGAs: Used off-the-shelf and easily repurposable for specific embedded applications. FPGAs are preferred for faster time-to-market over custom ASICs since they can be modified with RTL. However, for large models AI, FPGAs are not “programmable” in the SW sense because the hardware effectively changes every time the FPGA is programmed.
The bulk of compute-efficient AI accelerator products lie in-between these two categories to enable software, hardware, and algorithm co-optimization. Two such categories in between fall in the “compute with large memory” paradigm:
Parallel thread accelerators
Use a temporal architecture with traditional ALUs computed in a SIMT (Single Instruction, Multiple Threads / Data ) paradigm
Typically, one CUDA-thread processes one element of the target data set
All threads in a process share the same memory heap
Key Features
Warp Scheduler - controls a large number of ALU’s by broadcasting a single instruction to all of them to execute in lockstep
Memory Coalescing - combines multiple memory requests from parallel threads into a single, highly efficient memory transaction from the HBM
Memory coalescing is critically important for making data transfer efficient from the HBM
Advantages
Quick to program
The most flexible to handle a wide range of AI applications
Massive parallelism
Disadvantages
Low compute efficiency
Needs sophisticated control circuitry to coordinately data movement across ALUs and memory
Figure 7. A comparison of general purpose GPU systems. Source: C. Silvano et al. “A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms” https://arxiv.org/abs/2306.15552
Examples
NVIDIA H100
AMD Instinct MI250X
Intel Arc 770
Generally speaking, powerful GPUs are used for heavier, general purpose tasks that require large complex models.
Figure 8. A Spatial Architecture for dataflow processing. Source: V. Sze, Y. Chen, T. Yang, J Emer. “Efficient Processing of Deep Neural Networks: A Tutorial and Survey” https://arxiv.org/abs/1703.09039 Tensor Array
Tensors are mathematical objects that generalizes scalars, vectors, and matrices to higher dimensions
Tensor Arrays use Systolic Arrays or static spatial pipelines. A systolic array is a 2D grid of tightly integrated Processing Elements (PEs) which are a more specialized form of ALUs tailored for MAC operations.
A classic example of one is Eyeriss which I described in more detail in my previous post.
Key Features
Static scheduling - preload model parameters into accelerator before executing inference/training on them
Advantages
Data reuse - data doesn’t have to be stored in registers and can flow between ALUs that perform either Multiply or Accumulate operations
Dense Compute Density
Latency is deterministic - no complexity overhead from unpredictable scheduling
Disadvantages
Rigid, inflexible structure. The array size must be tailored to the specific application
Inefficient at handling sparsity. The sparse data still has to flow through the array
SW compiler complexity
Examples
Google TPU - showcased the first TPU in 2017
Groq LPU
Intel Gaudi
In short, Tensor arrays have the potential to be computationally efficient, but only for specific array sizes.
Compute Near Memory - Manycore
Manycore is an example of compute-near-memory. This architecture decentralizes both compute and memory arrays them along a grid. This shortens the data path from the local SRAM memory to the compute units.
Prominent examples of manycore include:
Cerebras
Perhaps the most well-known example of manycore with cores arranged on a wafer scale engine (WSE) with 850k cores
The memory is fully distributed, with the SRAM using 50% of the die area that provides full memory bandwidth to all data paths at full performance, removing the need for data reuse
Their WSE is efficient for sparse matrix operations
Tenstorrent
Tenstorrent has a very similar architecture to Cerebras, which fuses a NoC interface to move data to local SRAM to be computed on in each Tensix core.
Advantages
Memory path is short between compute and memory
Disadvantages
Additional software overhead to coordinate data movement to individual cores
Other accelerator options
There are other accelerator options available:
Neural Processing Unit (NPU): NPUs are built specifically to accelerate AI tasks directly on devices like smartphones and laptops
RISC-V processors: RISC-V is a open source ISA widely used among academia and emerging startups. Its modular nature allows easier integration capabilities with embedded applications.
Within the CiM space there are two major categories: Digital, and Analog.
Digital Compute-in-Memory
In standard digital PE-based accelerators, the partial sums of the matrix operations are stored in registers and accumulated with PEs. In Digital CiM, these intermediate registers are eliminated and uses digital adder trees after the digital multiply to perform the accumulation of partial sums.
Digital CiM offers the potential to improve energy efficiency MAC operation through bit serial / parallel operations as well as sophisticated control circuits to better control data movement.
Most efficiency analysis of DCIM is done by examining the intricacies of floating point operations and accounting for the energy consumption of individual operations, which is beyond the scope of this post. In general, with data accumulation trees, the floating point operations need to be synchronized, with a few optimization choices including alignment-first and multiply-first.
D-Matrix is one of the most prominent examples that leverages their in-SRAM computing techniques. Here they optimize data movement by building SRAM directly into the compute units and use block floating-point math to assign a single shared exponent to a group of numbers.
One issue with DCIM is that logic is incredibly bloated compared to memory. In the example above, 70% of the area is taken up by the MAC and 11% taken up by the memory.
When compared against a standard, state-of-the-art digital processing element (PE) on the exact same technology node, we see that there appears to be a big efficiency gain. However, most of the area efficiency gain comes from custom DCIM hard macros that can be generally made much denser, but adds to physical design time.
As a result, some people amongst the analog CiM community argue that DCIM reverts to standard digital PE acceleration with little efficiency gain when compared apples-to-apples. Here we see another slide with a side by side comparison of TOPs performance of digital accelerators vs digital IMC. Claims and counterclaims like this is partly why analyzing the performance benefits of IMC is so confusing because there are a lot of nuances.
So, yes, in theory, DCIM can be potentially compute dense if you hire an army of custom layout engineers to route the wires, but most companies don’t do that due to the time-to-market implications.
So in short, digital CiM offers marginal efficiency advantage for the effort involved, but it hasn’t stopped startups such as D-matrix from trying.
The Building Blocks of Analog Compute in Memory
SRAM Array Skeleton
Most work done in Analog CIM involves modifying a conventional SRAM array with either modulated inputs or alternative devices. In an conventional SRAM array, SRAM unit cells are arranged in rows of wordlines and columns of bit lines.
When SRAM arrays are modified for Analog CiM, peripheral circuits are modified to handle parallel analog operations. These include the following components:
Input
Decoder - asserts multiple wordlines simultaneously with modulated input values. You can’t normally do this in SRAM memories, but this is essential for parallel matrix operations.
Output
Bit Line Processing - Performs SIMD analog processing for each column (such as a S/H or integrator)
Cross bitline processor - Connects processing elements across different columns. This block performs analog averaging to clean up array non-idealities and common mode noise to enhance SNR
Residual digital unit - The ADC that converts the result to digital domain
Within this SRAM array the compute operation itself requires a few modifications.
Input Encoding
The input can be either an analog voltage or pulse width based, with the duty cycle representing the effective analog value to be either integrated or accumulated on.
Weight Encoding.
There are two primary classifications of devices used to encode the weights that this input computes on.
Charge Domain
Charge Domains devices perform parallel MAC operations by injecting charge across a target capacitance through PWM, PAM, or a direct analog voltage. The total accumulated charge creates a voltage shift that maps to the inner product output.
Figure 19. Charge Domain Devices. Source: J. Kulkarni. “Fundamentals of Compute-in-Memory: From Circuits to Systems” ISSCC 2026 Examples of charge domain devices include:
Conventional SRAM - uses either pulse amplitude or width modulation to charge the bitline voltage and reading the accumulated charged at fixed intervals
This bit line must be pre-charged to a high voltage before every compute cycle
Then, a wordline pulse allows the bitline to discharge, lowering the voltage on the line.
Note that additional switches are typically added to make the 6T SRAM either 8T or 10T
DRAM - relies on charge sharing between the internal cell capacitance and the bitline capacitance
Like with conventional DRAM, this can be potentially compute-dense, but required some modification to read out values non-destructively
Flash - Accumulating charge along a single bit line through non-volatile cells, effectively computing the entire parallel MAC operation natively.
This is perhaps the most “brain like” structure, but is very sensitive to device variation and array size as line parasitics matter more.
Note that flash looks like a current domain device below, but can be made charge domain by turning on the transistor with a highly precise, strict time window with one of the inputs I described.
Current Domain
Figure 20. RRAM Crossbar array. Source: R. Balasubramonian, "Making the Case for Feature-Rich Memory Systems: The March Toward Specialized Systems," in IEEE Solid-State Circuits Magazine, vol. 8, no. 2, pp. 57-65, Spring 2016, doi: 10.1109/MSSC.2016.2546198.
Also know as resistance based memory, this memory exploits the resistance characteristics of non volatile memory to act as analog variable resistor.
The resistance are programmed using sophisticated methods well beyond the scope of this post.
When a voltage is applied to these lines, a current accumulates at the output, which is then processed and converted to the digital domain.
Embedded non volatile memory can potentially be a high density option, but the memory devices themselves are subject to inconsistent resistance across devices that affect SNR, as well as typical memory nonidealities such as write endurance and speed.
Figure 21. Current Domain Devices. Source: J. Kulkarni. “Fundamentals of Compute-in-Memory: From Circuits to Systems” ISSCC 2026 Examples of such devices include:
Figure 22. An RRAM Cell. Source: Z.-R. Wang, Y.-T. Su, Y. Li, Y.-X. Zhou, T.-J. Chu, K.-C. Chang, T.-C. Chang, T.-M. Tsai, S. M. Sze, and X.-S. Miao, “Functionally complete boolean logic in 1t1r resistive random access memory,” IEEE Electron Device Letters, vol. 38, no. 2, pp. 179–182, 2016. RRAM - consists of a a top and bottom electrode with a thin oxide layer between. This devices exploits oxygen vacancies to create conductive filaments.
Figure 23. Phase Change Memory. Source: H. Tsai. “Algorithm Architecture Co-Design for Analog In-Memory Computing (AIMC)” ISSCC 2026 Phase Change Memories (PCM) - One of the most extensively researched technologies, PCMs stores data by passing precise electrical current pulses through a chalcogenide material (almost like a flexible glass) to alter its atomic structure between a Crystalline Phase and Amorphous Phase.
Figure 24. Typical MRAM Array. Source: S. Angizi, Z. He, A. S. Rakin, and D. Fan, “Cmp-pim: an energy-efficient comparator-based processing-in-memory neural network accelerator,” in Proceedings of the 55th Annual Design Automation Conference, 2018, pp. 1–6. MRAM - Magnetic RAM consists of two ferromagnetic layers with a fixed and a free layer. The orientation between the two layers affects the resistance.
