A 28nm Chip Beats by 1000x: The Compute-in-Memory Black Magic That Needs No EUV
Zhicun Technology turns 28nm Flash into a compute-in-memory chip, exploiting the physical property that efficiency rises with array size, so efficiency improves as models grow — and Transformers are pure upside for it.
The argument · tap a timestamp to hear it
Before he founded the company, nobody in the industry believed in compute-in-memory
Wang Shaodi wanted to build compute-in-memory chips while doing his PhD. Before graduating in 2016 he approached four or five leading chip companies in China and the US; none had heard of the technology, and none would open a new direction for a fresh PhD graduate. Academia was a dead end too — a Peking University professor asked him point-blank where the funding would come from, and at the time very few chip grants nationwide exceeded a million yuan, not even enough to pay for a tape-out. With both paths blocked, he had no choice but to start his own company.
— Wang ShaodiThe bigger the model, the faster the read
In a conventional von Neumann architecture, matrix multiplication requires reading data out of memory cells one by one and sending it to the arithmetic unit; the larger the memory array, the slower the readout — a law of physics. Compute-in-memory instead turns each memory cell itself into a compute node in the matrix multiplication, so a single activation completes an entire matrix computation without row-by-row, column-by-column readout. That means scaling an algorithm from a few rows to ten thousand rows still takes only one activation, and effective bandwidth and compute actually grow as the model grows — the exact opposite of the von Neumann rule.
— Wang ShaodiOhm's law is the principle behind the black magic
At the core of compute-in-memory is using Ohm's law — voltage times conductance equals current — to implement multiplication directly: apply a voltage along the rows of the memory, each memory cell is itself a conductance value, and the current that naturally flows out along the columns is the product of voltage and conductance; currents from multiple rows in the same column sum together, completing addition physically. The whole matrix multiplication is finished in a single read, with no need at all for the adders and multipliers of digital chips — it is essentially using the most basic electrical relationships from middle-school physics to do AI computation.
— Wang ShaodiWhy bet on the most ordinary Flash memory
Faced with SRAM, non-volatile Flash, the RRAM much discussed in academia and other storage media, Zhicun chose the seemingly most unremarkable NOR Flash, because the compute current Flash needs for analog multiplication is the lowest of any memory, and compute-in-memory requires activating as many cells at once as possible to process as much in parallel as possible — lower current means a larger array can be turned on and effective compute is stronger; Flash is also the most mature memory process, making tape-out and debugging fastest. The price was a hard fight on precision: it took several hundred people at the company seven or eight years to get precision to the level applications required.
— Wang ShaodiTransformers are pure upside for compute-in-memory
Moving from deep learning to Transformers gave compute-in-memory two dividends: first, deep learning often reads a parameter once to compute tens of thousands of times, so memory bandwidth pressure is modest; a Transformer reads each parameter once and computes once, so the read-to-compute ratio is inherently 1:1, whereas architectures like GPUs typically run 64:1 to 128:1 and the worst can reach 1000:1 — a perfect match for compute-in-memory's 1:1 storage-to-compute ratio. Second, Transformer matrices are 1000 to 10,000 times larger than those in deep learning, and compute-in-memory finishes the whole matrix in one go, so the larger the matrix the higher the compute efficiency — again by 1000 to 10,000 times. On top of that, large models need only INT4's 4-bit precision, far below the 12 to 16 bits the deep learning era demanded, which sharply lowers the bar to deployment.
— Wang ShaodiSix tape-outs in four years, and powering up is when the nightmare begins
There are no ready-made EDA tools for compute-in-memory, and each chip that comes back from tape-out may hide dozens of problems — for instance, a design targeting 8-bit precision that in fact achieves only 3 bits. The team has to locate them one by one: spending more than a month slicing the chip open and running experiment after experiment to pin down a single problem, only to find precision barely improves after the fix, because dozens of other problems are still dragging it down — so they redesign the experiment, slice the chip again, and hunt the next problem. For an ordinary chip, powering up means the job is done; for compute-in-memory, powering up is when debugging begins. That cycle lasted four years and six major failures, and the company has now taped out close to thirty times in total.
— Wang ShaodiAnker's earbuds: from earbuds to a Guinness World Record
After ChatGPT made generative models hot in 2023, Anker's algorithm team came to Zhicun wanting to run a generative audio algorithm in earbuds for call noise reduction — but that requires a chip with strong compute, large capacity, low power, small size and controllable cost all at once, nearly impossible with a von Neumann architecture. Zhicun took on a Transformer model with several million parameters (earbud chips had previously fit generally under 500,000 parameters) using a 28nm NOR Flash compute-in-memory chip, spent nearly three years cracking it, and ultimately won the Guinness World Record for the highest voice restoration in call noise reduction, far ahead of second place — which Wang Shaodi calls a dimensional strike against conventional earbud audio chips.
— Wang ShaodiChina doesn't need EUV, but compute-in-memory needs China speed
Wang Shaodi believes compute-in-memory's long-term advantage lies in China's manufacturing and iteration speed: China's consumer electronics and new energy vehicle supply chains can compress the entire chain from materials and components to products into a very short cycle of rapid iteration, which overseas players cannot do, and compute-in-memory precisely needs the shortest-distance interaction and feedback among materials and equipment, foundries, design companies and product applications. He judges that compute-in-memory has a chance to grow from a coprocessor in niche scenarios like earbuds into a universal memory that unifies the various dedicated algorithms on the edge — just as the market for this technology did not exist at all before it was actually built.
— Wang ShaodiIn their own words · checked verbatim
We talked to four or five leading chip companies in the US and China, and none of them had ever heard of this thing.
跟美国和中国的 头部的芯片公司 我们聊过四五家 大家都没听过这个东西
Wang Shaodi4:00
It's not that one or two of us did it — it's that several hundred people at our company worked on it for seven or eight years to get it to meet the needs of applications.
不是说我们一两个人做 是我们公司几百个人做了七八年的时间 把它做到满足应用的需求
Wang Shaodi13:02
Because each memory cell can only hold 1,000 to 2,000 electrons, it's as if I use 1,000 to 2,000 electrons to achieve a linear 256-way division — that's very hard.
因为每个存储单元里面只能装下1000到2000个电子 相当我用1000到2000个电子 才去实现一个线性的256等分 这个难度是很大的
Wang Shaodi14:02
Today's GPUs might be 64:1 or 128:1, the best ones 32:1, the worst maybe 1000:1, but compute-in-memory is inherently 1:1.
现在的GPU可能是64比1 128比1 做得很好的是32比1 做得差的可能是1000比1 但是存存一体天生就是1比1
Wang Shaodi16:04
I think tape-out failures in the early days of compute-in-memory are unavoidable. We've actually had maybe thirty tape-outs from start to now.
纯研一体前期投片失败 我觉得是一个必经的事情 我们其实前前后到现在 可能有三十次投片都是有了
Wang Shaodi21:05
For many chips, powering up is considered mission accomplished. For us, powering up is the beginning of a disaster.
对于很多芯片 点亮是认为已经任务完成了 我们的点亮是灾难的开始
Wang Shaodi25:06
This chip, on Anker's earbuds, won the Guinness World Record for the highest voice restoration in call noise reduction, and it was far ahead of second place.
这个芯片在安克兰机上 获得了通话降噪 是人生还原度最高的吉尼斯世界纪录 而且远高于第二名
Wang Shaodi32:10
Figures
| Early tape-out failures | 6 major failures in 4 years, about 30 tape-outs in total | 21:05 |
| Compute-in-memory read-to-compute ratio | Inherently 1:1 (GPUs about 64:1 to 128:1, the worst reaching 1000:1) | 16:04 |
| Flash cell compute current | 10 nanoamps to 100 nanoamps (other memories: several hundred nanoamps to tens of microamps) | 12:02 |
| Time spent cracking Flash precision | Several hundred people at the company worked on it for seven or eight years | 13:02 |
| Future efficiency headroom | 3-5x per year for the next three years, and up to several thousand times when combined with manufacturing-side improvements | 20:05 |
| Company R&D team size | More than 500 R&D staff, nearly 100 PhDs | 27:07 |
| Earbud chip parameter count and process | A Transformer model of 4 million to 6 million parameters, 28nm NOR Flash process | 33:11 |
Glossary
- Von Neumann architecture
- The conventional chip architecture that separates memory from compute and requires data to be shuttled back and forth
- NOR Flash
- The non-volatile memory medium Zhicun chose, whose compute current is far lower than other memories
- NAND Flash
- Three-dimensionally stacked high-density flash, slower to read and write but with enormous capacity, suited to storing large models
- EDA
- The software toolchain for chip design and verification, currently almost a blank in compute-in-memory
- INT4
- The low-precision format commonly used for large-model inference, a far lower bar for compute-in-memory than early deep learning
- Universal Memory
- The ideal memory concept that is large in capacity, fast, and able to serve multiple scenarios at once
How to listen
Engineers who care about the underlying architecture of AI chips, semiconductor investors, and founders who want to understand the nature of the compute bottleneck
The small talk about his time at Peking University at the start can be skipped without hurting comprehension of the technical material that follows