Samsung’s PIM Bet Targets AI’s Most Expensive Habit: Moving Data

The AI chip race is no longer just about who can build the fastest processor. A GPU is only useful when data arrives fast enough to keep it busy, and that increasingly makes memory the limiting factor.

At Hot Chips 2026, Samsung Electronics made its case for processing-in-memory, or PIM. The pitch is simple: instead of moving every piece of data to the processor, do more of the work where the data already lives.

AI Has a Data-Movement Problem

Running an AI model means constantly shuttling data around. The system reads model weights from memory, stores intermediate results, and sends those results back to the processor for the next operation.

GPU performance has improved much faster than memory transfer speeds. When the GPU finishes one calculation before the next batch of data arrives, it has to wait. Engineers call this bottleneck the memory wall.

Large language models make the problem especially visible. Producing even a short response can require reading billions of parameters. Add more users, and the system moves more data, consumes more power, and generates more heat.

High-bandwidth memory, or HBM, helps by stacking memory chips vertically and creating a much wider path to the processor. It is the semiconductor equivalent of adding lanes to a congested freeway.

But a wider freeway does not eliminate the commute. Data still travels back and forth between memory and the GPU. PIM challenges that architecture more directly.

PIM Moves the Work, Not the Data

PIM adds computational functions inside memory or very close to it.

Think of a warehouse full of financial records. In a conventional system, employees carry boxes to an office whenever they need to calculate something. HBM builds a faster conveyor belt between the two rooms. PIM puts a calculator inside the warehouse.

The goal is not to turn memory into a replacement for the GPU. PIM is better suited to repetitive, data-heavy operations that would otherwise create constant traffic between chips. The GPU can then spend more time on complex work.

That division of labor could reduce latency and improve processor utilization. More importantly, it could cut the energy spent moving data across a server.

For hyperscale data centers, that may be the real prize. Operators such as AWS, Microsoft, Google, and Meta care about raw speed, but they care just as much about power, cooling, and the number of users each server can support. PIM’s strongest selling point may therefore be performance per watt, not benchmark supremacy.

The Hard Part Is Software

Elegant chip architecture does not automatically become an industry standard. PIM’s biggest obstacle is the software stack built around today’s GPUs.

AI developers rely on ecosystems such as CUDA, PyTorch, and established inference runtimes. Those tools assume that most important computation happens on a CPU, GPU, or dedicated accelerator. PIM introduces another question: which operations should run near memory, and who decides?

Compilers and runtime software must identify suitable workloads without forcing developers to rewrite entire models. Frameworks need reliable ways to schedule those operations. Debugging tools must explain what happened when performance falls short.

PIM also works better for some tasks than others. Highly repetitive operations with predictable memory access are a natural fit. Workloads with frequent branching or irregular execution may gain little and could become harder to manage.

There are physical trade-offs as well. Adding logic to a memory chip consumes die area and produces heat. Samsung must balance computational capability against memory capacity, manufacturing yield, and cost.

Customers will not buy PIM because a conference demo looks impressive. They will buy it if an existing model can serve more users at the same cost, or deliver the same output while consuming less power.

Samsung Needs Partners More Than Headlines

Samsung has an unusual advantage here. It designs memory, manufactures semiconductors, and works on logic chips. That gives the company room to rethink both where data is stored and where computation happens.

Still, Samsung cannot build a PIM market alone.

GPU vendors, server manufacturers, cloud platforms, compiler teams, and AI framework developers all need to support the architecture. If PIM only works with a narrow set of proprietary hardware and custom software, it risks becoming another clever accelerator that never escapes specialized deployments.

That is why the important evidence after Hot Chips 2026 will not be a peak-performance slide. It will be production results: which models benefited, how much software had to change, and what happened to total cost and energy consumption.

Public discussion is not yet substantial enough to offer a meaningful verdict. Detailed reactions across communities such as Hacker News, Reddit, and X remain limited, so customer validation and software support will be more useful signals than early online enthusiasm.

PIM is unlikely to replace the GPU or demolish the memory wall in one dramatic move. Its more realistic promise is a smarter division of labor, with memory handling repetitive work before data ever reaches the processor.

The deciding question is not whether computation can happen inside memory. It is whether Samsung can make that capability cheaper and easier to use than simply buying more GPUs and HBM.