What is the GPU/AI chip landscape beyond Nvidia — AMD, custom silicon (TPUs, Trainium), and the race for inference efficiency
Who Makes the Chips That Run AI, and Why Is It So Hard to Compete With Nvidia?
Based on analysis of a 124-node, 420-edge knowledge graph mapping the competitive dynamics of AI accelerator hardware.
Two Very Different Jobs: Teaching vs. Answering
Imagine you want to train a dog. That takes weeks, a lot of repetition, consistent feedback, and a lot of energy. But once the dog is trained, getting it to sit on command takes about one second and almost no effort.
AI works the same way. Training an AI model is like teaching the dog — it’s expensive, slow, and happens mostly once. Running the model (called inference) is like asking the trained dog to sit — it happens billions of times a day and needs to be fast and cheap.
For a long time, the same chips (Nvidia’s GPUs) did both jobs. But recently, these two jobs have pulled apart so dramatically that the entire chip industry is reorganizing around the difference. This split — the graph calls it the “Training vs. Inference Hardware Bifurcation” — is the single most important structural fact in the analysis. Nearly every other dynamic in the graph flows from it.
Why Nvidia Still Dominates, Even Though Others Have Better Hardware
Here is the part that surprises most people: AMD has made chips that are, by many technical measures, as good as or better than Nvidia’s for certain tasks. Intel tried to compete. So why does Nvidia still control the market?
The answer is software, not hardware.
Think of it like a power outlet. Nvidia built a standard called CUDA — a way for programmers to talk to their chips. Over twenty years, millions of researchers and engineers wrote their code to work with CUDA. Entire companies were built on top of it. AI frameworks, research tools, cloud services — almost all of them were designed around CUDA.
AMD has chips with more memory and competitive raw performance. But when you plug AMD’s chip in, a huge fraction of existing code either doesn’t work or runs worse, because it was written for CUDA. This isn’t just inconvenient — it’s a fundamental barrier. Companies cannot afford to rewrite their entire software stack just to try a different chip.
The graph captures something interesting about how this lock-in works: Nvidia never had to prove its moat — its competitors proved it for them. The strongest evidence for CUDA’s dominance comes from what happened to Intel’s Gaudi 3 chip and AMD’s own software struggles. Their failures — documented in the graph with some of the highest edge weights in the entire analysis — are what demonstrate how real and deep the moat is. Nvidia didn’t build a wall; everyone else ran into it.
NVIDIA’s Three Fences
Nvidia isn’t just sitting on the CUDA advantage. The graph shows it has built three layers of defense, each serving a different purpose.
The first layer is CUDA itself — the software ecosystem described above.
The second layer is hardware interconnect. When you’re training a very large AI model, you need dozens or hundreds of chips to work together simultaneously. Nvidia built a proprietary connection system called NVLink that lets its chips talk to each other extraordinarily fast. Imagine a city where all the major roads were built by one company, and they only work smoothly with that company’s cars.
The third layer is the most strategic. An open-standards consortium called UALink is trying to build a public version of those roads — one that any chip company could use. Rather than fighting this directly, Nvidia created something called NVLink Fusion, which allows other chips to connect into Nvidia’s network rather than replacing it. The graph shows that Amazon’s custom AI chip, Trainium3, now depends on NVLink Fusion. A competitor’s chip has been embedded into Nvidia’s own infrastructure. The graph labels this move “embrace, extend, co-opt.”
The Memory Problem Nobody Talks About Enough
Modern AI models — especially the kind that answer questions and hold conversations — have a specific technical bottleneck called the KV Cache Memory Wall. Here is what it means in plain terms.
When an AI is answering a long question, it has to keep track of everything said so far, so it can make each new word consistent with the previous ones. That “keeping track” data lives in memory. As conversations get longer, as models get more capable, this memory requirement grows faster than chips have traditionally been able to accommodate.
This single technical constraint — the memory wall — is the most actively contested engineering problem in the graph. Seven distinct approaches are trying to solve it, from completely different angles:
- AMD built chips with unusually large memory (192GB, then more) so the problem simply fits
- Cerebras built a chip the size of a dinner plate with enormous on-chip memory, eliminating the bottleneck entirely
- Groq uses a different architecture that avoids the problem structurally
- Software tools like vLLM found clever ways to pack memory more efficiently
- Quantization techniques (reducing the precision of numbers) shrink the problem’s size
- Disaggregation splits the “question-receiving” and “answer-generating” parts of inference across different hardware
- Tenstorrent designed chips that don’t use the memory format (HBM) where the bottleneck lives
The fact that seven different approaches are targeting the same problem simultaneously tells you something important: this bottleneck is real, it’s current, and nobody has definitively solved it yet.
The Company Nobody Is Talking About
While the public conversation focuses on Nvidia vs. AMD vs. Google vs. Amazon, the graph identifies a quieter winner: Broadcom.
Broadcom is a chip design services company. They don’t build their own AI chips to sell to customers — they help other companies design their custom chips. Google’s most advanced AI chip? Broadcom helped design it. OpenAI’s forthcoming chip? Broadcom again. Amazon’s custom silicon program benefits Broadcom. The new open networking standard that is supposed to compete with Nvidia? Also benefits Broadcom.
The analogy is a gold rush town: while prospectors fight over claims, Broadcom sells the picks and shovels to everyone. Its revenue position strengthens regardless of which hyperscaler’s silicon ultimately wins, because it is embedded in all of them.
DeepSeek: An Unexpected Plot Twist
A Chinese AI lab called DeepSeek published models that were unusually efficient — they could do impressive things while consuming far less memory than comparable models. This seemed like it might hurt AMD, whose competitive advantage is having more memory than anyone else.
But the graph records something counterintuitive: DeepSeek’s efficiency-focused models actually run better on AMD’s high-memory chips in certain configurations. The architecture DeepSeek uses (called Mixture of Experts, or MoE) activates only parts of itself at a time, and AMD’s large-memory chips happen to handle this gracefully.
The graph also notes the opposite possibility: as quantization techniques improve (making models smaller and less memory-hungry), AMD’s memory advantage could be eroded faster than AMD can widen it by adding newer, larger memory. Both stories are in the graph, pointing in opposite directions, and neither is resolved. This is one of several places where the analysis records genuine uncertainty rather than a clean answer.
The Loop That Feeds Itself
One of the cleaner structural findings in the graph is a self-reinforcing cycle around inference economics:
Lower cost per inference → more inference gets used → total inference workload grows → more investment in inference-specific hardware → lower cost per inference.
This is called the Jevons Paradox — when something gets cheaper, people use more of it, and total consumption goes up even though efficiency improved. The graph shows this loop has no brake mechanism. Every edge in the cycle amplifies the next step. This means the inference hardware market is not just growing — it is structurally likely to grow faster than efficiency improvements alone would suggest.
Some Findings That Don’t Match Intuition
A few things in the graph are worth flagging specifically because they run counter to what you might expect:
Custom silicon depends on Nvidia’s moat. The hyperscalers (Google, Amazon, Microsoft, Meta) building their own AI chips are doing so specifically because Nvidia has locked up the training market so effectively. The inference market is where custom silicon can compete. If Nvidia’s training lock-in weakens, the economic case for custom silicon weakens too. The challenger’s strategy requires the incumbent’s strength.
Open-source inference software helps AMD. vLLM is a widely-used open-source tool for running AI models. By abstracting away the hardware details, it reduces how much AMD’s software gap actually matters in practice. If the software layer between your model and the hardware is doing the compatibility work, AMD’s CUDA gap is less of an obstacle. A public-domain software project is doing more for AMD’s market position than AMD’s own software team in this analysis.
Safety-oriented AI labs are accelerating Google’s commercial chip business. Labs focused on AI safety have been routing workloads to Google’s TPUs, partly to avoid concentrating more power in Nvidia. The graph shows this usage pattern is the demand signal that drove Google to open its TPU access to paying customers externally. The labs trying to reduce hardware concentration may be helping create a second large-scale hardware platform — which is either the outcome they wanted or the opposite of it, depending on how you count.
The Bottom Line
The graph’s structure points to several findings that are not obvious from reading industry news:
The training/inference split is the organizing fact of the market. Almost every competitive strategy — AMD’s memory focus, Google’s TPU pivot, Amazon’s Trainium program, NVIDIA’s interconnect moat — makes more sense when read as a response to this bifurcation than as a standalone decision.
CUDA lock-in is proven by failure cases, not by Nvidia’s claims. The moat is real, but its reality is demonstrated through what happens to competitors who try to work around it, not through anything Nvidia directly controls.
Broadcom is the most structurally stable position in the analysis. Its revenue is diversified across all competing programs simultaneously, including the open networking standard that threatens Nvidia and the custom silicon wave that competes with it.
The KV Cache Memory Wall is the most actively contested technical problem. The breadth of approaches targeting it — seven distinct methods across hardware and software — indicates it is currently the binding constraint on inference economics.
Hardware advantage does not automatically become market advantage. The graph documents four separate cases of technically superior or competitive chips failing to take market share. The pattern is consistent enough that the graph treats it as a structural rule rather than a series of coincidences. Software compatibility, ecosystem depth, and supply chain access appear to matter more than benchmark performance in determining who wins deployments.