NVIDIA Part III: The Dawn of the AI Era

The Setup

In Acquired's April 2022 Nvidia episodes, the word "generative" was never once spoken. That is how fast things changed.
Through 2022 markets fell off a cliff, crypto and Web3 burst, and Nvidia took a massive inventory write-off for capacity it thought it would not need.
Then, in the fall, large language models built on the transformer finally became useful. In November 2022, AI had its Netscape moment — and maybe its iPhone moment.
ChatGPT became the fastest app in history to 100 million active users. The entire thing ran on Nvidia hardware and software.

01

The Big Bang: AlexNet, 2012

Three researchers at the University of Toronto — Alex Krizhevsky, his advisor Jeff Hinton, and fellow PhD student Ilya Sutskever — submitted a convolutional neural network to the ImageNet competition.

ImageNet: 14 million hand-labeled images, the largest use of Mechanical Turk up to that point.
The team cut the error rate from 25% to 15% — a step change, not an increment.
They trained it on two GeForce GTX 580s: about $1,000 of consumer hardware and CUDA.
Everyone before had run these algorithms on CPUs. GPUs execute hundreds or thousands of instructions at a time.
Trivia: Jeff Hinton is the great-great-grandson of George and Mary Boole.

What Happened to the Team

They started a company. Google acquired it within six months.
Google Brain forms (Greg Corrado, Jeff Hinton, Andrew Ng); Google buys DeepMind about a year later; Facebook scoops up Yann LeCun. A duopoly on AI talent.
The payoff was enormous: the rebuilt YouTube algorithm, AI-powered Instagram recommendations — the feed that made Instagram a mega-asset for Facebook.
Astro Teller said the gains from Google Brain alone more than funded everything Google X was doing.

02

The Transformer and the Parallel Bet

Attention Is All You Need — Google Brain, 2017

The attention mechanism lets a model weigh the entire input text at once, instead of translating word by word.
Positional encoding keeps track of word order while still attending everywhere.
Computationally it is O(n²): double the input, quadruple the cost.
The unlock: transformer comparisons can run in parallel. Older recurrent networks and LSTMs had to finish one word before starting the next. GPUs love this.

Why the Data Center Is the Computer

Classic von Neumann architecture runs one instruction per clock cycle: load, load, add, store. Three of four cycles just move data. That is the bottleneck.
GPUs trade one fast core for thousands of parallel cores — an Archimedes lever on Moore's law.
Today the real constraint is memory. Models must sit in memory at once and take hundreds of gigabytes. The H-100 carries 80GB.
Chips cannot get bigger for now — they are already at the EUV reticle size limit.
So you network chips into servers, servers into racks, racks into one single "computer."

03

From a Dinner to ChatGPT

2015

Elon Musk and Sam Altman, then president of Y Combinator, host a dinner at the Rosewood Hotel on Sand Hill Road. They pitch top Google and Facebook AI researchers on breaking the duopoly. Almost everyone says no. Ilya Sutskever is the one who says yes.

2015

OpenAI is founded as an independent nonprofit AI research lab.

2015–2016

Andrej Karpathy writes about the unreasonable effectiveness of neural networks; he and Ilya appear in an Nvidia video describing language models that could one day let us talk to computers like we talk to each other.

2017

The transformer paper lands, but the rest of the field has not adopted it yet. OpenAI keeps doing scattershot research: Dota 2 bots, computer-vision self-driving simulation.

2018

Elon leaves OpenAI. A knife-point moment for the rest of the team.

March 2019

OpenAI creates a for-profit entity to raise enough capital for compute. Less than six months later, Microsoft invests $1 billion and becomes exclusive cloud provider.

2020–2021

GPT-3 ships. Microsoft licenses exclusive commercial use, adds another $2 billion, and GitHub Copilot launches.

Nov 30, 2022

ChatGPT launches — the AI heard around the world.

2023

Microsoft invests another $10 billion and integrates GPT across its products. GPT-4 arrives.

Scaling Is the Whole Story

GPT-1 had around 120 million parameters. GPT-2 had 1.5 billion. GPT-3 had 175 billion. GPT-4 is rumored at roughly 1.7 trillion.

Unsupervised pre-training means the model learns the structure and meaning of language from unlabeled text — then gets fine-tuned on labeled data. Below roughly 10 to 100 billion parameters, models were mostly nonsensical. Add more data and more parameters, and quality emerges almost unexpectedly.

Consider a detective novel. At the end, the detective names the criminal. The more accurately a model predicts that next word — the name of the criminal — the greater its understanding, not just of the novel but of human-level knowledge.Ilya Sutskever, to Jensen Huang at GTC 2023

04

The Three Legs of the Stool

1. Mellanox — the $7 Billion Networking Bet (2020)

InfiniBand was the faster, higher-bandwidth way to move data between racks. Everyone else assumed Ethernet was good enough and exited the market. Nvidia's thesis: the data center is the computer, so the wire between racks matters as much as on-chip memory.

2. Grace — Nvidia Makes a CPU (Sept 2022)

An Arm-based CPU built from the ground up to orchestrate massive GPU clusters. Not for laptops. The endgame of a ballet in motion for 30 years: the graphics card once lived in someone else's motherboard slot.

3. Hopper — Splitting the Architectures (Sept 2022)

Data center GPUs (Hopper) separate from consumer cards (Lovelace) for the first time. That lets Nvidia monopolize a huge share of TSMC's 2.5D chip-on-wafer-on-substrate packaging, co-packaging stacked high-bandwidth memory. CoWoS is only around 10–15% of TSMC's capacity.

30×faster than the A100 (9× for AI training)

18,500CUDA cores, plus 640 tensor cores

70 lbsone H-100: 35,000 parts, robots assemble it

$40,000price per H-100 chip

What It Sells

Hyperscalers buy chips. Enterprises buy systems. A DGX box runs $150k to $300k; a new DGX H-100 system starts at $500,000.
The DGX GH200 SuperPOD: 256 Grace Hopper racks in one wall. Pricing: call us.
Cloud rental runs about $30/hour for eight A100s, around $100/hour for eight H-100s.
DGX Cloud starts at $37,000 a month for a previous-generation A100 system — against an estimated build cost near $120k. Roughly a three-month payback. And it puts Nvidia in a direct sales relationship with the enterprise.

Jensen's pitch to customers: our solutions are expensive, but "the more you buy, the more you save." Training a model is like compressing the whole internet of text into a much smaller set of weights — you pay a lot once, then inference is cheap every time after.

05

CUDA: The Real Moat

Started in 2006 by Jensen Huang and Ian Buck: make it easy to use graphics cards for something other than graphics, and build a developer relationship Nvidia owns itself.
Today it is a compiler, a runtime, a debugger, a profiler, its own C++ language, and industry libraries.
Every GPU Nvidia has shipped since 2006 is CUDA-capable. Costly then. Genius in hindsight.

1,000developers after the first four years

4Mregistered developers by May 2023

~10,000person-years invested in CUDA

500MCUDA-capable GPUs in the world

The developer curve: 1 million by 2016, 2 million two years later, 3 million in 2022, 4 million a year after that.

The comparison that fits: Nvidia is Apple and iOS. PyTorch is Android — open, spread across many companies, moving to a foundation. But where iOS had roughly an 18-month head start, Nvidia's software lead here looks closer to 10 to 15 years.

06

The Numbers

$13.5BQ2 revenue, up 88% in a single quarter

$10.3Bdata center alone, up 171% year over year

70%gross margin, guiding to 72%

24%gross margin back in the commodity graphics era

The quarter before, revenue was $7.2B and Nvidia guided Q2 to $11B. The stock rose 25% after hours and the company crossed a trillion dollars in market cap — after falling below $300B months earlier.

Other Things Worth Knowing

Roughly half of data center revenue comes from cloud service providers. Consumer internet companies are next; enterprises third.
China was about 25% of revenue. Export controls ended sales of H-100s and A100s there. The nerfed A800 and H800 still sell extremely well.
26,000 employees versus Microsoft's 220,000 — about $46 million of market cap per employee.
Omniverse had 700 enterprise customers signed up, where 3D graphics and AI finally meet.
Jensen's reframed trillion-dollar market: $1 trillion of hard assets sitting in data centers, with $250 billion spent every year to refresh and expand them.

07

The Seven Powers

Counter-positioning

Incumbent data center vendors had every incentive not to rebuild the data center. Nvidia did it anyway.

Scale economies

Thousands of engineers on CUDA, amortized across four million developers. Roughly 16,600 LinkedIn profiles at Nvidia mention CUDA.

Switching costs

Data center architecture decisions last a decade or more. Some of the stickiest revenue known to humankind.

Cornered resource

A huge share of TSMC's 2.5D CoWoS capacity, reserved years ahead for a different purpose.

Network economies

Libraries, tooling and a community building on each other's work.

Brand

Nobody gets fired for buying Nvidia. Strength leads to strength.

Process power

The weakest of the set — though a six-month ship cycle and two GTCs a year are hard to copy.

08

Bull Case / Bear Case

Bull

Jensen is right about accelerated computing: most workloads still run on CPUs and could be accelerated.
Jensen is right about generative AI. The belief is showing up as real revenue, not clicks.
Nvidia moves faster than anyone. Whatever comes next, they are positioned to capture it.
$1 trillion installed, $250 billion a year in refresh spend — and Nvidia captures roughly 15 to 18% today.
TSMC says AI hardware is only 6% of its revenue and expects 50% annual growth for five years.
Nvidia is not Intel and not Cisco. It is Microsoft — or old-school IBM, selling a fully integrated system.

Bear

Everyone with resources wants a piece: custom silicon at the hyperscalers, and PyTorch moving to a foundation so it can aggregate customers and disintermediate.
The market may not be as large as the valuation implies. A crisis of confidence is a matter of when, not if.
Once models are good enough, spend shifts toward inference, where Nvidia is less differentiated.
Smarter, smaller models (Chinchilla, Llama 2) may need less compute, not more.
China becomes a closed market for top-end hardware for the foreseeable future.
A revenue spike at a high multiple is a dangerous setup — though this is a company that has risen from the ashes before.

09

What It Would Take to Compete

Design chips as good as Nvidia's.
Build chip-to-chip interconnect as good as NVLink.
Work with assemblers like Foxconn to build server systems.
Build rack-to-rack networking as good as Mellanox InfiniBand.
Convince CIOs — and you would need to be ten times better, not merely equal.
Secure TSMC's 2.5D CoWoS capacity, which is already largely spoken for.
Rebuild CUDA from scratch: about 10,000 person-years.
Then win over developers who already live inside that ecosystem.
And do all of it while Nvidia keeps shipping every six months.

The conclusion: it is nearly impossible to compete head on. If Nvidia is unseated, it comes from an unknown flank — or from accelerated computing turning out not to be the future, which seems very unlikely.

10

In Their Own Words

I relax all the time. I enjoy relaxing at work because work is relaxing for me. Solving problems is relaxing for me. Achieving something is relaxing for me.Jensen Huang

You build a great company by doing things that other people can't do. You don't build a company by fighting other people to do things that everyone can do.Jensen Huang

Build the product as a system, build the system full stack — but go to market in a disaggregated way.Jensen Huang

11

What to Watch Next

Whether enterprise AI adoption matures faster than the current wave of hype fades.
The training-versus-inference split, and how much of it stays on Nvidia silicon.
Hyperscaler custom silicon and the PyTorch foundation as the disintermediation vector.
Export controls, and what a closed Chinese market does to the demand curve.
Gross margin as a signal: 70% is an invitation for competition, not a permanent condition.
Culture as strategy — 26,000 people, six-month ship cycles, and a founder with no plans to slow down.