01
The accelerator — why a GPU, not a CPU
You should be able to: Explain why inference runs on GPUs and other accelerators rather than ordinary processors.
A model's forward pass is, at bottom, a very large number of multiply-and-add operations arranged as matrix maths. A CPU does a few things quickly and in sequence; that is the wrong shape for this work. A GPU — and its more specialised cousins, the accelerators built specifically for AI — does thousands of the same operation in parallel, which is exactly the shape a model needs.
This is why "compute" in the context of AI almost always means accelerators, not general servers. The same job that would take a CPU an unusable amount of time runs in milliseconds on the right chip. The chip is the productive asset; everything else in the data centre exists to keep it fed and cool.
The consequence is concentration. These chips are hard to design, hard to manufacture, and in constant demand, so they are scarce and expensive. Scarcity of the productive asset is the root fact from which the rest of this course — and the yield thesis — grows.
Source: Gatewayz — the unified inference gateway
02
Memory is the real constraint
You should be able to: Explain why an accelerator's memory, not its raw speed, usually decides what it can run.
The intuitive assumption is that a faster chip is a better chip. For inference, the binding constraint is usually memory: how many billions of parameters can physically fit on the accelerator, and how fast those parameters can be moved to where the arithmetic happens. A model that does not fit in memory cannot run on that chip at all, at any speed.
This is why the same model can be cheap on one setup and impossible on another, and why "how big is the model" translates directly into "how much hardware does one call require". Memory bandwidth — the rate parameters can be shuttled around — often decides throughput more than the raw operation count does.
It also explains batching, the single most important efficiency trick in inference. Loading a model's weights into an accelerator is expensive; once they are there, serving many requests together spreads that cost across all of them. A chip serving one request at a time is mostly wasting the effort it spent loading the model. Hold that thought — it is the seed of the utilisation argument.
03
Where compute comes from
You should be able to: Trace the supply chain from chip fabrication to a served token, and name the bottlenecks.
A served token has a supply chain behind it. A handful of foundries fabricate the accelerators. Those chips are assembled into servers, racked in data centres, connected with high-speed networking, and — the constraint people forget — supplied with enormous, continuous electrical power and cooling. Any one of those links can be the bottleneck, and lately most of them have been at once.
Power is increasingly the hard limit. A modern AI data centre draws on the scale of a small town, and you cannot conjure grid capacity on the timeline that chips ship. This is why compute buildout is talked about in the same breath as energy policy — the two are the same problem viewed from different ends.
For a builder, the takeaway is not to memorise the supply chain but to respect it: compute is genuinely scarce, its scarcity is physical rather than artificial, and that scarcity is what gives idle capacity real value to whoever can put it to work. That is the hinge the next lesson turns on.
04
Utilisation — why idle compute is the whole game
You should be able to: Explain why an idle accelerator is a loss and why utilisation is the central economic metric.
An accelerator costs almost the same whether it is working or idle. It was expensive to acquire, it draws power to stay ready, and it depreciates on a clock that does not care whether you used it. So the economic question for anyone who owns compute is not "how fast is it" but "what fraction of the time is it actually doing paid work". That fraction is utilisation, and it is the number that decides whether owning compute makes or loses money.
Demand for inference is spiky and uneven. A given owner's own workloads rarely keep their hardware busy around the clock, so a fleet sized for peak sits partly idle at the trough — burning power, depreciating, earning nothing. Idle capacity is not neutral; it is a running loss.
This is the problem the rest of the track solves. If idle compute could be pointed at someone else's demand — safely, on demand, and paid — then the owner's loss becomes income and the renter gets capacity that would otherwise not exist. Aggregating demand and matching it to idle supply is exactly what a gateway does, and it is why inference can bear a yield.
Source: Inference as Yield
05
From compute to inference: the conversion
You should be able to: Describe how raw compute becomes served inference, and where value is added at each step.
Raw compute is not a product a developer can use; it is a chip in a rack. Turning it into inference someone will pay for takes several layers: a serving stack that loads the model and manages batching, an API that accepts requests and returns tokens, billing and rate-limiting so usage can be metered, and reliability so a call does not silently fail. Each layer adds value and takes a margin.
This is why "I have GPUs" and "I sell inference" are very different businesses. The gap between them — the software that converts a productive asset into a metered, reliable, developer-facing service — is where most of the operational work and much of the value sits.
It is also why gateways matter. A gateway sits at the top of this stack and aggregates across many providers who have each done the conversion, so a developer gets one door to all of them and a compute owner gets access to demand they could not reach alone. With the machine understood, you are ready to see why the choice among all this inference is itself a decision worth making well.
Source: Why Choosing Inference Matters
