Curriculum · The inference layer

What Inference Is

5 lessons · For: Anyone who uses or builds with AI and wants to know what actually happens when a model answers

Almost everything written about AI is about training — the enormous, one-time job of building a model. But you never run training. What you run, every single time you call a model, is inference: the trained model reading your input and producing an output.

Inference is where the cost lives, where the latency lives, and where every economic decision about AI is actually made. If you understand it, the rest of this track — why the choice of provider matters, why compute can bear a yield, how a gateway earns its place — follows naturally. If you skip it, all of that reads as jargon.

Five short lessons, no prerequisites. By the end you will be able to read a model bill and know what each line is.

Lessons

Training builds the model; inference uses it

You should be able to: State the difference between training and inference, and why you only ever pay for the second.

A model has two halves to its life. Training is the months-long, capital-heavy job of showing a model enough data that its billions of parameters settle into something useful. It happens once, on a dedicated cluster, and the output is a fixed set of weights — a large file that does not change when you use it.

Inference is what happens afterwards, forever: you hand the frozen model an input, it runs a forward pass through those weights, and out comes an answer. Every chatbot reply, every agent action, every summary is one inference call. The weights are read, never written.

This split is the whole reason a market exists. Training is done by a handful of labs; inference is done by everyone, billions of times a day, and it is the part you rent. When this track talks about "the cost of AI" in production, it always means inference — training is a sunk cost someone else already paid.

Source: Gatewayz — the unified inference gateway

The token is the unit of work

You should be able to: Explain what a token is and why everything about inference is priced and measured in tokens.

A model does not read characters or words — it reads tokens, chunks of text that are roughly three-quarters of a word on average. "Inference" might be one token; "tokenization" might be three. Your prompt is split into tokens on the way in, and the model emits tokens one at a time on the way out.

This matters because a token is the atom of the entire economy. Providers price per token, latency is measured in tokens per second, and a model's context limit is stated in tokens. When you learn to think in tokens instead of words, every number about inference suddenly has a unit you can reason about.

The practical habit: before you build anything, estimate the tokens. A prompt with a long document in it is expensive not because it is "big" but because it is thousands of input tokens, every call. A short answer is cheap not because it is "simple" but because it is few output tokens. The bill is just token-counting.

The context window, and why it is not free

You should be able to: Describe what the context window is and why filling it has a real cost each call.

The context window is how many tokens the model can consider at once — your prompt plus its own answer-so-far, together. A large window lets you paste an entire contract or a long conversation and have the model reason over all of it. It is one of the headline numbers a model advertises.

The catch is that context is re-processed on every call. The model does not remember your last message; a stateless inference call is handed the whole conversation each time, and every one of those tokens is read again. A long-running chat that keeps appending gets more expensive with each turn, because the input grows even when your new message is short.

So the window is a budget, not a free container. Good systems manage it deliberately — trimming, summarising, or retrieving only the tokens a call actually needs. Understanding this is what separates a prototype that works from a production system whose bill does not quietly triple as conversations lengthen.

Latency, throughput, and what "fast" means

You should be able to: Distinguish time-to-first-token from tokens-per-second and know which one your product feels.

Inference speed is two numbers, not one. Time-to-first-token is how long you wait before anything appears — the model reading your input and starting to answer. Throughput, in tokens per second, is how fast the answer streams once it begins. A model can be excellent at one and poor at the other.

Which one matters depends on the product. A chat interface lives or dies on time-to-first-token: the user wants to see it start. A batch job summarising ten thousand documents overnight does not care about first-token latency at all and only wants raw throughput. The same model on the same provider can be the right or wrong choice depending on which you need.

This is the first hint of why choosing inference is an actual decision rather than a default. "Fast" is not a property of a model; it is a property of a model, on a provider, for a workload. The next course is about turning that into a routing choice instead of an accident.

Source: Why Choosing Inference Matters

What you actually pay for

You should be able to: Read an inference bill and account for input tokens, output tokens, and the model chosen.

An inference bill has three levers and no others. Input tokens: everything you sent — prompt, instructions, pasted context. Output tokens: everything the model generated. And the per-token price, which is set by the model and the provider serving it. Output tokens usually cost several times more than input tokens, because generating is harder than reading.

That structure tells you where to look when a bill surprises you. A high input cost means you are sending too much context every call — trim it. A high output cost means the model is being asked to write more than it needs to — constrain it. A high per-token rate means you have reached for a frontier model where a smaller one would have passed the check.

Everything else in this track builds on this arithmetic. Inference as yield, the case for a gateway, the routing decision — all of it is ultimately about moving these three levers in your favour, at scale, without giving up the quality the job needs.

Source: Gatewayz — the unified inference gateway

Frequently asked

What is the difference between AI training and AI inference?

Training builds a model once — a long, capital-heavy job that produces a fixed set of weights. Inference uses that finished model to answer a request, and it happens every time you call the model. You rent inference; training is a sunk cost someone else paid.

Why is everything measured in tokens?

A token is the chunk of text a model actually reads and writes — roughly three-quarters of a word. Providers price per token, speed is tokens per second, and context limits are in tokens, so a token is the single unit that makes every inference number comparable.

Why does a long conversation get more expensive?

Inference is stateless: each call is handed the whole conversation again, and every token of it is re-processed and re-charged. As a chat grows, the input grows, so the cost of each new turn rises even if your new message is short.