01
The request shape and streaming
You should be able to: Send a production-shaped request and stream tokens for a responsive interface.
A production request is the same OpenAI-compatible body you already send, with the fields you now understand: the model id, the messages, and a token cap so a runaway generation cannot cost the earth. Set the cap deliberately — it is your ceiling on the output-token lever from the inference course.
For anything a person watches, stream. Streaming returns tokens as they are generated rather than making the user wait for the whole answer, which turns the time-to-first-token metric into the thing they feel. Every OpenAI-compatible client supports it with a streaming flag; through a gateway it works the same way, because the gateway preserves the interface.
Streaming has one operational consequence worth knowing now: an error can arrive mid-stream, after some tokens have already been sent. Production code has to handle a stream that starts fine and then fails, which is a different shape from a single call that either works or does not — and it is the first reason the reliability patterns in this course exist.
Source: Flashy OS — @flashyos/llm-gateway (the inference seam)
02
Failover without a human — the circuit breaker
You should be able to: Explain automatic failover and why a circuit breaker beats a feature flag for provider outages.
A gateway's core reliability promise is that losing one provider does not cost you availability. When the chosen provider fails, the request is served by a configured fallback instead — automatically, in the same call. The estate's gateway states it as a rule: "losing a gateway must never cost availability, and a model id wrong for one provider can be right for the other."
The mechanism is a circuit breaker, and the reason it beats a manual switch is timing. A feature flag "needs a human awake; the breaker does not." In the estate's implementation, after three consecutive failures the failing provider is skipped outright for a cool-down window and traffic goes to the fallback, then the breaker heals on the next read — "nothing has to be running for it to recover". An outage is absorbed in seconds, without a page waking anyone.
The subtlety that makes a breaker trustworthy is what counts as a failure. A caller's own mistake — a malformed request, a bad model id — is a 4xx and must not trip the breaker, or "a typo could fail a provider over". The estate learned this: before it fixed the rule, "three malformed requests in a row opened the circuit and moved all traffic to the fallback for a minute". A real outage — a timeout, a dropped socket, a 429 meaning "not now" — is what the breaker is for.
Source: Flashy OS — @flashyos/llm-gateway (the inference seam)
03
An empty success is a failure
You should be able to: Explain why a zero-content 200 must be treated as an error and failed over rather than returned.
The most dangerous failure in inference is the one that looks like success. Some gateways report a failed upstream call as a response with no content and zero output tokens — a 200 status wrapping nothing. If you return that to your caller, you have shipped a silent non-answer: no error logged, no retry, just an empty result that some downstream code will treat as real.
The estate's gateway refuses this by name: "A response with no content and zero output tokens ... is raised as an error and falls over, rather than being returned as a free, silent non-answer." An empty success is reclassified as a failure precisely so the failover machinery catches it, the way it would catch a timeout.
The general lesson is one to carry beyond gateways: define what a valid result is and reject everything else, rather than trusting a status code. A 200 is a claim about the transport, not about whether the model actually answered. Systems that conflate the two fail plausibly, which is the hardest kind of failure to find.
Source: Directing Agents Well
04
Cost is the caller's — attributing spend per request
You should be able to: Attribute cost to individual requests using token counts rather than a provider total.
A gateway reports what a call used — the estate's reports "tokens and a requestId" — but deliberately "does not price the call". Pricing is the caller's job, computed against your own model registry, for a specific and important reason: "a provider's aggregate usage feed cannot attribute a figure to one request", so a total from the provider cannot tell you what any single call cost.
The pattern that follows: keep your own map of model id to per-token price, and at each call multiply the input and output token counts in the response by that model's rates to get the cost of that exact request. Now cost is attributable — to a user, a feature, a customer — instead of a monthly lump you cannot break down. The provider's total becomes a reconciliation check against the sum of your per-call figures, not your source of truth.
This is what makes routing economically legible. You cannot decide that easy requests should go to cheap capacity if you cannot see what each request costs. Per-request attribution is the measurement that turns the choosing-inference strategy from a belief into a managed number.
Source: Inference as Yield
05
Cutover is configuration, not a deploy
You should be able to: Change which provider serves traffic without shipping code, and explain why that is safe.
Because a gateway puts the provider behind a stable seam, changing who serves your inference is an environment change rather than a code change. The estate's gateway is explicit: "provider selects who serves inference; the other configured provider becomes the automatic fallback. Changing providers is an environment change, never a deploy." A cutover is flipping a config value.
This matters operationally more than it first sounds. A price rise, a provider degradation, or a better model becoming available are all handled by changing configuration — no build, no release, no risk window from shipping code. And the seam degrades safely by design: "an unknown provider id, or a primary without credentials, falls back" rather than failing, so a misconfiguration cannot take you down.
The discipline this rewards is keeping the seam clean: your application talks to the gateway and never imports a specific provider's client directly. The estate enforces exactly that boundary — the package "does not read your environment or import your error class". Keep provider knowledge on one side of the seam and cutover stays a config edit forever, instead of decaying back into a migration.
Source: Why Choosing Inference Matters
06
Advanced features — forwarded, but verify per gateway
You should be able to: Use tool calling and prompt caching through a gateway, and verify each behaves before relying on it.
The advanced features are where a gateway can silently alter your request, and the failure mode is silent, which makes this the lesson to internalise. Gatewayz forwards them: tools, tool_choice, parallel_tool_calls and response_format are passed to the provider unmodified (a parameter a given provider does not accept is pruned rather than errored), and prompt caching via cache_control is passed through and billed at the cache rate. So the features work — the discipline is confirming they worked, per gateway, because "seems fine" and "silently degraded" look identical from the outside.
Verify with the numbers, not the vibe. For caching, read cache_read_input_tokens in the response usage block: zero on a repeat request means your cache breakpoint is not taking effect (usually the prefix is below the provider's minimum cacheable length, or it changed between turns) — and you are quietly paying full input price. For tools, confirm tool_calls come back and the round-trip survives. Check GET /v1/models first: each entry's capabilities object tells you whether a model supports caching, tools or vision before you count on it — capabilities.prompt_caching is the one to read for caching.
Gatewayz's own coding-agent benchmark page is the model of this discipline: it publishes measured results only, and if a run sends cache_control but gets back zero cache reads, it emits a warning and publishes no cost-advantage claim from it. That is the estate's own rule — a number you cannot verify is not a number you publish — enforced by the gateway itself.
A distinction worth keeping straight: the estate's own inference seam, @flashyos/llm-gateway, still lists tool use and prompt caching as not-yet-verified through its code. That is a fact about the estate package's own test coverage, not about Gatewayz, which documents and forwards both. The rule generalises regardless of gateway: trust the common path, verify the advanced ones, and let the token counts settle it. The next course goes deep on exactly how.
Source: Prompt Caching and Tool Use
