Most teams treat rate limits as infrastructure. Someone puts a counter in front of the model client, returns 429 when the counter overflows, and moves on. That is a reasonable first step. It is also an incomplete description of what a rate limit does once real users are on the other side of it.
A rate limit is a product decision wearing an HTTP status code. It decides who waits, who fails, who gets a degraded answer, and who keeps going. It decides whether a busy hour looks like a queue, a partial outage, or a quiet refusal. If you only configure it in the gateway and never look at how it shows up in the product, you will eventually ship a feature that fails in ways your support team cannot explain and your customers cannot predict.
What the limit is actually allocating
In a conventional API, a rate limit mostly protects the server. You have finite CPU, memory, and connection slots. You shed load so the process stays up. The mental model is mechanical: capacity in, requests out, excess rejected.
LLM features complicate that model. The scarce resource is not only your process. It is provider quota, token budget, concurrent context windows, embedding throughput, and the human patience of whoever is waiting for the answer. A single user paste can burn more tokens than a hundred ordinary requests. A retry loop that looks polite in application logs can exhaust a shared organization quota in minutes. A background job that summarizes overnight tickets can starve the interactive path that a customer is staring at during a sales call.
When those things collide, the rate limit is not protecting "the system" in the abstract. It is choosing which of those workloads gets to continue. That choice has a product shape even if nobody wrote it down as one.
Teams often discover this the hard way. The interactive assistant starts returning capacity errors at 10 a.m. because a batch eval job, scheduled with the best of intentions, is hammering the same key. Or a power user with a large corpus triggers summarization that blows through the remaining daily tokens, and every other tenant in the same shared pool starts failing for reasons that look random from the outside. The logs say rate limited. The product experience says the feature is broken.
Make the policy visible to the product
If the only place a rate limit exists is in middleware, the product has almost no room to respond intelligently. The request fails. The UI shows a generic error. The user retries. The retries make the limit worse. Support opens a ticket that says "AI is down." Engineering digs through traces and eventually finds a 429 buried under three layers of client wrappers.
A better pattern is to treat the limit as a first-class signal that the product can see and explain. That does not mean dumping provider error text into the interface. It means deciding, ahead of time, what the product should do when capacity is scarce.
Sometimes the right answer is a hard stop with a clear message and a retry-after time the UI actually honors. Sometimes it is a queue with an honest wait estimate. Sometimes it is a cheaper or smaller model for the remainder of the window. Sometimes it is a truncated context with a note that the system used less history than usual. Sometimes it is deferring non-interactive work until the interactive path has headroom again.
Those are product behaviors. They require product ownership. An infra-only limit cannot choose among them because it does not know which request is a user waiting on a screen and which request is a nightly job that can wait an hour.
Separate the limits that matter
One shared counter across every LLM call is easy to implement and usually wrong. Interactive chat, background summarization, eval runs, retrieval embedding refreshes, and tool-calling loops have different latency tolerances and different failure costs. Collapsing them into a single bucket means the least important traffic can take capacity from the most important traffic whenever it happens to arrive first.
A practical setup usually needs at least a few independent ceilings. Per-tenant caps keep one customer from exhausting a shared provider key. Per-route caps keep batch work from starving interactive paths. Per-user caps keep a single session from turning into an accidental denial of service against everyone else. Global organization caps still matter, but they should be the backstop, not the only control.
You also want the counters to match the scarce resource. Requests per minute is a blunt instrument for LLM traffic. Tokens per minute, concurrent in-flight calls, and estimated dollar spend over a window are closer to the thing you are actually rationing. A 20-request burst of tiny completions is not the same event as three requests that each stuff 80k tokens of context into a frontier model. If your limiter cannot tell them apart, your product will behave as if they were the same.
Design the failure the user sees
The worst rate-limit UX is silence followed by a cryptic failure after the user has already waited. The second worst is an aggressive client retry that turns a short capacity blip into a longer outage. Both happen when the team thinks of the limit as an infrastructure detail rather than a user-visible state.
Decide the failure mode explicitly. If the call will not run, say so quickly. If it will run later, say when. If a fallback path is in use, say that the answer may be thinner or slower than usual. Users tolerate scarcity better when the system is coherent about it. They do not tolerate a feature that sometimes works, sometimes spins, and sometimes returns an error that looks like the model itself has vanished.
Retries belong in the same design conversation. Blind exponential backoff is fine for idempotent infrastructure calls. It is a poor default for a user-facing generation that may already have partially rendered, charged tokens, or written side effects. Prefer server-directed retry timing, idempotency keys where the provider supports them, and product-level decisions about whether a second attempt is even desirable.
Instrument the limit like a feature, not an exception
If rate limiting only appears in error logs, you will always be late to the conversation. Track how often limits are approached, which tenants are near their caps, which routes consume the budget, and how often the product took a fallback path instead of a hard failure. Those numbers are as useful as latency and token cost. They tell you whether the limit is doing protective work or just converting demand into confused customers.
They also tell you when the product needs a different shape. Persistent saturation on an interactive route is not an ops ticket by itself. It may mean the feature is too expensive per action, the context is too large, the model is too heavy for the job, or the pricing and packaging assume unlimited use that the infrastructure cannot deliver. A rate limit that trips every afternoon is feedback about the product, not only about capacity.
Ownership
The common failure mode is split ownership. Platform owns the gateway. Product owns the UI. Nobody owns the behavior in between. The gateway returns 429. The UI shows a toast. The customer churns for a reason that never quite makes it into a postmortem.
Give someone the job of defining the scarce-resource policy for each LLM surface: what is protected, what is deferred, what is degraded, what is refused, and what the user is told. Write it down next to the feature, not only next to the ingress config. Review it when model pricing changes, when a new batch job lands, and when a tenant starts generating traffic that looks unlike the traffic you designed for.
Rate limits will always be infrastructure in the narrow sense. Packets still get counted. Keys still have quotas. Providers still return 429. The operational mistake is stopping the design there. Once an LLM feature is in front of users, the limit is part of the product's behavior under load. Treat it that way, and the failures get smaller, clearer, and much easier to operate.