Request Timeouts
Per-attempt time limits on the gateway, how they interact with streaming and agentic workloads, and how to change them.
The gateway enforces hard time limits on every upstream attempt. These protect the platform from stuck upstream connections, but they matter to you directly if you run long generations or agentic pipelines: a single generation that runs longer than the limit is terminated. A client request can only run longer in total when a retry or provider fallback starts a fresh attempt.
| Limit | Default | Applies to |
|---|---|---|
| Streaming | 20 minutes | Streaming completions (stream: true) |
| Non-streaming | 10 minutes | Non-streaming completions |
How the limits behave
- They are total-duration limits, not idle limits. The timer starts when the gateway opens the upstream provider request and keeps running for the entire response — a stream is cut at the limit even while it is actively producing tokens.
- They apply per upstream attempt, not per session or conversation. Every completion call gets a fresh window, and so does every retry or provider fallback within a call — the gateway enforces no separate end-to-end timer on top of the per-attempt limits. A long-running agent that makes many calls over hours is unaffected; only a single attempt exceeding the limit fails.
- Tool execution doesn't count. When a completion finishes with
tool_calls, the HTTP request ends. The time your application spends running tools (or coordinating subagents) between requests never consumes the window. - Deploys can end very long streams early. When a gateway instance is retired during a deploy, in-flight requests get up to 8 minutes to finish before the connection is closed. Retry a stream that ends without its final chunk.
- On timeout the gateway returns a
504with typetimeout_error(see Error Handling). If the limit is hit mid-stream, the stream terminates.
Long-running agentic workloads
Coordinator/subagent architectures often hold a "master" completion open while work happens elsewhere. To stay within the limits:
- Keep any single completion under 20 minutes. Structure the coordinator as a loop of discrete completions (the standard tool-calling pattern) rather than one long-lived request that spans the whole investigation.
- Stream long generations. Non-streaming requests are capped at 10 minutes; streaming raises the per-attempt budget to 20 minutes and delivers partial output as it is produced.
- Split very long outputs. If a single generation legitimately needs more than 20 minutes (very large outputs on slow models, extensive reasoning), break it into continuation requests.
Changing the limits
Per-project overrides (Enterprise) can be set under Project Settings → Routing — see Per-Project Routing Configuration. Overrides can only lower the timeouts: the defaults above are the infrastructure ceiling on the hosted platform and cannot be raised per project.
Self-hosted deployments control the limits with environment variables on the gateway service:
| Variable | Default | Controls |
|---|---|---|
AI_STREAMING_TIMEOUT_MS | 1200000 | Streaming completions |
AI_TIMEOUT_MS | 600000 | Non-streaming completions |
GATEWAY_TIMEOUT_MS | 1500000 | Caps the default streaming timeout at 80% of this value; not a separate end-to-end timer |
When raising the limits on a self-hosted deployment, raise the surrounding
infrastructure in lockstep: your load balancer's backend/response timeout and
the gateway's shutdown grace period (SHUTDOWN_GRACE_PERIOD_MS, and e.g.
Kubernetes terminationGracePeriodSeconds) must all be at least as long as
the longest stream you allow, or rollouts and intermediaries will still cut
long requests.
If your workload genuinely needs single completions longer than 20 minutes on the hosted platform, contact us at contact@llmgateway.io — the ceiling is an infrastructure setting, not a per-model constraint.
How is this guide?
Last updated on