Envoy 503 UO: Find the Circuit Breaker That Opened

An Envoy access log with HTTP 503 and response flag UO records a local upstream-overflow decision. Envoy’s circuit breaker refused the request; the status alone does not prove that an upstream application returned 503. The next question is not “How high should max_pending_requests be?” It is “Which cluster, priority, and resource limit rejected this request?”

Correlate the log event with a short before/after slice of cluster overflow counters, the matched route’s priority, priority-specific breaker gauges, and the configuration actually loaded by the affected proxy. Connections, pending requests, active requests, retries, and connection pools have different breakers. Change only the boundary supported by that combined evidence, then prove under controlled load that refusals stopped without turning into upstream latency, retry amplification, or host saturation.

Prove that Envoy refused the request locally

Envoy’s current response-flag reference defines UO as UpstreamOverflow: upstream circuit breaking in addition to HTTP 503. That makes UO materially different from UH (no healthy upstream), UF (upstream connection failure), UT (upstream request timeout, normally 504), and OM (Overload Manager termination).

Capture more than status and flag. A structured access log should retain route, upstream cluster, response-code details, request duration, upstream host, and attempt count. The exact HTTP connection-manager wrapper varies by deployment, but these fields form a useful evidence core:

log_format:
  json_format:
    start_time: "%START_TIME%"
    request_id: "%REQUEST_HEADER(X-REQUEST-ID)%"
    route: "%ROUTE_NAME%"
    upstream_cluster: "%UPSTREAM_CLUSTER%"
    response_code: "%RESPONSE_CODE%"
    response_flags: "%RESPONSE_FLAGS%"
    response_code_details: "%RESPONSE_CODE_DETAILS%"
    upstream_attempts: "%UPSTREAM_REQUEST_ATTEMPT_COUNT%"
    upstream_host: "%UPSTREAM_HOST%"
    duration_ms: "%DURATION%"

For one rejected request, UO plus upstream_attempts: 0 is strong corroboration that Envoy never began an upstream attempt. A populated upstream host or nonzero attempt count does not erase the flag; retries and connection-pool paths can create more complex records, so retain the entire event.

Client abandonment belongs to a different evidence chain. If the downstream disconnects before the proxy can answer, compare the event against nginx 499 client-closed-request evidence rather than relabeling every early failure as an Envoy breaker.

One UO flag can point to five resource limits

Envoy implements circuit breaking per upstream cluster and routing priority. The circuit-breaking architecture names five important resources:

• max_connections limits connections Envoy establishes across the cluster; upstream_cx_overflow increments when that breaker refuses another allocation.
• max_pending_requests limits work waiting for a ready connection-pool connection; upstream_rq_pending_overflow records failed pending work.
• max_requests limits outstanding HTTP requests; current Envoy uses upstream_rq_active_overflow for that rejection path.
• max_retries or a retry budget limits concurrent retries; upstream_rq_retry_overflow records refused retry work.
• max_connection_pools limits concurrently instantiated pools; upstream_cx_pool_overflow identifies that less common boundary.

Backend health and capacity are separate. A healthy endpoint can coexist with a local concurrency limit, while an unhealthy endpoint produces different flags and health statistics. When active probes are failing, follow HAProxy backend-health evidence as a conceptual boundary: first prove what the probe tested, then decide whether health or admission capacity owns the incident.

Envoy’s v3 circuit-breaker API reference documents defaults of 1024 for connections, pending requests, and active requests, and 3 for active retries when fields are omitted. Defaults are not capacity recommendations. Each Envoy process enforces distributed limits without coordinating a fleet-wide global total; worker threads inside one process share the limit with eventual-consistency races.

Take one synchronized evidence slice

Counters are cumulative. A large number collected after several unrelated deploys does not identify the current refusal. Record two samples around a short, known interval and calculate deltas for the affected cluster and priority.

Keep the admin listener private. Envoy’s administration-interface documentation warns that it exposes cluster names, certificate/configuration details, stats, and destructive operations. Bind it to localhost or a secured management network; never publish port 9901 to the internet for convenience.

From the affected Envoy instance, filter the local stats surface. Envoy’s current cluster-statistics reference documents the overflow counters at cluster.<name> and breaker state at cluster.<name>.circuit_breakers.<priority>:

curl -sS 'http://127.0.0.1:9901/stats?filter=^cluster\.payments\.(upstream_cx_(active|overflow|pool_overflow)|upstream_rq_(active|pending_active|active_overflow|pending_overflow|retry_overflow))$'
curl -sS 'http://127.0.0.1:9901/stats?filter=^cluster\.payments\.circuit_breakers\.(default|high)\.(cx_open|cx_pool_open|rq_pending_open|rq_open|rq_retry_open|remaining_cx|remaining_pending|remaining_rq|remaining_retries)$'

Save sample A, reproduce or wait through a bounded traffic window, and save sample B. Pair each overflow delta with the related active gauge, request rate, upstream latency, connection-establishment failures, endpoint health, and proxy instance identity. The aggregate counter that increased names the resource boundary, not its priority. A priority-specific *_open gauge shows whether DEFAULT or HIGH was at capacity during the slice; remaining_* appears only when track_remaining is enabled.

Loaded state matters more than a repository file. Query the affected process for cluster data and narrowly masked cluster configuration rather than assuming that CDS delivered the expected threshold:

curl -sS 'http://127.0.0.1:9901/clusters?format=json&filter=^payments$'
curl -sS 'http://127.0.0.1:9901/config_dump?name_regex=^payments$'

Then use the logged route name to find its loaded route action. Envoy’s route API reference defines the optional RouteAction.priority; an omitted value uses DEFAULT. If HIGH is loaded for the matched route, compare against the HIGH threshold and gauges. If both priorities were active or the gauge slice missed the event, preserve the ambiguity instead of assigning a priority from the cluster-level overflow delta alone.

Redact configuration before attaching it to an incident. Typed TLS private-key/password fields are redacted by Envoy where supported, but cluster names, endpoints, metadata, and other deployment details can still be sensitive.

Read the counter that moved

Pending overflow means no ready pool capacity

upstream_rq_pending_overflow grows when requests waiting for a ready connection cannot enter the pending queue. Look for slow or failed connection establishment, too few usable connections, endpoint churn, and burst arrival while the pool is warming. Raising the pending limit creates a larger waiting room; it does not create upstream service capacity.

HTTP/2 changes the shape. Without restrictive concurrent-stream or requests-per-connection settings, many requests multiplex over an established connection, so the pending breaker may appear mainly while no connection is ready. A cold connection, TLS handshake delay, stream limit, draining connection, or connection churn can still make pending work real.

Active overflow means in-flight request capacity is full

upstream_rq_active_overflow identifies exhaustion of max_requests while Envoy attaches work to a ready upstream connection. Current documentation says the legacy pending-overflow counter is no longer incremented for this path by default. Older builds or a runtime setting that preserves compatibility can make dashboards show both. Check upstream_rq_active_overflow and the deployed runtime/version before blaming max_pending_requests.

Compare active requests with upstream latency. At roughly steady state, concurrency follows request rate multiplied by service time. If latency doubled while arrival rate stayed flat, the same threshold can open even though traffic did not double. Raising max_requests may simply allow more work to accumulate inside the application.

Connection overflow needs a socket-capacity explanation

upstream_cx_overflow means Envoy reached the cluster connection limit, but active connections can sometimes exceed the configured maximum because Envoy ensures selected hosts can receive at least one connection and because pools/workers affect the bound. Read the official caveat before treating the threshold as an exact observed ceiling.

When connection attempts or established sockets are slow at the upstream host, inspect its Linux accept-queue and listener-drain evidence before allocating more proxy connections. A higher Envoy limit cannot make an application call accept() faster or repair a saturated file-descriptor/process boundary.

Retry overflow is often the safety mechanism working

upstream_rq_retry_overflow says the retry breaker rejected another parallel retry. Do not automatically raise it. Retries multiply load precisely when an upstream may already be failing or slow. Envoy recommends retry budgets; a budget relates retry concurrency to active plus pending work and overrides the static retry breaker.

Pool overflow is not the same as connection overflow

upstream_cx_pool_overflow identifies exhaustion of concurrent connection pools. Features that create pools by source, protocol, transport option, or other key can expand pool cardinality even when individual connections eventually close. Find the pool key owner and reclaim idle pools where supported before increasing an effectively unbounded topology.

Turn workload evidence into one breaker change

Use an incident window or a load test to estimate the owner before editing YAML. For active requests, a useful starting model is concurrency ≈ arrival rate × service time. Use a high but representative latency percentile, add a measured burst allowance, and reserve upstream recovery headroom. For pending work, measure connection-ready delay and burst duration rather than borrowing the active-request number.

The configuration below is illustrative, not a universal capacity answer. It makes the chosen limits observable and uses a retry budget instead of granting retries an unrelated static ceiling:

circuit_breakers:
  thresholds:
  - priority: DEFAULT
    max_connections: 400
    max_pending_requests: 120
    max_requests: 800
    retry_budget:
      budget_percent:
        value: 15.0
      min_retry_concurrency: 3
    track_remaining: true

track_remaining: true publishes remaining-resource gauges for configured breakers, except retry resources when a retry budget replaces max_retries. Alert on overflow deltas and sustained low remaining headroom, not on a single static percentage. When a scraper disappears, treat missing telemetry separately from a real zero by applying Grafana No Data versus Error policy.

Priority is part of the contract. DEFAULT and HIGH thresholds are separate; if no threshold exists for a priority, Envoy uses defaults. Duplicate entries for the same priority do not merge—the first one is used—so review the loaded cluster rather than assuming later YAML overrides earlier values.

Change the breaker without moving the bottleneck

Validate the exact assembled configuration with the same Envoy binary family before rollout. The Envoy CLI reference documents validate mode:

envoy --mode validate -c /etc/envoy/envoy.yaml

For xDS-managed clusters, schema validation of a local bootstrap does not prove the control plane will deliver the intended cluster resource. Roll to one proxy or a small canary cohort, confirm the loaded threshold and priority, then apply representative traffic. Preserve an immediate rollback to the prior resource version.

Stop the rollout if UO shifts into rising upstream latency, connection failures, host CPU/run queue, application queue depth, or retry volume. A larger breaker is acceptable only when the upstream has measured headroom for the admitted work. If the original limit was intentionally protecting a dependency, retain it and reduce concurrency at the caller, shed optional work, shorten overload duration, or add verified upstream capacity instead.

Because breakers are fully distributed, changing one Envoy instance changes only that process’s share. Fleet acceptance must account for instance count, uneven load balancing, canaries, and mixed configuration versions; multiplying one per-process threshold by fleet size is only an upper-bound model, not a guaranteed global concurrency cap.

Close with a local-refusal receipt

Record the UTC window, request ID sample, response code/flags/details, upstream attempt count, Envoy build/runtime, cluster and priority, loaded thresholds, two counter snapshots, relevant gauge/latency deltas, change version, canary scope, rollback trigger, and upstream acceptance signals. Repeat the same controlled window after rollout.

The incident closes when the intended workload crosses Envoy without a new overflow while the upstream remains inside its latency, error, connection, CPU, and queue budgets. Browse related DevOps operations guides only after that receipt is complete; this page’s result is narrower: one UO event traced to one measured admission boundary and one proportionate change.

Envoy 503 UO is useful failure evidence, not merely an error to suppress. Read the counter that moved, preserve fast backpressure, and raise a limit only when the next system in the path can safely accept the work.