NATS JetStream Redelivery Is Not the Same as Consumer Lag

An illustrative JetStream incident snapshot can look contradictory: num_pending is 18,240, num_ack_pending is fixed at 64, and num_redelivered is 27. One dashboard may call all three values “consumer lag,” but they describe different ownership states. Pending messages have not reached a worker. Ack-pending messages are already held by workers. Redelivered messages have entered at least one additional delivery attempt.

That distinction changes the repair. Adding workers can reduce undelivered backlog. Raising MaxAckPending can let more work enter the pool. Extending an acknowledgment deadline can protect legitimately slow processing. None of those changes repairs a handler that is wedged, a client that sends immediate negative acknowledgments, or side effects that are unsafe to run twice.

Start with the three counters that disagree

Current NATS JetStream health documentation defines the state precisely. num_pending counts matching messages that the consumer has not delivered. num_ack_pending counts messages already delivered but not finally acknowledged. num_redelivered counts messages the server is currently tracking as delivered more than once; it is not a lifetime incident total.

Read those values together, not as interchangeable symptoms:

• Pending rises while ack pending stays below its ceiling: workers are not fetching fast enough, are absent, or receive less work than publishers create.
• Ack pending sits at its ceiling while pending rises behind it: the in-flight window is full. Handlers may be slow, blocked, oversized in batch behavior, or missing acknowledgments.
• Redelivered rises while the same business keys reappear: acknowledgments are late, workers die after delivery, code requests retries, or poison messages consume repeated attempts.

Comparing protocols helps only when the state models remain separate. Kafka partition-lag attribution follows committed and log-end offsets per partition; JetStream adds an explicit delivered-but-unacknowledged window that must be read independently.

Freeze one consumer window before changing limits

Capture stream state, consumer state, configuration, and worker behavior inside the same short interval. A single high number without direction cannot tell whether a consumer is draining, stalled, or recovering.

stream='ORDERS'
consumer='shipping'
nats stream info "$stream" --json | jq '.state | {messages,first_seq,last_seq,num_subjects}'
nats consumer info "$stream" "$consumer" --json | jq '{num_pending,num_ack_pending,num_redelivered,num_waiting,delivered,ack_floor,config:{ack_policy:.config.ack_policy,ack_wait:.config.ack_wait,max_ack_pending:.config.max_ack_pending,max_deliver:.config.max_deliver,backoff:.config.backoff}}'

Repeat the read after one representative processing interval and preserve both outputs. If delivered.stream_seq advances and num_pending falls, the pool is draining. If num_ack_pending remains pinned while acknowledgments do not advance, workers already own the blockage. If delivery advances but redelivery also grows, throughput may be hiding duplicate work.

Put worker evidence beside broker evidence

JetStream cannot distinguish a slow handler from a dead process until the acknowledgment timer expires. Align consumer state with worker logs, process restarts, downstream latency, and completed business operations. A payment call that succeeded before the worker crashed is operationally different from a task that never began, even though both lack a final ack.

Host scheduling can stretch handler time without saturating the whole machine. cgroup CPU-throttling evidence helps prove whether a container or service quota delayed progress before anyone expands an ack deadline.

Monitoring also needs a security boundary. Current NATS monitoring-endpoint documentation identifies :8222 as unauthenticated by default. Keep the monitoring listener private or protect it through a controlled system-account/exporter design; never expose detailed /jsz or /connz data directly to the public internet.

Find the response or clock that caused another delivery

With explicit acknowledgments, a delivered message remains in flight until the client sends a final response. The current acknowledgment and redelivery guide gives the client four verbs:

1. ack closes successful work.
2. nak requests another delivery, immediately unless the client supplies a delay.
3. term stops attempts for work the client knows can never succeed.
4. in-progress resets the acknowledgment timer while the worker still owns valid work.

Silence takes a different route. AckWait starts when the message is delivered; if no final response or in-progress signal arrives before the deadline, JetStream assumes the worker failed and redelivers. A worker pool can therefore process the same message on two machines when one worker is merely slow or when the first worker completed an external side effect but lost its ack.

A plain NAK does not wait for consumer BackOff. It asks for immediate redelivery. Use a delayed NAK for a known temporary dependency failure. Consumer BackOff governs timeout-driven redelivery and replaces AckWait; its first interval becomes the first delivery’s effective acknowledgment deadline. A schedule beginning at one second will create premature retries for jobs that normally need ten seconds, even if an older AckWait field still looks reassuring in a deployment file.

Restarting workers is another separate control. Docker health and restart-policy boundaries explain why a health label does not prove lifecycle action, while systemd restart-budget behavior helps keep repeated worker failure from becoming a host-level restart storm.

Tune the ownership contract, not one dashboard number

Choose values from measured handler behavior and business risk. Record normal, high-percentile, and worst accepted processing time; worker count; messages held per worker; payload memory; downstream concurrency; and cost of duplicate side effects. No universal AckWait or MaxAckPending value is safe for every consumer.

For an existing explicit-ack consumer that uses AckWait rather than BackOff, set an evidence-based deadline and a bounded attempt count. Ack policy is not editable through nats consumer edit; create a replacement durable consumer if that policy must change. Read the result back rather than trusting an edited file:

nats consumer edit ORDERS shipping --wait=45s --max-deliver=6
nats consumer info ORDERS shipping

When timeout retries need increasing distance, use BackOff as an alternative timing contract. The current CLI builds a linear schedule; its minimum replaces AckWait for the first delivery, so start above normal successful processing time:

nats consumer edit ORDERS shipping --backoff=linear --backoff-steps=5 --backoff-min=45s --backoff-max=10m
nats consumer info ORDERS shipping

Do not paste both examples into production as a ritual. Choose the timing model, canary it on one consumer, and preserve the previous configuration for rollback. A larger deadline delays real crash recovery. A shorter deadline creates parallel work. BackOff reduces retry pressure after silent timeouts but does not slow a bare NAK.

MaxAckPending belongs to the consumer, not each worker

Current JetStream worker-pool guidance makes the scope explicit: several processes sharing one durable consumer also share one MaxAckPending ceiling. Ten workers do not each receive 1,000 slots when the consumer limit is 1,000. Conversely, a limit of ten can idle most of a large pool even when thousands of messages remain pending.

Plan the ceiling from intentional concurrency, per-message memory, downstream limits, and failure blast radius. Raising it can improve throughput when healthy workers are starved, but it also increases the amount of unfinished work that may redeliver after correlated worker loss. A pinned ceiling is evidence to investigate, not automatic permission to enlarge it.

Broker-side pressure needs its own branch. RabbitMQ disk-alarm recovery illustrates a storage watermark that blocks publishing across a broker. JetStream ack saturation is consumer-side state; do not borrow a broker-capacity fix until NATS stream, server, and storage evidence independently points there.

Poison messages need a durable exit receipt

MaxDeliver bounds delivery attempts; it is not a dead-letter queue. Once the limit is exceeded, JetStream stops delivering that message to this consumer and emits a max-deliveries advisory. Stream retention determines whether the original stored message still exists.

Advisories are transient unless captured. Current NATS advisory guidance creates a file-backed stream that is always subscribed, so an unattended event is available for later inspection:

nats stream add ADVISORIES --subjects '$JS.EVENT.ADVISORY.>' --storage file --retention limits --max-age 168h --defaults
nats stream view ADVISORIES --subject '$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>'

Review retention, authorization, storage capacity, and alert ownership before creating that stream. A complete dead-letter workflow must also identify the original message, preserve its business key and failure reason, copy it to a governed parking stream when appropriate, and define who may replay it. Synadia’s current JetStream reliable-delivery pattern makes the same boundary explicit: NATS supplies composable primitives, not a one-line DLQ switch.

Prove recovery with one idempotent message

Use a non-production subject or a safely isolated test consumer whose handler records a unique business key. First prove one normal delivery and acknowledgment. Next introduce a controlled delay longer than the current deadline, observe the delivery count rise, and confirm the second attempt cannot repeat the external side effect. Finally apply the bounded change and replay the same timing condition.

Acceptance needs more than a falling chart. Require num_pending to drain at the expected rate, num_ack_pending to retain headroom, redeliveries to return near the workload’s normal baseline, worker errors to stop, and the business result to remain single and correct. Alert states should distinguish missing telemetry from real lag or retry growth; Grafana No Data and Error policy provides that separate monitoring contract.

Rollback restores the prior consumer configuration and stops the test before production throughput or duplicate cost grows. Keep the two counter snapshots, worker receipt, configuration diff, advisory record, and business-key result together. JetStream recovery is proven when delivery progresses and repeated attempts remain safe—not when one counter merely becomes smaller.