NATS JetStream Deduplication Ends When the Window Expires

NATS JetStream deduplication is temporary publisher protection, not a permanent uniqueness constraint. When two publishes carry the same Nats-Msg-Id inside a stream’s duplicate window, JetStream stores the first message and returns its original sequence for the retry with duplicate: true. Once that window expires, the same ID can be stored again with a new stream sequence.

That boundary matters to operators designing retry policy, not only to NATS client developers. This article is for readers who can run Bash and administer a self-hosted message service. It uses a three-second window so expiry can be observed quickly; a production duration must come from measured publisher retry behavior, message rate, memory evidence, and business-side reconciliation requirements.

The server remembers an ID for a bounded time

JetStream’s current publishing guide describes the ambiguous failure that deduplication addresses. A server may store a message while its publish acknowledgement, or PubAck, is lost in transit. The publisher sees a timeout and cannot know whether a retry would be the first write or a duplicate. A stable Nats-Msg-Id lets the stream recognize that retry while the ID remains in its tracking window.

Three fields in the PubAck drive the decision:

• stream names the stream that accepted the publish.
• sequence identifies the stored stream position.
• duplicate reports whether JetStream recognized the ID inside the active window.

Current JetStream header reference defines Nats-Msg-Id specifically as a duplicate-detection key within the configured duration. The setting belongs to the stream, not to each message. In the stream API, duplicate_window is expressed in nanoseconds; zero requests the server default, according to the stream-create schema.

Deletion or acknowledgement does not turn that timed memory into a permanent key registry. A historical NATS WorkQueue reproduction showed the same message ID still returning duplicate: true after the stored message had been consumed and removed. Producer admission and consumer lifecycle are deliberately different planes.

Choose the duration from the retry envelope

Start with the longest period during which a publisher may legitimately retry an ambiguous publish. Include client timeout, reconnect delay, exponential backoff, broker or network failover, process restarts that retain an outbox, and the maximum age at which the producer still considers the original business operation pending. Add a reviewed margin for scheduling and clock uncertainty.

For example, a client using one 2-second timeout followed by retries after 1, 2, 4, and 8 seconds can attempt the same publish more than 17 seconds after the first send once connection and execution time are included. A 10-second duplicate window would not cover that policy. The correct response is to align retry and stream contracts, not to hope most retries arrive early.

Longer is not automatically safer. The server must retain more IDs as the window and publish rate grow. Synadia’s current large deduplication-window check advises matching the duration to the real retry interval and monitoring memory at the actual message rate. Its per-entry estimate is useful planning evidence, but operators should validate their own version, workload, cardinality, and cluster topology before turning an estimate into capacity.

A practical admission record contains:

1. percentile and maximum observed age of ambiguous retries;
2. proposed window plus explicit margin;
3. peak unique message IDs per second;
4. server memory and latency during a representative replay burst;
5. business-side idempotency and reconciliation owner.

Metrics design matters here because message IDs are high-cardinality values. Keep raw IDs in logs or traces when needed; expose aggregate retry-age buckets, duplicate rates, and window-exceeded counts as metrics. Prometheus cardinality controls explain why putting unbounded identity values into labels can create another memory problem while attempting to observe the first one.

Reproduce one three-second boundary on loopback

The following lab ran on Debian 13 on August 12, 2026 UTC with checksum-verified NATS Server 2.14.4 and NATS CLI 0.4.0. It binds only to 127.0.0.1:14244, stores data beneath one mode-0700 directory, refuses an existing path or listener, and records the exact server process before cleanup. GitHub release downloads are verified against each project’s published SHA256SUMS file.

Run all tested blocks in the same Bash session. The first block installs nothing system-wide and creates no shell context outside the owned path.

set -euo pipefail

LAB=/tmp/voxfor-nats-dupe-144-lab
PORT=14244
URL="nats://127.0.0.1:${PORT}"
SERVER_VERSION=2.14.4
CLI_VERSION=0.4.0
MARKER_VALUE=voxfor-nats-dupe-window-144

[[ ! -e "$LAB" ]]
if ss -H -ltn "sport = :$PORT" | grep -q .; then
  printf 'Port %s is already listening; choose another isolated port.\n' "$PORT" >&2
  exit 1
fi
mkdir -m 700 "$LAB"
printf '%s\n' "$MARKER_VALUE" > "$LAB/.voxfor-owner"
mkdir -m 700 "$LAB/downloads" "$LAB/config" "$LAB/store"
export XDG_CONFIG_HOME="$LAB/config" NO_COLOR=1

cleanup_on_error() {
  status=$?
  if [[ $status -ne 0 && -s "$LAB/server.pid" ]]; then
    pid=$(cat "$LAB/server.pid")
    if [[ "$pid" =~ ^[0-9]+$ ]] && [[ -r "/proc/$pid/cmdline" ]] && tr '\0' ' ' < "/proc/$pid/cmdline" | grep -Fq -- "$LAB/store"; then
      kill "$pid" 2>/dev/null || true
      wait "$pid" 2>/dev/null || true
    fi
  fi
  return "$status"
}
trap cleanup_on_error EXIT

curl -fsSLo "$LAB/downloads/nats-server.tgz" \
  "https://github.com/nats-io/nats-server/releases/download/v${SERVER_VERSION}/nats-server-v${SERVER_VERSION}-linux-amd64.tar.gz"
curl -fsSLo "$LAB/downloads/nats-server-SHA256SUMS" \
  "https://github.com/nats-io/nats-server/releases/download/v${SERVER_VERSION}/SHA256SUMS"
grep " nats-server-v${SERVER_VERSION}-linux-amd64.tar.gz$" "$LAB/downloads/nats-server-SHA256SUMS" \
  | sed "s#nats-server-v${SERVER_VERSION}-linux-amd64.tar.gz#$LAB/downloads/nats-server.tgz#" \
  | sha256sum -c -
tar -xzf "$LAB/downloads/nats-server.tgz" -C "$LAB/downloads"

curl -fsSLo "$LAB/downloads/nats-cli.zip" \
  "https://github.com/nats-io/natscli/releases/download/v${CLI_VERSION}/nats-${CLI_VERSION}-linux-amd64.zip"
curl -fsSLo "$LAB/downloads/nats-cli-SHA256SUMS" \
  "https://github.com/nats-io/natscli/releases/download/v${CLI_VERSION}/SHA256SUMS"
grep " nats-${CLI_VERSION}-linux-amd64.zip$" "$LAB/downloads/nats-cli-SHA256SUMS" \
  | sed "s#nats-${CLI_VERSION}-linux-amd64.zip#$LAB/downloads/nats-cli.zip#" \
  | sha256sum -c -
unzip -q "$LAB/downloads/nats-cli.zip" -d "$LAB/downloads"

SERVER="$LAB/downloads/nats-server-v${SERVER_VERSION}-linux-amd64/nats-server"
NATS="$LAB/downloads/nats-${CLI_VERSION}-linux-amd64/nats"
[[ "$($SERVER -v)" == "nats-server: v${SERVER_VERSION}" ]]
[[ "$($NATS --version)" == "$CLI_VERSION" ]]
"$SERVER" -js -sd "$LAB/store" -a 127.0.0.1 -p "$PORT" > "$LAB/server.log" 2>&1 &
SERVER_PID=$!
printf '%s\n' "$SERVER_PID" > "$LAB/server.pid"
for _ in {1..80}; do
  grep -Fq 'Server is ready' "$LAB/server.log" && break
  kill -0 "$SERVER_PID"
  sleep 0.1
done
grep -Fq 'Server is ready' "$LAB/server.log"
printf 'LAB_READY server=%s cli=%s endpoint=%s pid=%s\n' \
  "$SERVER_VERSION" "$CLI_VERSION" "$URL" "$SERVER_PID"

The failure trap stops only a process whose command line contains the exact lab store path. It intentionally leaves failed files for inspection; the final cleanup block removes them only after the marker and process identity agree.

Create one file-backed ORDERS stream that captures orders.created, caps retained messages at 100, and remembers message IDs for three seconds. The current NATS first-stream guide shows the default two-minute value; this short duration is a laboratory instrument, not a recommendation.

"$NATS" --server "$URL" stream add ORDERS \
  --subjects 'orders.created' --storage file --retention limits \
  --replicas 1 --max-msgs 100 --dupe-window 3s --defaults > "$LAB/stream-add.txt"
"$NATS" --server "$URL" stream info ORDERS -j > "$LAB/stream-initial.json"
jq -e '.config.duplicate_window == 3000000000 and .state.messages == 0' \
  "$LAB/stream-initial.json" >/dev/null
printf 'STREAM_READY name=%s dupe_window_ns=%s messages=%s\n' \
  "$(jq -r '.config.name' "$LAB/stream-initial.json")" \
  "$(jq -r '.config.duplicate_window' "$LAB/stream-initial.json")" \
  "$(jq -r '.state.messages' "$LAB/stream-initial.json")"

Machine-readable inspection prevents a human-readable duration from hiding a different stored value. The assertion requires 3,000,000,000 nanoseconds and an empty starting stream before any publish evidence is accepted.

Read the PubAcks on both sides of expiry

Publish order-42-created, then retry immediately with the same ID but a deliberately different payload. JetStream deduplicates by message ID inside the window; it does not compare the payload and choose the “better” version. Reusing an ID for a different business event is therefore a producer bug.

"$NATS" --server "$URL" pub --jetstream orders.created \
  --header 'Nats-Msg-Id:order-42-created' \
  '{"order_id":"order-42","event":"created","attempt":1}' \
  > "$LAB/publish-first.txt" 2>&1
"$NATS" --server "$URL" pub --jetstream orders.created \
  --header 'Nats-Msg-Id:order-42-created' \
  '{"order_id":"order-42","event":"created","attempt":2}' \
  > "$LAB/publish-inside-window.txt" 2>&1
grep -Fq 'Sequence: 1' "$LAB/publish-first.txt"
grep -Fq 'Sequence: 1 Duplicate: true' "$LAB/publish-inside-window.txt"
"$NATS" --server "$URL" stream info ORDERS -j > "$LAB/stream-after-duplicate.json"
jq -e '.state.messages == 1 and .state.last_seq == 1' "$LAB/stream-after-duplicate.json" >/dev/null
printf 'INSIDE_WINDOW first_seq=1 retry_seq=1 duplicate=true stored_messages=%s\n' \
  "$(jq -r '.state.messages' "$LAB/stream-after-duplicate.json")"

Sequence 1 appears in both PubAcks, but only one stored message exists. That is the exact result a publisher wants after an acknowledgement timeout: the retry confirms the original write instead of adding another record.

A different message ID remains independent even while the first ID is still remembered:

"$NATS" --server "$URL" pub --jetstream orders.created \
  --header 'Nats-Msg-Id:order-43-created' \
  '{"order_id":"order-43","event":"created","attempt":1}' \
  > "$LAB/publish-independent-id.txt" 2>&1
grep -Fq 'Sequence: 2' "$LAB/publish-independent-id.txt"
"$NATS" --server "$URL" stream info ORDERS -j > "$LAB/stream-after-independent.json"
jq -e '.state.messages == 2 and .state.last_seq == 2' "$LAB/stream-after-independent.json" >/dev/null
printf 'INDEPENDENT_ID msg_id=order-43-created sequence=2 stored_messages=2\n'

After waiting four seconds—longer than the configured window—the original order-42-created ID is published again. A new sequence and a three-message state prove expiry changed the admission result.

sleep 4
"$NATS" --server "$URL" pub --jetstream orders.created \
  --header 'Nats-Msg-Id:order-42-created' \
  '{"order_id":"order-42","event":"created","attempt":3,"after_window":true}' \
  > "$LAB/publish-after-window.txt" 2>&1
grep -Fq 'Sequence: 3' "$LAB/publish-after-window.txt"
if grep -Fq 'Duplicate: true' "$LAB/publish-after-window.txt"; then
  printf 'Expired message ID was unexpectedly suppressed.\n' >&2
  exit 1
fi
"$NATS" --server "$URL" stream info ORDERS -j > "$LAB/stream-after-expiry.json"
jq -e '.state.messages == 3 and .state.last_seq == 3' "$LAB/stream-after-expiry.json" >/dev/null
printf 'AFTER_WINDOW msg_id=order-42-created sequence=3 duplicate=false stored_messages=3\n'

The evidence maps directly to operator decisions:

Consumer redelivery is a separate failure plane

Producer deduplication controls how many messages JetStream stores from ambiguous publishes. Consumer acknowledgement controls how many times one stored sequence may be delivered. The distinction is easy to lose when both symptoms are called “duplicates.” JetStream redelivery analysis covers AckWait, BackOff, MaxDeliver, pending work, and poison-message evidence in depth.

This control creates a pull consumer with a one-second acknowledgement wait. It reads sequence 1 without acknowledging, waits two seconds, then receives and acknowledges the same stored sequence a second time.

"$NATS" --server "$URL" consumer add ORDERS RETRY-CHECK \
  --filter orders.created --pull --ack explicit --wait 1s \
  --max-deliver 2 --deliver all --replay instant --defaults \
  > "$LAB/consumer-add.txt"
"$NATS" --server "$URL" consumer next ORDERS RETRY-CHECK \
  --count 1 --no-ack --wait 2s > "$LAB/delivery-first.txt" 2>&1
sleep 2
"$NATS" --server "$URL" consumer next ORDERS RETRY-CHECK \
  --count 1 --ack --wait 2s > "$LAB/delivery-second.txt" 2>&1
grep -Eq '/ tries: 1 / .* / str seq: 1 /' "$LAB/delivery-first.txt"
grep -Eq '/ tries: 2 / .* / str seq: 1 /' "$LAB/delivery-second.txt"
printf 'CONSUMER_REDELIVERY stream_sequence=1 delivery_counts=1,2 stored_messages=3\n'

One stored sequence was delivered twice. Business code must therefore make its effect idempotent or reconcile it transactionally even when producer-side deduplication is configured. Retry reconciliation controls demonstrate the broader rule: an infrastructure acknowledgement cannot prove that an external payment, email, deployment, or database mutation happened exactly once.

Queue deployment also needs a shared state and worker-ownership contract. n8n queue-mode acceptance is a useful adjacent example of proving producer, broker, worker, and application handoff separately rather than treating one healthy process as end-to-end success.

Convert the experiment into production admission

Final lab assertions re-read both stream and consumer state. They require the three-second configuration, three stored messages, last sequence 3, two deliveries of stream sequence 1, an acknowledgement floor at that sequence, and no pending acknowledgement for the control.

"$NATS" --server "$URL" stream info ORDERS -j > "$LAB/final-stream.json"
"$NATS" --server "$URL" consumer info ORDERS RETRY-CHECK -j > "$LAB/final-consumer.json"
jq -e '.config.duplicate_window == 3000000000 and .state.messages == 3 and .state.last_seq == 3' \
  "$LAB/final-stream.json" >/dev/null
jq -e '.delivered.consumer_seq == 2 and .delivered.stream_seq == 1 and .ack_floor.stream_seq == 1 and .num_ack_pending == 0' \
  "$LAB/final-consumer.json" >/dev/null
printf 'ACCEPTANCE window=3s inside_duplicate=true after_expiry_stored=true distinct_id_stored=true consumer_redelivery_independent=true messages=3 last_seq=3\n'

Representative output from the completed run is below. The process ID is local evidence only and will differ on another host.

/tmp/voxfor-nats-dupe-144-lab/downloads/nats-server.tgz: OK
/tmp/voxfor-nats-dupe-144-lab/downloads/nats-cli.zip: OK
LAB_READY server=2.14.4 cli=0.4.0 endpoint=nats://127.0.0.1:14244 pid=1625214
STREAM_READY name=ORDERS dupe_window_ns=3000000000 messages=0
INSIDE_WINDOW first_seq=1 retry_seq=1 duplicate=true stored_messages=1
INDEPENDENT_ID msg_id=order-43-created sequence=2 stored_messages=2
AFTER_WINDOW msg_id=order-42-created sequence=3 duplicate=false stored_messages=3
CONSUMER_REDELIVERY stream_sequence=1 delivery_counts=1,2 stored_messages=3
ACCEPTANCE window=3s inside_duplicate=true after_expiry_stored=true distinct_id_stored=true consumer_redelivery_independent=true messages=3 last_seq=3
CLEANUP=complete path=/tmp/voxfor-nats-dupe-144-lab listener=absent

The window is understood only when the configured nanoseconds equal the reviewed duration, an immediate same-ID retry returns the original sequence with duplicate: true, a distinct ID stores independently, the same original ID receives a new sequence after expiry, and consumer redelivery repeats one stored sequence without changing the stream’s message count. Production admission additionally requires a measured retry envelope, resource evidence, stable ID generation, business-side idempotency, and recovery ownership.

For a self-hosted deployment, a single-node experiment can graduate to root-controlled VPS capacity when measured message rate, retention, disk, memory, and recovery ownership fit one host. The live page offers selectable CPU, RAM, disk, bandwidth, operating systems, and server control. Clustering changes failure tolerance and replica cost; it does not make an expired message ID permanent.

Resource alarms must also be interpreted at the correct layer. RabbitMQ disk-alarm recovery covers a different broker and failure mode, yet its useful principle transfers: prove whether admission, storage, or delivery owns the stall before changing queue policy. Broader self-hosted application responsibility mapping helps assign patching, monitoring, backups, and incident response when NATS is one component of a larger stack.

Roll back policy without replaying uncertainty

Changing a window modifies future duplicate detection; it does not remove business effects already caused by a late retry. Avoid combining a configuration rollback with a blind replay.

If a canary shows that legitimate retries outlive the proposed window, stop the affected publisher rollout, preserve PubAcks and retry timestamps, restore the previously reviewed stream configuration through the normal change path, and reconcile every ambiguous business key before replay. If the new duration causes unacceptable memory or latency, shorten it only after the publisher retry envelope and outbox policy are changed together; keep consumer idempotency active throughout.

Alternative stream designs require their own evidence. NATS describes discard-new-per-subject as a way to preserve one message per subject in an infinite deduplication pattern. That can fit workloads where a unique business key belongs safely in the subject and retention policy. It is not a drop-in replacement for arbitrary message IDs, multi-event subjects, consumer idempotency, or permanent exactly-once side effects.

After retaining the secret-free receipt, stop only the recorded server. The cleanup checks its command line for both the exact store path and port before sending a signal, confirms the listener is gone, validates the ownership marker, deletes only descendants of the guarded lab, and removes the empty directory.

PID_FROM_FILE=$(cat "$LAB/server.pid")
[[ "$PID_FROM_FILE" =~ ^[0-9]+$ ]]
[[ -r "/proc/$PID_FROM_FILE/cmdline" ]]
tr '\0' ' ' < "/proc/$PID_FROM_FILE/cmdline" | grep -Fq -- "$LAB/store"
tr '\0' ' ' < "/proc/$PID_FROM_FILE/cmdline" | grep -Fq -- "-p $PORT"
kill "$PID_FROM_FILE"
wait "$PID_FROM_FILE"
if ss -H -ltn "sport = :$PORT" | grep -q .; then
  printf 'Owned listener still exists after stop.\n' >&2
  exit 1
fi
[[ "$(cat "$LAB/.voxfor-owner")" == "$MARKER_VALUE" ]]
find "$LAB" -depth -mindepth 1 -delete
rmdir "$LAB"
trap - EXIT
printf 'CLEANUP=complete path=%s listener=absent\n' "$LAB"

Keep a receipt that can reject the next rollout

A useful release record names server and CLI versions, exact stream, duplicate-window nanoseconds, publisher timeout and retry schedule, maximum observed retry age, unique-ID rate, PubAck stream/sequence/duplicate fields, inside-window and after-expiry controls, consumer redelivery evidence, business idempotency owner, resource measurements, change reference, and rollback result.

The rule is compact: the duplicate window must outlive every retry it is expected to suppress, while the application remains safe when a retry arrives later anyway. If either half lacks evidence, the design is not ready for an exactly-once claim.