Follow Late Writes Through a Meilisearch Index Swap

Meilisearch can exchange two index identities atomically. It cannot decide what your application should do with writes that arrive after the replacement index is ready. In an isolated Meilisearch v1.53.1 run, a deliberately late document with ID 9 completed on the public catalog index before the swap task was submitted. After the swap, the new public catalog contained candidate IDs 1, 2 and 3; ID 9 stayed with the old physical index, now reachable as catalog_next.

That result does not make the swap unsafe. It defines the boundary: the UID exchange is atomic, but write reconciliation is a separate application responsibility. The procedure below makes that boundary observable, forces an invalid swap to fail without changing state, and reverses the valid pair before either index is deleted.

Read the Swap Receipt as an Identity Map

An index UID in Meilisearch is the name clients use, such as catalog. A replacement index can be built under a temporary UID, such as catalog_next, while searches continue against catalog. The official zero-downtime deployment workflow then swaps the two indexes so the rebuilt data appears under the stable public UID.

Official index documentation also favors explicit index creation when settings or primary-key choices must be controlled before documents arrive. That is the safer candidate-building boundary used here.

“Swap” matters. Meilisearch does not copy every document during this operation. The swap-index API specification describes one asynchronous global task that exchanges the indexes associated with the supplied UIDs. Documents, settings and task history move with their index identity.

Four lab states make the distinction concrete:

Take a recovery artifact before a production rebuild. Meilisearch snapshots and dumps solve different recovery and migration needs; neither should be improvised after a bad cutover. Keeping the old index available is the fast application-level reversal, while a tested snapshot or dump covers a wider failure boundary.

Build Two Deliberately Different Indexes

This test uses the exact Meilisearch v1.53.1 release for Linux amd64, whose SHA-256 is pinned before execution. It binds to 127.0.0.1:27700, runs as nobody, disables analytics and caps indexing at 256MiB on one thread. Those limits keep the synthetic run bounded; they are not production sizing advice.

Ownership through a marker-owned directory and an unused-port check prevent the lab from colliding with an existing service. The full script also verifies the process executable before stopping it and removes only the directory carrying its ownership marker.

set -euo pipefail

VERSION=1.53.1
EXPECTED_SHA256=cd8e446b29cefe44cdbc872ffb2de906ada165f4b96a33bb9e1a706b1e9279a0
MEILI_PORT=27700
MEILI_URL="http://127.0.0.1:$MEILI_PORT"
MEILI_LAB=$(mktemp -d /tmp/voxfor-meili-swap.XXXXXX)

if ss -H -ltn "sport = :$MEILI_PORT" | grep -q .; then
  echo "Loopback port $MEILI_PORT is already in use" >&2
  exit 1
fi

touch "$MEILI_LAB/.voxfor-owned"
curl -fsSL \
  "https://github.com/meilisearch/meilisearch/releases/download/v$VERSION/meilisearch-linux-amd64" \
  -o "$MEILI_LAB/meilisearch"
chmod 0755 "$MEILI_LAB/meilisearch"

ACTUAL_SHA256=$(sha256sum "$MEILI_LAB/meilisearch" | awk '{print $1}')
test "$ACTUAL_SHA256" = "$EXPECTED_SHA256"
test "$("$MEILI_LAB/meilisearch" --version)" = "meilisearch $VERSION"

chown -R nobody:nogroup "$MEILI_LAB"
runuser -u nobody -- "$MEILI_LAB/meilisearch" \
  --db-path "$MEILI_LAB/data.ms" \
  --dump-dir "$MEILI_LAB/dumps" \
  --http-addr "127.0.0.1:$MEILI_PORT" \
  --env development \
  --no-analytics \
  --max-indexing-memory 256MiB \
  --max-indexing-threads 1 \
  --log-level WARN >"$MEILI_LAB/server.log" 2>&1 &
MEILI_WRAPPER_PID=$!

for attempt in $(seq 1 100); do
  curl -fsS "$MEILI_URL/health" >/dev/null 2>&1 && break
  sleep 0.1
done
curl -fsS "$MEILI_URL/health" | jq -e '.status == "available"'
curl -fsS "$MEILI_URL/version" |
  jq -e --arg version "$VERSION" '.pkgVersion == $version'

Meilisearch writes are asynchronous. HTTP 202 means a task was accepted, not that indexing or swapping finished. A reusable waiter must retrieve the returned taskUid, poll to a terminal state, and let the caller require succeeded or the expected failure.

wait_task() {
  task_uid=$1

  for attempt in $(seq 1 200); do
    task=$(curl -fsS "$MEILI_URL/tasks/$task_uid")
    status=$(jq -r '.status' <<<"$task")

    case "$status" in
      succeeded|failed|canceled)
        printf '%s\n' "$task"
        return 0
        ;;
    esac
    sleep 0.05
  done

  echo "Task $task_uid did not finish" >&2
  return 1
}

submit_and_wait() {
  method=$1
  path=$2
  body=$3

  accepted=$(curl -fsS -X "$method" "$MEILI_URL$path" \
    -H 'Content-Type: application/json' \
    --data-binary "$body")
  task_uid=$(jq -r '.taskUid' <<<"$accepted")
  test "$task_uid" != null
  wait_task "$task_uid"
}

Now make the indexes impossible to confuse. The live generation has IDs 1 and 2 and one filterable attribute. The candidate has IDs 1, 2 and 3, different titles, a color field and two filterable attributes.

submit_and_wait POST /indexes \
  '{"uid":"catalog","primaryKey":"id"}' |
  jq -e '.status == "succeeded"'

submit_and_wait POST /indexes/catalog/documents \
  '[{"id":1,"title":"Blue mug","generation":"live-v1"},
    {"id":2,"title":"Red mug","generation":"live-v1"}]' |
  jq -e '.status == "succeeded"'

submit_and_wait PATCH /indexes/catalog/settings \
  '{"filterableAttributes":["generation"]}' |
  jq -e '.status == "succeeded"'

submit_and_wait POST /indexes \
  '{"uid":"catalog_next","primaryKey":"id"}' |
  jq -e '.status == "succeeded"'

submit_and_wait POST /indexes/catalog_next/documents \
  '[{"id":1,"title":"Blue mug improved","generation":"candidate-v2","color":"blue"},
    {"id":2,"title":"Red mug improved","generation":"candidate-v2","color":"red"},
    {"id":3,"title":"Green mug","generation":"candidate-v2","color":"green"}]' |
  jq -e '.status == "succeeded"'

submit_and_wait PATCH /indexes/catalog_next/settings \
  '{"filterableAttributes":["generation","color"]}' |
  jq -e '.status == "succeeded"'

Before cutover, inspect the replacement through the same API surface the application depends on. A count alone is weak: three wrong documents can still satisfy a count of three. Check representative IDs, generation markers, settings and searches relevant to the real application.

LIVE_BEFORE=$(curl -fsS "$MEILI_URL/indexes/catalog/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

CANDIDATE_BEFORE=$(curl -fsS "$MEILI_URL/indexes/catalog_next/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

CANDIDATE_SETTINGS=$(curl -fsS "$MEILI_URL/indexes/catalog_next/settings" |
  jq -c '.filterableAttributes')

jq -e 'map(.id) == [1,2]' <<<"$LIVE_BEFORE"
jq -e 'map(.id) == [1,2,3] and all(.generation == "candidate-v2")' \
  <<<"$CANDIDATE_BEFORE"
jq -e '. == ["generation","color"]' <<<"$CANDIDATE_SETTINGS"

Parallel rebuilds temporarily need storage for both index states and working headroom. Use a latency-aware VPS disk benchmark if storage behavior is uncertain; capacity and latency under rebuild load matter more than a headline IOPS number. For a new self-hosted deployment, size CPU, RAM and storage headroom before the rebuild; Voxfor’s VPS hosting plans expose those resource and operating-system choices, while cutover acceptance still comes from your own workload.

Keep the administrative API private in production. Bind it to a trusted interface, require a master key, and use private administrative access rather than exposing a maintenance endpoint to the public internet. Development mode without a key exists only inside this disposable loopback lab.

Put One Write Between Candidate Readiness and Swap

Instead of another seed document, the critical experiment. It is a write whose ordering is explicit:

1. Candidate verification finishes.
2. ID 9 is submitted to the live public UID.
3. Its document-update task reaches succeeded.
4. Only then is the swap task enqueued.

LATE_TASK=$(submit_and_wait POST /indexes/catalog/documents \
  '[{"id":9,"title":"Late amber mug","generation":"late-live-v1"}]')

jq -e \
  '.status == "succeeded" and .type == "documentAdditionOrUpdate"' \
  <<<"$LATE_TASK"

LIVE_WITH_LATE_WRITE=$(curl -fsS \
  "$MEILI_URL/indexes/catalog/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

jq -e \
  'map(.id) == [1,2,9] and any(.id == 9 and .generation == "late-live-v1")' \
  <<<"$LIVE_WITH_LATE_WRITE"

This ordering proves one narrow claim: a completed pre-swap write belongs to the old physical index. It does not model a request racing at the same nanosecond as task creation. The specification adds another useful rule: tasks enqueued after the index-swap task against a UID are processed against the index that owns that UID after the swap. Production designs should still establish a clean write boundary rather than depend on a guessed scheduling race.

Swap the UIDs and Find ID 9

Submit one pair and retain the asynchronous receipt. A valid request returns a global indexSwap task with no single indexUid. Wait for that task to succeed before directing validation traffic to the result.

SWAP_ACCEPTED=$(curl -fsS -X POST "$MEILI_URL/swap-indexes" \
  -H 'Content-Type: application/json' \
  --data-binary '[{"indexes":["catalog","catalog_next"]}]')

jq -e \
  '.status == "enqueued" and .type == "indexSwap" and .indexUid == null' \
  <<<"$SWAP_ACCEPTED"

SWAP_TASK=$(wait_task "$(jq -r '.taskUid' <<<"$SWAP_ACCEPTED")")
jq -e '.status == "succeeded" and .type == "indexSwap"' <<<"$SWAP_TASK"

PUBLIC_AFTER=$(curl -fsS "$MEILI_URL/indexes/catalog/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

OLD_AFTER=$(curl -fsS "$MEILI_URL/indexes/catalog_next/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

PUBLIC_SETTINGS=$(curl -fsS "$MEILI_URL/indexes/catalog/settings" |
  jq -c '.filterableAttributes')
OLD_SETTINGS=$(curl -fsS "$MEILI_URL/indexes/catalog_next/settings" |
  jq -c '.filterableAttributes')

jq -e 'map(.id) == [1,2,3] and all(.generation == "candidate-v2")' \
  <<<"$PUBLIC_AFTER"
jq -e 'map(.id) == [1,2,9] and any(.id == 9)' <<<"$OLD_AFTER"
jq -e '. == ["generation","color"]' <<<"$PUBLIC_SETTINGS"
jq -e '. == ["generation"]' <<<"$OLD_SETTINGS"

Post-swap, the public UID exposes the candidate documents and settings. ID 9 is not there because no mechanism replayed it into the candidate. The old live object was not destroyed; it moved behind catalog_next, taking ID 9 and its original settings with it.

Representative output from the isolated run:

{
  "swapTask": {"uid": 7, "type": "indexSwap", "status": "succeeded"},
  "afterSwap": {
    "catalog": {"ids": [1, 2, 3], "filterableAttributes": ["generation", "color"]},
    "catalog_next": {"ids": [1, 2, 9], "filterableAttributes": ["generation"]}
  },
  "lateWriteProof": {
    "documentId": 9,
    "publicContainsLateWrite": false,
    "oldIndexContainsLateWrite": true
  },
  "missingIndexControl": {
    "uid": 8,
    "status": "failed",
    "code": "index_not_found",
    "stateHashUnchanged": "bcc6c40197acd7a497abeef0897e7b9e6aa1a5b062b700d4512ce64831c05feb"
  },
  "rollback": {"uid": 9, "status": "succeeded", "exactBaselineRestored": true},
  "cleanup": {"listenerClosed": true, "ownedDirectoryRemoved": true}
}

A production cutover is admissible only when the candidate’s schema, settings, counts and representative searches pass; every build and swap task is terminal-successful; writes in the gap are paused, dual-written or replayed; reconciliation finds no missing late records; clients still use the public UID; application health remains good; and the old index stays available for the declared rollback window.

Make a Missing Pair Fail Without Changing State

One happy-path swap shows the mechanism, not failure containment. The negative control names a nonexistent candidate after the successful cutover. The API accepts an asynchronous task, that task finishes as failed with index_not_found, and a hash of the two visible document arrays remains identical.

STATE_HASH_BEFORE=$(printf '%s\n%s\n' "$PUBLIC_AFTER" "$OLD_AFTER" |
  sha256sum | awk '{print $1}')

BAD_ACCEPTED=$(curl -fsS -X POST "$MEILI_URL/swap-indexes" \
  -H 'Content-Type: application/json' \
  --data-binary '[{"indexes":["catalog","missing_candidate"]}]')

BAD_TASK=$(wait_task "$(jq -r '.taskUid' <<<"$BAD_ACCEPTED")")
jq -e \
  '.status == "failed" and .type == "indexSwap" and
   .error.code == "index_not_found"' <<<"$BAD_TASK"

PUBLIC_AFTER_BAD=$(curl -fsS "$MEILI_URL/indexes/catalog/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')
OLD_AFTER_BAD=$(curl -fsS "$MEILI_URL/indexes/catalog_next/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

STATE_HASH_AFTER=$(printf '%s\n%s\n' "$PUBLIC_AFTER_BAD" "$OLD_AFTER_BAD" |
  sha256sum | awk '{print $1}')

test "$STATE_HASH_AFTER" = "$STATE_HASH_BEFORE"
printf 'cutover_state_sha256=%s\n' "$STATE_HASH_AFTER"

According to the specification, a multi-pair request is atomic: if one pair is invalid, none of the requested swaps is applied. The lab uses one valid-looking UID plus one missing UID and then reads both states again. For a production multi-pair cutover, validate every UID and uniqueness constraint before submission, then retain the global task’s full error object rather than recording only the initial 202 response.

Choose a Write-Gap Control

ID 9 disappears from the new public view on purpose. Production must close that gap through the application’s actual write path.

For a low write rate, a bounded pause is often the simplest and the application already has maintenance-mode behavior. Dual write can reduce the pause but creates partial-failure handling: one accepted task and one failed task cannot be treated as success. Replay scales better when a durable change source already exists; inventing a log during the migration is a separate project.

One authoritative write history is the essential invariant. The ownership reasoning resembles a single-writer migration plan: a reversible cutover is easier when the system can name exactly where writes are accepted at each stage. For another example of building a parallel candidate and retaining a reversal path, see this parallel cutover plan. Those patterns are analogous; they do not change Meilisearch’s own task semantics.

Do not delete catalog_next immediately after the successful swap. It is the fastest exact reversal of the old search state. First close the late-write reconciliation, run representative queries through the application, verify settings and synonyms, watch error and latency signals, and record the task UID that admitted the cutover.

Reverse the Pair Before Deleting Either Index

Reversal uses the same pair, waits for another successful global task, and compares both arrays with the exact pre-swap baselines. Cleanup then verifies the owned executable, closes the loopback listener and removes only the marker-owned directory.

ROLLBACK_ACCEPTED=$(curl -fsS -X POST "$MEILI_URL/swap-indexes" \
  -H 'Content-Type: application/json' \
  --data-binary '[{"indexes":["catalog","catalog_next"]}]')

ROLLBACK_TASK=$(wait_task "$(jq -r '.taskUid' <<<"$ROLLBACK_ACCEPTED")")
jq -e '.status == "succeeded" and .type == "indexSwap"' <<<"$ROLLBACK_TASK"

RESTORED_PUBLIC=$(curl -fsS "$MEILI_URL/indexes/catalog/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')
RESTORED_CANDIDATE=$(curl -fsS \
  "$MEILI_URL/indexes/catalog_next/documents?limit=100" |
  jq -c '[.results[] | {id,generation,color:(.color // null)}] | sort_by(.id)')

test "$RESTORED_PUBLIC" = "$LIVE_WITH_LATE_WRITE"
test "$RESTORED_CANDIDATE" = "$CANDIDATE_BEFORE"

MEILI_SERVER_PID=$(pgrep -P "$MEILI_WRAPPER_PID" -f "$MEILI_LAB/meilisearch" |
  head -n 1)
test "$(readlink -f "/proc/$MEILI_SERVER_PID/exe")" = "$MEILI_LAB/meilisearch"
kill "$MEILI_SERVER_PID"
wait "$MEILI_WRAPPER_PID" 2>/dev/null || true

test -f "$MEILI_LAB/.voxfor-owned"
test "$MEILI_LAB" != /tmp
case "$MEILI_LAB" in /tmp/voxfor-meili-swap.*) ;; *) exit 1 ;; esac
rm -rf -- "$MEILI_LAB"

! curl -fsS "$MEILI_URL/health" >/dev/null 2>&1
test ! -e "$MEILI_LAB"

If document parity, settings, representative search, task status, write-watermark reconciliation or application health misses its threshold, reverse the pair while both indexes still exist. Wait for the rollback task, verify the original public generation, keep the task/error receipt, and investigate before attempting another cutover. The safe endpoint is not merely a successful swap task; it is one authoritative write history, a verified public UID and an old index that remains recoverable until the decision is closed.