MySQL Replica Lag: Separate Receive Delay from Apply Delay

Two replicas can both report Seconds_Behind_Source: 0 while only one is current. On the healthy replica, the receiver has fetched the source’s latest transaction and the applier has committed it. On the stale replica, a slow receiver may be late to the source while the applier is merely caught up to the last event it received. One lag value cannot prove both stages are healthy.

Treat MySQL replication as a short pipeline: the receiver copies binary-log events into the relay log, then an applier or coordinator-and-worker set executes those transactions. A useful incident response samples both ends over time, names the slow stage, and changes only the resource or workload that owns it. More workers, a restart, or a skipped transaction is not a diagnosis.

The commands below target self-managed MySQL 8.4 terminology. MySQL 8.0 builds may expose fewer timestamp columns, while MariaDB and managed database services have different controls. Confirm the product and version before adapting any change.

One lag number hides three clocks

MySQL’s SHOW REPLICA STATUS documentation explains an important boundary: Seconds_Behind_Source compares replica time with the source timestamp of the event currently being processed. When network receipt is slow, the applier can report zero because it has consumed everything locally available even though newer source events have not reached the relay log.

Three questions therefore belong in every sample:

• Is the receiver connected and advancing toward the source?
• Is the relay backlog growing or shrinking between receipt and execution?
• Are the applier workers committing transactions without errors, lock waits or resource starvation?

A stopped receiver or applier is not ordinary lag. It is replication failure with a recorded error or disabled thread, and the failing stage must be repaired before catch-up speed matters. MySQL preserves useful error fields in status and Performance Schema; capture them before any reset erases context.

Sustained write streams that require predictable CPU and storage may eventually justify dedicated server capacity, but infrastructure is a conclusion, not the first guess. The same symptoms can come from one large transaction, schema drift, a lock, network interruption or insufficient parallelism.

Capture a time series before restarting anything

A single screenshot freezes a moving queue. Collect several samples across at least one representative busy interval so direction becomes visible. Record UTC time, channel name, receiver/applier state, errors, relay-log space, retrieved and executed GTID sets, and the application symptom that triggered the investigation.

Begin with the channel’s public status

Run the status statement on the replica through an administrative MySQL session:

SHOW REPLICA STATUS;

Keep these fields together: Replica_IO_Running, Replica_SQL_Running, Seconds_Behind_Source, Relay_Log_Space, Retrieved_Gtid_Set, Executed_Gtid_Set, Last_IO_Error and Last_SQL_Error. For multiple channels, capture each channel separately rather than mixing progress into one note.

The replication-status guide warns that individual values can jump while a long or old event is being processed. Five samples showing direction are more useful than one dramatic number.

Add connection and worker evidence

Performance Schema exposes the receiving side without requiring log parsing:

SELECT CHANNEL_NAME, SERVICE_STATE, LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE,
       LAST_QUEUED_TRANSACTION
FROM performance_schema.replication_connection_status;

For a multithreaded replica, inspect each applier worker:

SELECT CHANNEL_NAME, WORKER_ID, SERVICE_STATE,
       LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE,
       LAST_APPLIED_TRANSACTION, APPLYING_TRANSACTION
FROM performance_schema.replication_applier_status_by_worker
ORDER BY CHANNEL_NAME, WORKER_ID;

MySQL documents the connection table as the receipt/queue surface and the worker table as the per-worker apply surface. Empty transaction fields may simply mean a worker is idle; combine them with service state, relay-log growth and repeated samples.

Receiver delay lives before the relay log

When the receiver does not advance, changing applier workers cannot help. Check Last_IO_Error, source reachability, DNS, routing, firewall/TLS behavior and whether the source still retains the binary logs the replica needs. MySQL’s PURGE BINARY LOGS reference states that expiration should not be lower than the maximum period a replica might lag.

Network throughput matters most when the receiver is actually transferring a sustained binlog stream. Compare source write volume with link capacity and packet loss, but do not confuse distance alone with the cause. A quiet source should not require high bandwidth, and a fast link cannot repair rejected credentials or purged coordinates.

Receiver repair ends when repeated samples show the last queued transaction moving again and no new I/O error. Catch-up may still be incomplete because the relay log now contains accumulated work; continue into the applier branch before declaring recovery.

Apply delay grows inside the replica

Relay-log growth with an advancing receiver proves that work arrives faster than it commits. Start with transaction shape and waits before sizing hardware. One hours-long DDL statement, a bulk update, missing replica-side schema, row-lock contention or commit-order serialization can hold progress even when several workers exist.

Find the resource that limits useful parallelism

Capture system evidence during the same interval as MySQL status:

date -u
vmstat 1 10
iostat -xz 1 10

iostat belongs to the optional sysstat package on many distributions; do not install it during an incident without normal change approval. CPU run queue, steal time, swap activity, device utilization and latency help separate database work from host contention. Before increasing MySQL concurrency, use CPU steal-time evidence to exclude a virtualization boundary the guest cannot tune away.

Map a persistent bottleneck to dedicated hardware components only after measurements identify CPU, memory, NVMe or network as the owner. Buying faster storage for a lock-bound applier or more cores for one giant transaction leaves the queue model unchanged.

Treat worker count as a controlled experiment

MySQL 8.4 documents a default replica_parallel_workers value of four and deprecates zero. Current configuration is easy to read:

SHOW VARIABLES LIKE 'replica_parallel_workers';
SHOW VARIABLES LIKE 'replica_preserve_commit_order';

Do not copy a worker count from another server. Record baseline apply rate, CPU, I/O latency, lock/commit-order waits and memory; test one bounded change through the supported configuration path; then compare catch-up slope. Rollback if resource pressure rises without a material increase in executed progress.

Source-side relief can be safer than aggressive replica tuning. Pause an approved bulk job, reduce batch size, move maintenance outside peak writes or temporarily stop routing analytical reads to the lagging node. Never throttle business writes blindly; name the workload owner and acceptance boundary first.

Decide whether to throttle, tune, or rebuild

Choose the smallest action that matches the proven owner:

• Throttle source workload when an identifiable batch or migration produces more write volume than the replica can absorb and the business permits a slower producer.
• Tune the replica when worker evidence and system headroom show usable parallelism or a specific storage/CPU constraint with a reversible remedy.
• Repair data or schema when the applier stopped on a concrete error. Do not classify divergence as capacity lag.
• Rebuild from a trusted copy when required history is unavailable, consistency cannot be proved, or catch-up would exceed the recovery objective.

MySQL’s Clone plugin can transfer data and replication coordinates when versions, privileges and topology fit. Other backup/restore methods may be correct for the estate. Stage replacement capacity through cloud VPS infrastructure only after defining private connectivity, source compatibility, storage headroom and acceptance tests; a fresh host is not automatically a valid replica.

Transaction skipping deserves a separate approval record. MySQL documents non-GTID and GTID methods, but a successful restart proves only that the applier moved past an event. It does not prove table contents still match. Unknown divergence makes reseeding safer than silent continuation.

Verify catch-up without routing stale reads

Keep the replica removed from read traffic until several signals agree: both threads are running, errors stay empty, receiver and applied transaction progress advance, relay-log space stops growing, the lag budget is met, and a representative read returns expected current data. A service ping cannot prove replication freshness.

Application acceptance should use a harmless source write with a unique marker, followed by a timed replica read through the same connection policy the application will use. Record commit UTC, first visible replica UTC and the selected endpoint. If read routing or proxy caches can choose another node, identify the actual backend so the test cannot pass against the source by accident.

Private Uptime Kuma monitoring can verify an external application read path after routing resumes, while database-native metrics continue to measure receiver and applier progress. Neither view replaces the other.

Close with an explicit lag budget

“Caught up” is temporary unless the team defines how much delay the workload tolerates and how long recovery may take. Alert separately on thread errors, receiver staleness, apply delay, relay-log growth and remaining source-binlog runway. A zero seconds value should never silence a stale receiver alarm.

Save the incident series, source write rate, receiver/apply owner, chosen action, rollback boundary, marker-read result and post-change resource envelope. That evidence lets the next operator distinguish a recurring workload shape from infrastructure regression.

Continue through Voxfor database operations guides for adjacent capacity and recovery workflows. MySQL replica recovery is complete when receipt is current, apply is current, data is trusted and the real read path proves freshness.