Linux RAID Recovery Speed Has a Production Latency Budget

Two clocks run during Linux md RAID recovery: the array must regain redundancy, while production requests still have latency objectives. The safe target is not the highest rebuild number. It is the fastest rate that keeps workload latency inside an agreed budget without leaving the array degraded longer than necessary.

Copying a target from another server ignores media type, RAID level, controller path, filesystem, foreground I/O and service-level objectives. Linux exposes per-array controls for this reason. An operator should first identify the active md operation, capture workload and member-device evidence, adjust one bound in small steps, and keep a tested rollback value.

Scope boundary: this procedure applies to software md arrays the operating system can see. Workloads that need direct control over disks and recovery policy should begin with dedicated-server storage planning, passthrough devices or attached volumes explicitly assembled inside the guest. A normal VPS usually cannot tune RAID running below its virtual disk.

Name the Operation Before Tuning It

/proc/mdstat is a useful first view, but the kernel’s md sysfs files provide the unambiguous state. According to Linux kernel RAID documentation, sync_action can report resync, recover, check, repair or idle.

• recover builds a spare or replacement member after a device is missing or failed.
• resync regenerates redundancy after creation or an unclean state.
• check reads and compares redundancy information.
• repair performs a requested check-and-repair pass without using the write-intent bitmap as an optimization.

Those operations do not carry the same risk. A foreground latency spike during a scheduled check may justify pausing or rescheduling the check. A degraded recovery has a different boundary: slowing it extends the period in which another member failure may become consequential.

Capture the state without changing anything:

mdadm --detail /dev/md0
cat /sys/block/md0/md/sync_action
cat /sys/block/md0/md/degraded
cat /sys/block/md0/md/sync_completed
cat /sys/block/md0/md/sync_speed

sync_completed reports completed sectors over the possible total. sync_speed is the current actual speed averaged over the previous 30 seconds, so it should not be treated as an instantaneous sample. Save the output with a UTC timestamp and array UUID from mdadm --detail; device names alone may not be durable enough for a shift handoff.

Confirm who owns the physical queue

Before changing md values, verify that /sys/block/md0/md/ exists and that member devices appear under /sys/block/md0/slaves/. If the guest sees only one provider virtual disk, the recovery queue is probably outside guest control. KVM and hypervisor storage boundaries explain why a virtual disk can hide host RAID geometry and member-device pressure.

Hardware RAID has another control plane entirely. Use controller tooling and vendor documentation rather than writing Linux md attributes to an unrelated device. Likewise, cloud volume replication is governed by the provider even when Linux exposes an ordinary block device.

Build a Baseline Around the Workload

Recovery throughput alone cannot tell whether users are being harmed. Pick one application signal that represents the service: database transaction latency, API p95 or p99, job duration, VM storage latency, queue age, or filesystem operation time. Record its normal range under comparable traffic before calling any md limit safe.

At the block layer, iostat provides device throughput, request latency and queue depth. Its current manual page defines await as queue plus service time and aqu-sz as average queue length. It also warns that %util does not represent the performance limit of parallel devices such as RAID arrays and modern SSDs. Do not accept a rebuild merely because %util stays below 100%.

Sample several intervals and ignore the since-boot aggregate:

iostat -xmy 5 7

Read both /dev/md0 and its member devices. The md device shows the logical workload, while members expose an overloaded disk, asymmetric latency or a replacement drive that cannot sustain the requested rate. A single five-second sample is too noisy for a limit decision; use a window long enough to include normal request bursts.

Freeze an evidence row

For each test step, preserve the same fields:

Use baseline values, not generic thresholds. A database with strict commit latency can fail its objective while a bulk file server remains healthy at the same queue depth.

Separate Global Defaults From One Array

Linux exposes system-wide defaults at /proc/sys/dev/raid/speed_limit_min and speed_limit_max. Per-array files live under /sys/block/md0/md/sync_speed_min and sync_speed_max. The current md manual defines the goal in kibibytes per second per device, not as aggregate array throughput; an array with more members can therefore move more total data for the same target. Reading a per-array file also shows whether its active value is (local) or inherited from (system).

Inspect both scopes before making a change:

sysctl dev.raid.speed_limit_min dev.raid.speed_limit_max
cat /sys/block/md0/md/sync_speed_min
cat /sys/block/md0/md/sync_speed_max

Changing global sysctls during one incident can affect every active md array on the host. Prefer a per-array limit when only one recovery needs calibration. Preserve the complete original text, including whether it said (system) or (local), because rollback differs.

Understand minimum and maximum

sync_speed_max caps how quickly the operation may run when the storage path has capacity. sync_speed_min influences the minimum recovery effort when foreground I/O competes for the devices. Neither value promises an exact observed rate; device capability and workload contention still matter.

Avoid starting with a higher minimum to make a slow recovery move faster. Raising the floor can force more recovery I/O into busy periods. For production-latency incidents, the first experiment is usually a bounded per-array maximum, followed by observation of both application and member-device metrics.

Calibrate One Bound in Measured Steps

Choose a proposed maximum from current evidence, not from a tutorial’s universal number. If the active maximum is 180000 KiB/s while application latency is unacceptable, a first step might test 140000 rather than dropping immediately to a tiny value. The exact numbers are examples; the ratio and measured response are what matter.

Apply one per-array value:

echo 140000 | sudo tee /sys/block/md0/md/sync_speed_max

Hold the test for at least one representative application window and long enough for the 30-second md speed average to settle. During that window, record application latency, member await/aqu-sz, sync_speed, progress and estimated completion time.

Accept, step again, or roll back

Use three outcomes:

1. Latency returns inside budget and progress remains credible: keep the temporary value for the current traffic band.
2. Latency is still outside budget but storage errors are absent: reduce the maximum another measured step, then repeat the full window.
3. Latency does not improve or recovery behaves unexpectedly: restore the previous setting and investigate another bottleneck rather than continuing to squeeze md.

Restore system inheritance when the array originally used global defaults:

echo system | sudo tee /sys/block/md0/md/sync_speed_max

If the original state was a local number, write that exact number back. Do not persist the temporary value yet. A limit accepted during peak traffic may be unnecessarily slow overnight, while an off-peak limit may violate daytime objectives.

Do not substitute ionice for md controls

ionice is useful for userspace processes supported by the active I/O scheduler, but md recovery is kernel-managed work. Do not assume changing a shell or mdadm process priority controls the recovery stream. Use the documented md limits, then prove the result at the member devices and application.

Keep Degraded-Time Risk Visible

Reducing I/O pressure can stabilize production, yet every reduction changes the recovery ETA. Record both sides of that decision. If degraded is greater than zero, state how long the array is expected to remain degraded, which additional failure the RAID level can tolerate, whether a verified backup exists, and who owns replacement-media health.

Stopping recovery by writing idle to sync_action is not a normal tuning shortcut. Kernel documentation says idle stops active recovery or resync but does not guarantee that it will stay stopped; another event can start it again. More importantly, a paused degraded recovery leaves redundancy unrestored. Prefer rate control unless an evidence-backed maintenance decision explicitly requires a stop.

RAID recovery is not data recovery

A completed rebuild proves that md restored its redundancy contract. It does not prove application consistency, detect every latent data problem, or create an independent copy. Before storage work, guest-aware backup consistency helps align filesystem and application state. Separately, complete host-failure restore testing so one surviving array is not the only recovery plan.

For RAID5 or RAID6, be especially cautious around dirty and degraded assembly. Linux kernel documentation warns that parity cannot be trusted in that combined state and normally refuses automatic start. Forcing assembly is a data-risk decision, not a performance tweak, and sits outside this speed-calibration procedure.

Turn the Temporary Limit Into an Operating Envelope

One successful interval is useful evidence, not a permanent policy. Repeat the measurement through peak and off-peak traffic, then document a small operating envelope: acceptable application latency, recovery maximum for each traffic band, minimum allowed recovery progress, estimated degraded duration, rollback value and responsible owner.

If persistence is justified, use configuration management that identifies the intended array reliably and records why the value exists. Per-array sysfs settings may need to be applied after array assembly; a blind boot-time write to md0 can target the wrong array if naming changes. Global sysctl persistence should be reserved for hosts where one policy genuinely fits every md array.

Monitor state changes, not only percentages

mdadm --monitor can alert on array events. The mdadm manual specifically notes that a one-shot scan from scheduled automation can notify about degraded arrays. Test the alert path; an installed monitor that cannot deliver its notification is not coverage.

Your handoff record should include:

• array UUID, RAID level and member list;
• active sync_action, degraded count and progress;
• original and current min/max scope and value;
• application SLO plus before/after measurements;
• member-device queue evidence and any error events;
• ETA, backup/restore evidence, rollback command and owner.

That receipt lets the next operator distinguish a deliberate limit from a stuck recovery. It also prevents a temporary incident value from becoming unexplained infrastructure folklore.

Recovery speed becomes defensible when it is tied to workload evidence, redundancy risk and a reversible setting. Keep the final receipt with the incident record, then use Voxfor’s Linux operations library for adjacent server diagnostics after the array is healthy.