Replacing a Failed Drive in Synology SHR

The first time my NAS started beeping at me, I had no idea whether to power it down, which drive to pull, or whether the orange LED meant my data was gone. It wasn't. On SHR with one-disk redundancy the pool keeps serving files with a failed member. The risk is the window until the rebuild ends, when a second failure would be fatal.

This is what I do on DSM 6.2, from the first beep to a healthy pool. Before you touch anything: have a replacement at least as large as the failed drive (larger is fine on SHR), make sure you've got a recent backup because a rebuild stresses the surviving drives for hours, and know whether your model is hot-swappable. Most desktop plus models and all rack models are; a few entry-level units need a shutdown.

Which drive died?

The beep and the orange status LED tell you something is wrong; Storage Manager tells you what. Open Storage Manager → Storage Pool: the pool shows "Degraded". Then HDD/SSD: one drive is marked "Crashed" or "Failed" (or has disappeared from the list), while the others read "Normal". Note the drive slot number and the serial.

Turn off the alarm from Storage Manager → Overview → Beep Off, otherwise it continues for the whole rebuild.

From the shell you can see the same picture:

cat /proc/mdstat
sudo mdadm --detail /dev/md2

mdstat shows [U_U] style flags with an underscore for the missing member. On SHR with mixed sizes there are several md devices, one per size tier, and all of them will show the same missing slot.

Also read the S.M.A.R.T. history: HDD/SSD → the failed drive → Health Info. Reallocated sector count and pending sectors climbing over the previous weeks confirm a genuine disk failure rather than a loose connection.

Find the right bay

Storage Manager numbers drives 1 to N and the bays are numbered on the chassis, but I still double-check. Pulling the wrong drive from a degraded pool is the mistake you can't undo. Select the drive in HDD/SSD and use Action → Locate (on models that support it) to blink the bay LED. Failing that, match the serial number printed on the drive label against the one in Health Info before pulling anything.

Swap the disk

On a hot-swappable model, with the NAS running, unlock and remove the tray for the failed drive, swap the disk, and slide the tray back in. Wait for the slot LED to come on and for the drive to appear in HDD/SSD as "Not Initialized".

On a model without hot swap: Control Panel → Hardware & Power → Shut Down, replace the drive, power on. The pool will still be degraded at boot; that is expected.

If the replacement previously lived in another system, DSM may show it as containing a foreign volume. That is fine; the repair step overwrites it.

Repair the pool

Storage Manager → Storage Pool → select the degraded pool → Action → Repair. The wizard lists available drives; select the new one and confirm. DSM partitions the drive, adds it to each md array, and the status changes to "Repairing" with a percentage.

Rebuild time depends on size and load. As a rule of thumb, 4 TB rebuilds in 8 to 12 hours on an idle unit, double that if users are hammering it. You can adjust the priority in Storage Manager → Storage Pool → the pool → Configuration → RAID Resync Speed Limits; I set it to "Lower impact on overall system performance" during the day and switch to faster at night.

watch -n 30 cat /proc/mdstat

shows the progress and estimated finish time per array.

Don't skip the scrub

Once the pool shows "Normal" again, run a data scrub if the volume is Btrfs: Storage Manager → Storage Pool → Action → Data Scrubbing. It reads everything and verifies checksums, which catches silent errors that the rebuild may have propagated from a marginal remaining drive. On ext4 a RAID parity consistency check is available from the same menu.

Is it healthy again?

• Storage Manager → Storage Pool shows "Normal" and Volume shows "Normal". If you inserted a larger drive and the pool now has unusable extra space, that is normal on SHR until the other drives are also enlarged.
• HDD/SSD → the new drive → Health Info → S.M.A.R.T. → run a Quick Test now and schedule Extended tests monthly.
• cat /proc/mdstat shows all [UUUU] with no underscores and no recovery lines.
• The status LED is back to steady green and the notification log under Control Panel → Notification → Advanced does not show new storage warnings.

The drive that came out

Do not throw it out untested if it was still under warranty; run the vendor's diagnostic (SeaTools, WD Data Lifeguard) on a PC and file the RMA with the log. If it held sensitive data, wipe it before RMA or disposal. A secure erase from a Linux box:

sudo hdparm --user-master u --security-set-pass p /dev/sdX
sudo hdparm --user-master u --security-erase p /dev/sdX

If the drive is too far gone for that, physical destruction is the answer.

When the repair doesn't go smoothly

• The new drive is not listed in the Repair wizard. It is smaller than the failed one, or it has a partition table DSM refuses to touch. Check the size, and if needed wipe it in another machine with wipefs -a /dev/sdX.
• Repair fails with "Failed to repair the storage pool" partway through. A second drive is throwing read errors; look at HDD/SSD for a drive now marked "Warning" and check Health Info. This is the scenario the backup is for: stop, copy off what matters, then replace the second drive.
• Storage Manager shows the pool as "Crashed" rather than "Degraded" after a single failure. Either two drives failed, or the pool was SHR without data protection (single drive or SHR with no redundancy). A crashed pool is not repairable through the GUI; restore from backup.
• The rebuild restarts from zero after a reboot. Rebuilds resume from a checkpoint normally, but if the NAS was powered off mid-rebuild without a clean shutdown, md starts over. Let it finish this time.

Buy the spare when you buy the NAS. The swap itself is routine; the dangerous part is running degraded for weeks because nobody noticed, which is what the email notifications in Control Panel → Notification are for.