Ggtkuk

I've been using a RAID5 HDD storage (8x6TB) at my HP P840 for like 2 years now and it has always had unusually many drive failures. Everything was good for half a year, but now drives are failing in a strange way.
For example 2 new drives failed a few days after they have been added to the RAID.
I have also already replaced the RAID controller and am using up-to-date firmware on Mainboard and RAID controller.

Also I have tried to use different drives.
Initially there were HGST DeskStar 6TB drives used in that RAID, now I have been replacing them with HGST UltraStar 6TB when replacing failed drives. But the behaviour is the same.

Also it seems (most of) the drives are not really failed, because once I replaced the RAID controller, one failed drive was recognized as OK again and the rebuild started.

My hoster's support says the problem is that I'm actually using RAID5 and I should switch to RAID10 instead. It's hard for me to believe as I've been using RAID5 without problems at other systems (no drive failures in years).

Can anyone give me a hint, what else the culprit could be? Something wrong with the way the RAID controller is configured?

Thank you!

EDIT:

The server is a HP DL180 G9

Reason for drive failure is always "Write retries failed"

  Smart Array P840 in Slot 1                (sn: PDNNF0ARH321GD)





     Port Name: 1I



     Port Name: 2I



     Internal Drive Cage at Port 1I, Box 2, OK



     Internal Drive Cage at Port 1I, Box 2, OK



     Internal Drive Cage at Port 2I, Box 1, OK

     array A (Solid State SATA, Unused Space: 0  MB)





  logicaldrive 1 (447.1 GB, RAID 1+0, OK)



  physicaldrive 2I:1:1 (port 2I:box 1:bay 1, Solid State SATA, 240.0 GB, OK)

  physicaldrive 2I:1:2 (port 2I:box 1:bay 2, Solid State SATA, 240.0 GB, OK)

  physicaldrive 2I:1:3 (port 2I:box 1:bay 3, Solid State SATA, 240.0 GB, OK)

  physicaldrive 2I:1:4 (port 2I:box 1:bay 4, Solid State SATA, 240.0 GB, OK)



     array B (SATA, Unused Space: 0  MB)





  logicaldrive 2 (38.2 TB, RAID 5, Interim Recovery Mode)



  physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SATA, 6001.1 GB, OK)

  physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SATA, 6001.1 GB, OK)

  physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SATA, 6001.1 GB, OK)

  physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SATA, 6001.1 GB, OK)

  physicaldrive 1I:2:5 (port 1I:box 2:bay 5, SATA, 6001.1 GB, Failed)

  physicaldrive 1I:2:6 (port 1I:box 2:bay 6, SATA, 6001.1 GB, OK)

  physicaldrive 1I:2:7 (port 1I:box 2:bay 7, SATA, 6001.1 GB, OK)

  physicaldrive 1I:2:8 (port 1I:box 2:bay 8, SATA, 6001.1 GB, OK)

Detailed Info:

     Smart Array P840 in Slot 1

        Bus Interface: PCI

        Slot: 1

        Serial Number: PDNNF0ARH321GD

        Cache Serial Number: PEYFP0BRH323YZ

        RAID 6 (ADG) Status: Enabled

        Controller Status: OK

        Hardware Revision: B

        Firmware Version: 6.60

        Rebuild Priority: High

        Expand Priority: Medium

        Surface Scan Delay: 3 secs

        Surface Scan Mode: Idle

        Parallel Surface Scan Supported: Yes

        Current Parallel Surface Scan Count: 1

        Max Parallel Surface Scan Count: 16

        Queue Depth: Automatic

        Monitor and Performance Delay: 60  min

        Elevator Sort: Enabled

        Degraded Performance Optimization: Disabled

        Inconsistency Repair Policy: Disabled

        Wait for Cache Room: Disabled

        Surface Analysis Inconsistency Notification: Disabled

        Post Prompt Timeout: 15 secs

        Cache Board Present: True

     Cache Status: OK

     Cache Ratio: 10% Read / 90% Write

     Drive Write Cache: Enabled

     Total Cache Size: 4.0 GB

     Total Cache Memory Available: 3.2 GB

     No-Battery Write Cache: Enabled

     SSD Caching RAID5 WriteBack Enabled: True

     SSD Caching Version: 2

     Cache Backup Power Source: Batteries

     Battery/Capacitor Count: 1

     Battery/Capacitor Status: OK

     SATA NCQ Supported: True

     Spare Activation Mode: Activate on physical drive failure (default)

     Controller Temperature (C): 51

     Cache Module Temperature (C): 38

     Number of Ports: 2 Internal only

     Encryption: Disabled

     Express Local Encryption: False

     Driver Name: hpsa

     Driver Version: 3.4.16

     Driver Supports HP SSD Smart Path: True

     PCI Address (Domain:Bus:Device.Function): 0000:06:00.0

     Negotiated PCIe Data Rate: PCIe 3.0 x8 (7880 MB/s)

     Controller Mode: RAID

     Controller Mode Reboot: Not Required

     Latency Scheduler Setting: Disabled

     Current Power Mode: MaxPerformance

     Host Serial Number: CZ270500GM

     Sanitize Erase Supported: False

     Primary Boot Volume: logicaldrive 1 (600508B1001CE0F9FACF3A1358647115)

     Secondary Boot Volume: logicaldrive 1 (600508B1001CE0F9FACF3A1358647115)





     Port Name: 1I

           Port ID: 0

           Port Connection Number: 0

           SAS Address: 5001438038AD05A0

           Port Location: Internal

           Managed Cable Connected: False



     Port Name: 2I

           Port ID: 1

           Port Connection Number: 1

           SAS Address: 5001438038AD05A8

           Port Location: Internal

           Managed Cable Connected: False



     Internal Drive Cage at Port 1I, Box 2, OK

        Power Supply Status: Not Redundant

        Drive Bays: 4

        Port: 1I

        Box: 2

        Location: Internal



     Physical Drives

        physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SATA, 6001.1 GB, OK)

        physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SATA, 6001.1 GB, OK)

        physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SATA, 6001.1 GB, OK)

        physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SATA, 6001.1 GB, OK)

        None attached





     Internal Drive Cage at Port 1I, Box 2, OK

        Power Supply Status: Not Redundant

        Drive Bays: 4

        Port: 1I

        Box: 2

        Location: Internal



     Physical Drives

        physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SATA, 6001.1 GB, OK)

        physicaldrive 1I:2:2 (port 1I:box 2:bay 2, SATA, 6001.1 GB, OK)

        physicaldrive 1I:2:3 (port 1I:box 2:bay 3, SATA, 6001.1 GB, OK)

        physicaldrive 1I:2:4 (port 1I:box 2:bay 4, SATA, 6001.1 GB, OK)

        None attached





     Internal Drive Cage at Port 2I, Box 1, OK

        Power Supply Status: Not Redundant

        Drive Bays: 4

        Port: 2I

        Box: 1

        Location: Internal



     Physical Drives

        physicaldrive 2I:1:1 (port 2I:box 1:bay 1, Solid State SATA, 240.0 GB, OK)

        physicaldrive 2I:1:2 (port 2I:box 1:bay 2, Solid State SATA, 240.0 GB, OK)

        physicaldrive 2I:1:3 (port 2I:box 1:bay 3, Solid State SATA, 240.0 GB, OK)

        physicaldrive 2I:1:4 (port 2I:box 1:bay 4, Solid State SATA, 240.0 GB, OK)

        None attached



     Array: A

        Interface Type: Solid State SATA

        Unused Space: 0  MB (0.0%)

        Used Space: 894.2 GB (100.0%)

        Status: OK

        MultiDomain Status: OK

        Array Type: Data

        HP SSD Smart Path: disable







  Logical Drive: 1

     Size: 447.1 GB

     Fault Tolerance: 1+0

     Heads: 255

     Sectors Per Track: 32

     Cylinders: 65535

     Strip Size: 256 KB

     Full Stripe Size: 512 KB

     Status: OK

     MultiDomain Status: OK

     Caching:  Enabled

     Unique Identifier: 600508B1001CE0F9FACF3A1358647115

     Disk Name: /dev/sda

     Mount Points: / 18.6 GB Partition Number 2

     OS Status: LOCKED

     Logical Drive Label: 0216D6F9PDNNF0ARH502MC7DFA

     Mirror Group 1:

        physicaldrive 2I:1:1 (port 2I:box 1:bay 1, Solid State SATA, 240.0 GB, OK)

        physicaldrive 2I:1:2 (port 2I:box 1:bay 2, Solid State SATA, 240.0 GB, OK)

     Mirror Group 2:

        physicaldrive 2I:1:3 (port 2I:box 1:bay 3, Solid State SATA, 240.0 GB, OK)

        physicaldrive 2I:1:4 (port 2I:box 1:bay 4, Solid State SATA, 240.0 GB, OK)

     Drive Type: Data

     LD Acceleration Method: Controller Cache



  physicaldrive 2I:1:1

     Port: 2I

     Box: 1

     Bay: 1

     Status: OK

     Drive Type: Data Drive

     Interface Type: Solid State SATA

     Size: 240.0 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Firmware Revision: N2010101

     Serial Number: PHDV712004AG240AGN

     Model: ATA     INTEL SSDSC2BB24

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 31

     Maximum Temperature (C): 39

     SSD Smart Trip Wearout: Not Supported

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 2I:1:2

     Port: 2I

     Box: 1

     Bay: 2

     Status: OK

     Drive Type: Data Drive

     Interface Type: Solid State SATA

     Size: 240.0 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Firmware Revision: N2010101

     Serial Number: PHDV706303CH240AGN

     Model: ATA     INTEL SSDSC2BB24

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 29

     Maximum Temperature (C): 36

     SSD Smart Trip Wearout: Not Supported

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 2I:1:3

     Port: 2I

     Box: 1

     Bay: 3

     Status: OK

     Drive Type: Data Drive

     Interface Type: Solid State SATA

     Size: 240.0 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Firmware Revision: N2010101

     Serial Number: PHDV712003V8240AGN

     Model: ATA     INTEL SSDSC2BB24

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 29

     Maximum Temperature (C): 35

     SSD Smart Trip Wearout: Not Supported

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 2I:1:4

     Port: 2I

     Box: 1

     Bay: 4

     Status: OK

     Drive Type: Data Drive

     Interface Type: Solid State SATA

     Size: 240.0 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Firmware Revision: N2010101

     Serial Number: PHDV712004GA240AGN

     Model: ATA     INTEL SSDSC2BB24

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 31

     Maximum Temperature (C): 37

     SSD Smart Trip Wearout: Not Supported

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False





     Array: B

        Interface Type: SATA

        Unused Space: 0  MB (0.0%)

        Used Space: 43.7 TB (100.0%)

        Status: Failed Physical Drive

        MultiDomain Status: OK

        Array Type: Data

        HP SSD Smart Path: disable



        Warning: One of the drives on this array have failed or has been removed.









  Logical Drive: 2

     Size: 38.2 TB

     Fault Tolerance: 5

     Heads: 255

     Sectors Per Track: 32

     Cylinders: 65535

     Strip Size: 256 KB

     Full Stripe Size: 1792 KB

     Status: Interim Recovery Mode

     MultiDomain Status: OK

     Caching:  Enabled

     Parity Initialization Status: Initialization Failed

     Unique Identifier: 600508B1001CF94F84873C91FD89B549

     Disk Name: /dev/sdb

     Mount Points: None

     Logical Drive Label: 04DA1DD6PDNNF0ARH502MC546F

     Drive Type: Data

     LD Acceleration Method: Controller Cache



  physicaldrive 1I:2:1

     Port: 1I

     Box: 2

     Bay: 1

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: APGNW7JH

     Serial Number: NAHN3UZY

     Model: ATA     HGST HDN726060AL

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 37

     Maximum Temperature (C): 43

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 1I:2:2

     Port: 1I

     Box: 2

     Bay: 2

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: APGNT517

     Serial Number: NAHLKP0X

     Model: ATA     HGST HDN726060AL

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 37

     Maximum Temperature (C): 56

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 1I:2:3

     Port: 1I

     Box: 2

     Bay: 3

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: T7MH

     Serial Number: NCH8E81Z

     Model: ATA     HUS726060ALE610

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 33

     Maximum Temperature (C): 41

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 1I:2:4

     Port: 1I

     Box: 2

     Bay: 4

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: APGNW7JH

     Serial Number: NAHYMAUY

     Model: ATA     HGST HDN726060AL

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 34

     Maximum Temperature (C): 41

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 1I:2:5

     Port: 1I

     Box: 2

     Bay: 5

     Status: Failed

     Last Failure Reason: Write retries failed

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: T7MH

     Serial Number: K1H942MD

     Model: ATA     HUS726060ALE610

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Maximum Temperature (C): 43

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Applicable

     Sanitize Erase Supported: False



  physicaldrive 1I:2:6

     Port: 1I

     Box: 2

     Bay: 6

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: TDR2

     Serial Number: K8JM5TKN

     Model: ATA     HUS726060ALE610

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 33

     Maximum Temperature (C): 38

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 1I:2:7

     Port: 1I

     Box: 2

     Bay: 7

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: APGNW7JH

     Serial Number: K8H9BW2N

     Model: ATA     HGST HDN726060AL

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 34

     Maximum Temperature (C): 39

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False



  physicaldrive 1I:2:8

     Port: 1I

     Box: 2

     Bay: 8

     Status: OK

     Drive Type: Data Drive

     Interface Type: SATA

     Size: 6001.1 GB

     Drive exposed to OS: False

     Native Block Size: 4096

     Rotational Speed: 7200

     Firmware Revision: T7MH

     Serial Number: K1H623JD

     Model: ATA     HUS726060ALE610

     SATA NCQ Capable: True

     SATA NCQ Enabled: True

     Current Temperature (C): 35

     Maximum Temperature (C): 40

     PHY Count: 1

     PHY Transfer Rate: 6.0Gbps

     Drive Authentication Status: Not Authenticated. Smart Array will not control drive LEDs.

     Sanitize Erase Supported: False

You probably have an heavily punctured array, which cause an early "planned death" of the replaced due to failed stripe reconstruction. You can read more information here and here

The solution is to backup, destroy the array, recreate it and restore from backup.

Next time avoid using a RAID5 array with such big drives. I strongly suggest using RAID6 or, even better, RAID10.

You should be using RAID6 with the size and types of disks in the system. However, some details about the server may be helpful.

Is this an HPE server, or are you just using an HPE controller?

These don't appear to be HPE drives or HPE drive carriers. That's a bad sign.

The hpssacli output you've provided would also show the reason for the disk failure. If you're not on an HPE server and there's a backplane issue or SATA timeouts (noticed you're on SATA disks), there's chance that you're getting false positives.

Example: (see the Last Failure Reason line):

  physicaldrive 2I:2:8

     Port: 2I

     Box: 2

     Bay: 8

     Status: Failed

     Last Failure Reason: Aborted Command

     Drive Type: Data Drive