Solved The pool is DEGRADED

Del.Mar · Dec 5, 2024

Hi mates!
After a sudden power outage my raidz pool state has changed to DEGRADED. My box is FreeBSD 14.1

Code:

# zpool status -v storage
  pool: storage
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME                      STATE     READ WRITE CKSUM
        storage                   DEGRADED     0     0     0
          raidz1-0                DEGRADED     0     0     0
            11915794483624591419  FAULTED      0     0     0  was /dev/ada1p1
            ada2                  ONLINE       0     0     0
            ada3                  ONLINE       0     0     0

errors: No known data errors

Also

Code:

# gpart list ada1
Geom name: ada1
modified: false
state: CORRUPT
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 4000785960960 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   efimedia: HD(1,GPT,b22564e3-97d3-11ef-b987-b47af1365428,0x800,0x1d1c0b688)
   rawuuid: b22564e3-97d3-11ef-b987-b47af1365428
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 4000785960960
   offset: 1048576
   type: freebsd-zfs
   index: 1
   end: 7814037127
   start: 2048
Consumers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Is there a way to fix it without destroy the pool?

Argentum · Dec 5, 2024

Del.Mar said:
Hi mates!
After a sudden power outage my raidz pool state has changed to DEGRADED. My box is FreeBSD 14.1

Is there a way to fix it without destroy the pool?

Replace the disk. That is the whole idea of Raidz. You had /dev/ada1p1, so you must create that partition on the new disk first.

Your pool lives as long as another disk breaks. In that case ALL data will be lost.

Del.Mar · Dec 5, 2024

Argentum said:
Replace the disk

But disk is alive. How to fix it without replace?

shurik · Dec 5, 2024

You have corrupted gpt partition table. Try to recover it and bring vdev online:

Code:

# gpart recover /dev/ada1
# zpool online storage /dev/ada1p1

cracauer@ · Dec 5, 2024

I would first look at the SMART data and test the disk before adding it back in.

Del.Mar · Dec 5, 2024

shurik said:
Try to recover it and bring vdev online

Done

Code:

# gpart list /dev/ada1
Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT

Code:

# zpool online storage /dev/ada1p1
warning: device '/dev/ada1p1' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present

cracauer@ said:
I would first look at the SMART data

Here.

Code:

# smartctl -a /dev/ada1
smartctl 7.4 2023-08-01 r5530 [FreeBSD 14.1-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST4000VN008-2DR166
Serial Number:    ZGY7W30J
LU WWN Device Id: 5 000c50 0c700b73f
Firmware Version: SC60
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Dec  5 19:03:10 2024 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  581) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 624) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   079   064   044    Pre-fail  Always       -       81186599
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       130
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   094   060   045    Pre-fail  Always       -       2530899469
  9 Power_On_Hours          0x0032   062   062   000    Old_age   Always       -       33524 (223 57 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       130
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   063   050   040    Old_age   Always       -       37 (Min/Max 34/40)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       50
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1663
194 Temperature_Celsius     0x0022   037   050   000    Old_age   Always       -       37 (0 18 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       33330h+10m+34.470s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       73857089836
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2184601666122

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     14152         -
# 2  Short offline       Completed without error       00%     14128         -
# 3  Short offline       Completed without error       00%     14104         -
# 4  Short offline       Completed without error       00%     14080         -
# 5  Short offline       Completed without error       00%     14057         -
# 6  Short offline       Completed without error       00%     14034         -
# 7  Short offline       Completed without error       00%     14008         -
# 8  Short offline       Completed without error       00%     13984         -
# 9  Short offline       Completed without error       00%     13960         -
#10  Short offline       Completed without error       00%     13936         -
#11  Short offline       Completed without error       00%     13912         -
#12  Extended offline    Completed without error       00%     13896         -
#13  Short offline       Completed without error       00%     13840         -
#14  Short offline       Completed without error       00%     13816         -
#15  Short offline       Completed without error       00%     13792         -
#16  Short offline       Completed without error       00%     13768         -
#17  Short offline       Completed without error       00%     13744         -
#18  Short offline       Completed without error       00%     13720         -
#19  Short offline       Completed without error       00%     13698         -
#20  Short offline       Completed without error       00%     13672         -
#21  Short offline       Completed without error       00%     13648         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try 'smartctl -x' for more

cracauer@ · Dec 5, 2024

Looks fine. I would still hammer it a bit with UFS before adding it back in.

You wouldn't want to start a resilver, which puts stress on your other disks, only to not make it through.

freejlr · Saturday at 12:04 PM

Hi

You executed a:

zpool clear storage

To clean up previous errors from your zpool? Then check the status again in verbose mode:

zpool status -v storage

Del.Mar · Saturday at 7:07 PM

freejlr said:
To clean up previous errors from your zpool?

I've made offline corrupted disk and replace it with the same /dev/ada1p1. After resilver works fine now.

gpw928 · Sunday at 1:37 AM

I recommend that you read the TrueNAS Hard Drive Troubleshooting Guide, use their excellent step-by-step "Troubleshooting Procedure".

For Seagate disks, the SMART data are a little different. See Appendix B. To get the critical numbers from Seagate disks use:

Code:

sudo smartctl -v 1,hex48 -v 7,hex48 -v195,hex48 -a /dev/ada1

[I know you don't have Code 195, but some Seagate disks do.]

See my previous post for a short summary of the numbers that really matter.

Given that your disk looks OK, I'd be double checking the security of all the cables and connectors [edit: after the resilver is complete!].