ZFS Help me understand raidz1-1

Status
Not open for further replies.
B

BAAlghamdi

Hi
I am a Data Recovery engineer and this is my first post here and also the first time I work on raidz,
I got a customer needs to recover data from damaged NAS storage with 36 hard drives.
I have a good experience recovering data from Other types of raid like 5, 10 etc, but this is my first time working on ZFS.

the 36 hard drives are divided into 4 raidz1 but there is a number after the dash like raidz1-0, raidz1-1, raidz1-2, raidz1-3.

the customer started with 12 drives then added another 12 then 6 and lastly 6 drives

the log files show like this


Code: 
SEDataStorage                DEGRADED     0     0     0
      raidz1-0                   DEGRADED     0     0     0
        c0t50014EE25E9A12E0d0    ONLINE       0     0     0
        c0t50014EE25E970BA6d0    ONLINE       0     0     0
        c0t50014EE2094171A8d0    ONLINE       0     0     0
        c0t50014EE2B3E9B8C5d0    ONLINE       0     0     0
        replacing-4              DEGRADED     0     0     0
          c0t50014EE25E9A2C13d0  ONLINE       0     0     0
          7419237290135383910    UNAVAIL      0     0     0  was /dev/dsk/c0t50014EE25F8AE32Dd0s0
        c0t50014EE2B3EC69C9d0    ONLINE       0     0     0
        c0t50014EE20942C9B3d0    ONLINE       0     0     0
        c0t50014EE25E96C76Fd0    ONLINE       0     0     0
        c0t50014EE25E986C48d0    ONLINE       0     0     0
        c0t50014EE25E99B63Ad0    ONLINE       0     0     0
        c0t50014EE20941614Dd0    ONLINE       0     0     0
        c0t50014EE25E96EB55d0    ONLINE       0     0     0
      raidz1-1                   ONLINE       0     0     0
        c0t50014EE6051E6E86d0    ONLINE       0     0     0
        c0t50014EE65A73221Ad0    ONLINE       0     0     0
        c0t50014EE6AFC996BDd0    ONLINE       0     0     0
        c0t50014EE65A74537Cd0    ONLINE       0     0     0
        c0t50014EE0AE97F869d0    ONLINE       0     0     0
        c0t50014EE0AE97F8F8d0    ONLINE       0     0     0
        c0t50014EE059426EEBd0    ONLINE       0     0     0
        c0t50014EE6AFC9DC54d0    ONLINE       0     0     0
        c0t50014EE0AE9807FEd0    ONLINE       0     0     0
        c0t50014EE6AFC8D0B0d0    ONLINE       0     0     0
        c0t50014EE6AFC87B73d0    ONLINE       0     0     0
        c0t50014EE6051F051Ad0    ONLINE       0     0     0
      raidz1-2                   ONLINE       0     0     0
        c0t50014EE0AECFE27Bd0    ONLINE       0     0     0
        c0t50014EE004252FC5d0    ONLINE       0     0     0
        c0t50014EE0042527DDd0    ONLINE       0     0     0
        c0t50014EE0AECFF013d0    ONLINE       0     0     0
        c0t50014EE0AECFE372d0    ONLINE       0     0     0
        c0t50014EE0AECFEA42d0    ONLINE       0     0     0
      raidz1-3                   ONLINE       0     0     0
        c0t50014EE003A7028Ed0    ONLINE       0     0     0
        c0t50014EE003A4EA42d0    ONLINE       0     0     0
        c0t50014EE003A794E7d0    ONLINE       0     0     0
        c0t50014EE003A74322d0    ONLINE       0     0     0
        c0t50014EE058FC3D85d0    ONLINE       0     0     0
        c0t50014EE003A75CA7d0    ONLINE       0     0     0

the question is
How the 4 groups linked?
Are they together in raidz or raidz1 or JBOD ?

or if you have any suggestion on how to recover the files from this pool please help
 
It looks like you have a pool with name SEDataStorage with 4 vdevs of type RAIDZ1

Code: 
raidz1-0 DEGRADED
raidz1-1 ONLINE
raidz1-2 ONLINE
raidz1-3 ONLINE

The problem is in vdev raidz1-0 , it appears like the the 5th disk (count begin in 0: replacing-4) have failed, but I can't understand why it say replacing-4

Code: 
replacing-4 DEGRADED 0 0 0
c0t50014EE25E9A2C13d0 ONLINE 0 0 0
7419237290135383910 UNAVAIL 0 0 0 was /dev/dsk/c0t50014EE25F8AE32Dd0s0

Can you put the output of zpool status? inside of a code block to not lose the indentation

Read the Handbook, the part that Dealing with Failed Devices
 
Martin Paredes said:
The problem is in vdev raidz1-0 , it appears like the the 5th disk (count begin in 0: replacing-4) have failed, but I can't understand why it say replacing-4
Click to expand...
It's probably because the disk failed and the customer replaced it with a fresh new one. Then unfortunately the 7-th disk from the first VDEV also failed before the 5-th disk was completely resilvered. This made the whole pool unusable.

To get to the primary question: "How the 4 groups linked?"
I don't know the details about the technical implementation, but from a user perspective the VDEVs (raidz1-0 to raidz1-3) are striped together. So this would be functionally equivalent to RAID-0. If the format of the data on disk is the same I don't know.

What I know that people use the tool zdb(8) in such data recovery scenarios, along with zpool(8) and zfs(8). It's quite important to always use the readonly=on property.
There is a video on youtube that skims over a data recovery session:

I hope this helps a bit.
 
RAID-Z is a raid system similar to RAID5 that stripes that data across the disks, losing the capacity of 1 for redundancy. In a pool with multiple vdevs, the data is striped across them. The second number is just an incremental counter to give each vdev a unique name (although I don't know if that name actually has any real use).

c0t50014EE25E9A12E0d0 does not look like a disk label FreeBSD would use.

Then unfortunately the 7-th disk from the first VDEV also failed before the 5-th disk was completely resilvered.
Click to expand...

I can only see one failed disk. The first vdev should look like this -

Code: 
SEDataStorage DEGRADED 0 0 0
    raidz1-0 DEGRADED 0 0 0
        c0t50014EE25E9A12E0d0 ONLINE 0 0 0
        c0t50014EE25E970BA6d0 ONLINE 0 0 0
        c0t50014EE2094171A8d0 ONLINE 0 0 0
        c0t50014EE2B3E9B8C5d0 ONLINE 0 0 0
        replacing-4 DEGRADED 0 0 0
            c0t50014EE25E9A2C13d0 ONLINE 0 0 0
            7419237290135383910 UNAVAIL 0 0 0 was /dev/dsk/c0t50014EE25F8AE32Dd0s0
        c0t50014EE2B3EC69C9d0 ONLINE 0 0 0
        c0t50014EE20942C9B3d0 ONLINE 0 0 0
        c0t50014EE25E96C76Fd0 ONLINE 0 0 0
        c0t50014EE25E986C48d0 ONLINE 0 0 0
        c0t50014EE25E99B63Ad0 ONLINE 0 0 0
        c0t50014EE20941614Dd0 ONLINE 0 0 0
        c0t50014EE25E96EB55d0 ONLINE 0 0 0

I haven't replaced a disk in a while so I'm not sure whether the DEGRADED status on the replacing vdev suggests an issue or just that it hasn't finished rebuilding. We also haven't been given the rest of the status output, which would be highly useful as it lists the current/previous resilver details.

I have known situations a while back where replacing vdevs wouldn't clear after a rebuild. It may be worth running the following command to see if that replace section will clear. (Especially if the rest of the status output suggests a successful resilver)
Code: 
zpool detach SEDataStorage c0t50014EE25F8AE32Dd0s0

Is the pool actually unusable? A DEGRADED status does not suggest that the pool would not be working.

Of course this doesn't appear to be a FreeBSD issue.
 
Oh yeah, you're right. It shows both the replaced and the replacing disks. I was fooled by the missing indentation. My bad, I take it back.

I have had detaching disks issues with a pool recently (RAIDZ2). A pool in a DEGRADED state is still usable. It's in the zpool(8) man page:
Code: 
DEGRADED  One or more top-level vdevs is in the degraded state because
               one or more component devices are offline. Sufficient replicas
               exist to continue functioning.

               One or more component devices is in the degraded or faulted
               state, but sufficient replicas exist to continue functioning.
               The underlying conditions are as follows:

                 •   The number of checksum errors exceeds acceptable levels
                     and the device is degraded as an indication that
                     something may be wrong.  ZFS continues to use the device
                     as necessary.

                 •   The number of I/O errors exceeds acceptable levels. The
                     device could not be marked as faulted because there are
                     insufficient replicas to continue functioning.

@BAAlghamdi: Just try to import the ZFS pool with readonly=on and see if the data is readable. It should be still usable.

zpool import
zpool import -o readonly=on SEDataPool
 
Status
Not open for further replies.
Back
Top