Solved 3Ware 9690SA-4I4E - Drives Offline

X

xa0z

Hello,

Running FreeBSD 11.1-RELEASE.

I recently purchased a 3ware 9690SA-4I4E controller to swap into of my SQL server but have run into an issue. When I put the 9690SA-4I4E card in, the controller's BIOS shows all the drives are available but marked "offline". I've been looking for a method to allow the card to work so the drives are visible to the system without having to setup the drives in the RAID controller itself as the drives are already configured with zfs as a 4-way mirrored vdev.

If I put my old controller back in, everything works and the system boots, but the new one is having trouble bringing the drives online for some reason.

Is there a simple fix without having to backup the pool and reconfigure the drives for the new controller?
 
You'll have to spend some time with Google. That 3Ware card is *capable* of doing RAID. That's excellent, because RAID is the first line of defense against disk failures and data loss. The bad news is that RAID-capable cards are sometimes obnoxious, and insist that you create a RAID volume out of the raw disks drives, and then they present that RAID volume as a storage device to the OS (in your case, FreeBSD). If you want to use individual disks without any redundancy and without any striping/concatenation on a RAID controller, you have to create RAID volumes out of a single disk drive, and that's the pain you are going through.

Some RAID controllers are capable of being put into "JBOD" mode (JBOD stands for "just a bunch of disks"). In that mode, they don't attempt to create RAID volumes (actually, they lose the ability to do RAID), and instead become just a disk adapter. Some other RAID controllers can be put into a hybrid mode, where they can both have some RAID volumes (built out of raw drives), and the remaining drives for which they have no RAID information are made directly into disk devices. On some RAID controllers, you can create a single-disk RAID volume *without* destroying the data on the disk (those RAID controllers store the RAID configuration solely in their own internal NVRAM configuration storage). And finally, the worst ones for you are those RAID controllers that require wiping the disk clean before it can be turned into a single-disk volume.

I'm really busy with plumbing and gardening work today (I'm typing a few posts while eating my lunch right now), so I don't have the hour that it would take to look up documentation for that specific 3Ware card, so please go do that yourself. If all else fails, you may have to backup the pool as you said.
 
The card is capable of JBOD but it has to be done from the CLI utility. I was under the impression that if I enable JBOD on the card that it would present the OS with 1 drive while the controller combined the individual drives.

I just wasn't sure if I had any options or not. I've spent the greater part of 2 days looking this up and there doesn't appear to be many posts with similar issues using this same controller.
 
Nope, on all cards I've ever seen JBOD mode means every attached drive is given to the OS individually. Combining them all into a single drive would mean RAID-0 over multiple drives, which is not JBOD. (Side remark: Remember that RAID-0 is not actually RAID at all, since it is not *r*edundant.)

Another suggestion: By "CLI utility" you probably mean you have to install some card-specific software. If that's difficult to find for FreeBSD, you might have to propare a DOS installation disks (in some cases, those utilities exist in a DOS version, and you can do tiny little free DOS distributions to run them). That might be easier. If you are lucky, and only have to do this to one machine, the easiest would be to do this through the BIOS, if available.

Good luck!
 
Setting the controller to JBOD mode is not available in the BIOS.

tw_cli is the utility I have to use to set it to JBOD according to the manual, which is available for FreeBSD. I might give that a shot unless you see any reason not to.
 
If it's available for FreeBSD, by all means go use it! My suggestions were only in case such a utility isn't readily available.
 
I was meaning more towards use case scenario. Would I be better of using the card as JBOD or making each disk as a "Single Disk" and using the functions of the card + zfs? I don't know much about using hardware RAID controllers, but I do know my old card I just used in IT mode (I believe).
 
Use JBOD if you plan on using ZFS. Hardware RAID and ZFS don't work well together and you won't be able to take advantage of the all the features of ZFS.
 
A: What SirDice said. The RAID that's built into ZFS is better than any external RAID can be. One reason: When a drive fails, ZFS only has to resilver allocated space. An external RAID controller has to resilver the whole drive, even if the file system is only 50% full. That means ZFS finishes resilvering finishes faster, which is both more convenient for the user (back to full performance sooner), and improves the overall reliability of the system.

B: Using a RAID controller with single-drive volumes (one drive -> one volume) is generally silly. There is no redundancy, nor performance improvement from striping over multiple drives. It adds the hassle of having to configure each drive at the controller, with the potential of making mistakes. It's sometimes necessary (if one has a controller that can only do that), but should otherwise be avoided if possible.
 
Yes, that old thread does make valid points, but I don't agree with them 100%.

Having battery-backed up RAM on your RAID controller card is vitally important when using real RAID (meaning redundancy, the same data is written to more than one disk). That's because otherwise the power might fail in the middle of a replicated write, and then you have inconsistent data on the disks, which will bite you (perhaps not right away, but eventually).

Once you have battery-backed RAM on the controller card, it can also be used to improve performance (which is vital for RAID, in particular parity-based RAID, and a good thing to have for non-redundant use like a single-disk virtual volume). But: How much memory does your card have? And how much memory does your computer have? If the buffer cache in the operating system is much larger than the (battery-backed-) RAM on the card, then the card will not add much. And the buffer cache in the operating system will be used more efficiently, because the file system knows what's important and what isn't. So the performance gain is not always big.

And having battery-backed RAM in the card opens the door to a lot of fascinating failure modes. Just one example: You have a sudden power failure while writes were occurring. Now some of the writes are on disk, some in the BB-RAM on the card. But while the power is down, you have to take one of the drives out (maybe you did what I did and spilled some soda on the machine and it needs to be cleaned). Then you power the machine up by mistake with the drive missing. The poor controller still has data for to write, but notices that the disk is gone; it might keep the write data in cache, or it might discard it. The next day, while you are at the dentist, your helpful but clueless colleague notices that the disk is missing, but connects it back to a motherboard SATA connector instead of to the controller (he's trying to help, but he doesn't know). ZFS sees the disk, and thinks that everything is fine, and a few hours later (right when you're back from the dentist), starts getting checksum errors because the content of the disk is wrong. That one is going to be fun to debug, and not fun to fix!

And this is why I don't like cache RAM on controller cards: typically little gain, and risk for screw-ups.
 
xa0z said:
Okay. I didn't know if there would be any caching benefits with the card or not, even with using zfs. The card has the BBU add-on, so thought there might be something else I need to do.

I saw this post and kind of got a bit confused. https://forums.freebsd.org/threads/15176/#post-88479
Click to expand...

In JBOD mode, all the advanced features of the card (including caching) is disabled, and it's treated/works like a standard HBA.

If you configure the drives as individual "Single Drive" devices, then you get access to all the advanced features of the card *and* you expose individual block devices to the OS. This gives you access to the different caching profiles and queuing profiles, and enables the BBU for the cache. The nice thing about the way 3Ware does "single drive" is that you can take one of those drives and move it to a normal HBA without having to reformat it. There's no proprietary metadata on disk locking it to that controller.

3Ware is unique in that way.

In theory, If fou have a large enough cache on the card with BBU enabled, you can disable the ZIL and configure the cache for Write-Back mode and it will (in essence) become the ZIL. Or enable the ZIL and set sync=always to have all writes go to the cache.

But, in a mixed controller scenario, you'd want to configure it in JBOD mode to make it act like a "dumb" HBA.
 
Well, unfortunately today doing some testing, either the backplane is bad, or the 3Ware card is bad. If I populate all 16 bays, only 12 of the drives are detected. If I only populate the first 8 bays, 2 of them are not detected.

If I populate bays 2, 3, 6, 7, 10, 11, 14, and 15, all drives are detected and the system boots, except that I'm given errors twa0 disk timeout on slot 11, and 15 over and over and the activity light on those bays is solid. I honestly don't understand what the deal is.

I'm leaning towards the 3ware card being the issue, but not really sure what other card to get to replace it with to test and ensure everything works right. I know I need a card to support 16 drives like the 3ware does, but I also need it to support drives at least up to 4tb each.
 
LSI/Avago/Broadcom 9211-8i are great HBAs that support 16 drives, and are significantly less expensive than the 3Ware RAID controller. These work great with FreeBSD, using the mps(4) driver. No problem supporting 4 TB drives with these (or larger). There's a bunch of different LSI chipsets supported by this driver (2000-series and 3000-series) so any HBA that uses those chipsets will use this driver.

Areca also makes really well-supported HBAs for FreeBSD systems. Pricing is decent, although slightly more than the Broadcom ones.
 
So I bought a Sun LSI SAS9211-8i off eBay for $30 and I seem to be having a slight situation.

Anytime I power on the box the system will POST and go to the LSI initialization screen and show 0 drives, then proceed to the "no boot media found" screen... but then I'll see all the drive lights start blinking, but it won't boot. So if I do CTRL + ALT + DEL (reboot) the system, at the LSI initialization screen it will show all 8 drives that are present. This happens on every initial power-up. I honestly don't see any setting that would control this behavior, so I'm not sure what to do about it.

Other than that, the system is up and running.
 
You must log in or register to reply here.
Back
Top