Noob question, RAID 6 software or hardware RAID

[night vision]

Hello everyone,


Building a NAS to replace two aging QNAP devices. Having tons of hardware issues but have a question before I get started regarding my drive controller.


Do I need to create the array in a raid 0/1/3/5/6/10 (or whatever) on the controller in order to have RAID capabilities or does the software do that for me? Does it have to be setup in hardware first or can it be setup later in the FreeNAS software? I prefer a RAID 6 because I can lose two drives instead of one like in RAID 5.


I'm currently having issues with my drive enclosures not powering on the drives. Outside of the cage they spin up just fine.

Motherboard: Super Micro Computer X9SCM-F
Processor: Intel Xeon E3-1230 Sandy Bridge 3.2 GHz
Controller: 10Gtek Internal PCI Express SAS/SATA HBA RAID Controller Card
NIC: Intel X520-DA2 Dual Port 10 Gigabit
HD: 8 Western Digital Red 4TB drives

In regards to my drives not spinning up -- you know those 3 half height 5.25" trayless bays that hold 4 3.5" drives? I have two by Athena Power BP-TLA3141SAC and the drives won't spin up inside the units. They work fine outside and plugged in. I can't solve this problem short of replacing them with different ones hoping they're just defective. I see basically all of these use the same backplane so I don't know if I got two broken ones. Waiting on new ones from iStarUSA.

I'm just at a loss and growing frustrated because this is becoming an expensive build if I keep having to swap out parts.


Many thanks!

Phil

p.s. I have been building and repairing computers for 35 years, just new to this arrangement.

 

[Ericloewe]

Do I need to create the array in a raid 0/1/3/5/6/10 (or whatever) on the controller in order to have RAID capabilities or does the software do that for me? Does it have to be setup in hardware first or can it be setup later in the FreeNAS software? I prefer a RAID 6 because I can lose two drives instead of one like in RAID 5.

Well, if that's an option you have, I'm afraid you have already chosen an improper controller. You need an HBA, which by definition cannot support hardware RAID (for practical reasons that nobody makes a RAID controller that works well as an HBA).

You will probably want to read the following resources:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

Controller: 10Gtek Internal PCI Express SAS/SATA HBA RAID Controller Card

The bad news is that the vendor is dodgy.
The good news is that the design should be solid, if they're actually following it.
The bad news is that it's hard to tell which firmware it's using.
The good news is that - in the worst case - you can crossflash the card to IT mode and end up with a functional HBA.
More details here, but the high-level summary is:

• Prepare a USB flash drive with the required firmware image and tools for UEFI shell
• Boot into the UEFI shell
• Take note of the SAS address for the card
• Wipe the card
• Flash IT firmware and, optionally, extension ROMs
• Program the SAS address

I'm currently having issues with my drive enclosures not powering on the drives. Outside of the cage they spin up just fine.

Are you using an adapter that does not provide +3.3V when testing them outside the cage? Sounds like you were a victim of the industry's silliest feature this side of SGI's fsn: PWDIS, as detailed here, for instance.
If this is indeed the case, you're sort of stuck between modding the cages and/or cables, or replacing the disks.

 

[night vision]

Well, if that's an option you have, I'm afraid you have already chosen an improper controller. You need an HBA, which by definition cannot support hardware RAID (for practical reasons that nobody makes a RAID controller that works well as an HBA).

You will probably want to read the following resources:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

The bad news is that the vendor is dodgy.
The good news is that the design should be solid, if they're actually following it.
The bad news is that it's hard to tell which firmware it's using.
The good news is that - in the worst case - you can crossflash the card to IT mode and end up with a functional HBA.
More details here, but the high-level summary is:

• Prepare a USB flash drive with the required firmware image and tools for UEFI shell
• Boot into the UEFI shell
• Take note of the SAS address for the card
• Wipe the card
• Flash IT firmware and, optionally, extension ROMs
• Program the SAS address

Are you using an adapter that does not provide +3.3V when testing them outside the cage? Sounds like you were a victim of the industry's silliest feature this side of SGI's fsn: PWDIS, as detailed here, for instance.
If this is indeed the case, you're sort of stuck between modding the cages and/or cables, or replacing the disks.

I have an LSI controller (URL below from Newegg) with the IT firmware but it is not recognized by the motherboard. I can get it pop up during POST, but the card doesn't exist when you go to flash it. The flash utility cannot find it, but those systems have motherboards that are over 10 years old so that cannot be an option to use here. Everything has to be relatively new. This is a production NAS for home use, not just a server to store some old photos. This is critical data which gets a weekly backup both on and off site.


Can you recommend a card? I also have a Syba SI-PEX40137
https://www.amazon.com/dp/B07NFRXQHC

​

This one I suspect to be a knockoff. This is the one that doesn't work with the SuperMicro motherboard and won't flash. Shows up as having IT firmware.
https://www.newegg.com/p/14G-0006-00159?Item=9SIA4A0C555953

 

[night vision]

Are you using an adapter that does not provide +3.3V when testing them outside the cage? Sounds like you were a victim of the industry's silliest feature this side of SGI's fsn: PWDIS, as detailed here, for instance.
If this is indeed the case, you're sort of stuck between modding the cages and/or cables, or replacing the disks.

Forgot to add...

The connector has five pins. 3.3v / G / 5v / G / 12v

 

[jgreco]

I can get it pop up during POST, but the card doesn't exist when you go to flash it. The flash utility cannot find it, but those systems have motherboards that are over 10 years old so that cannot be an option to use here. Everything has to be relatively new. This is a production NAS for home use, not just a server to store some old photos. This is critical data which gets a weekly backup both on and off site.

This is the weirdest paragraph I've read today. The age of the mainboard has relatively little to do with anything. Supermicro boards are built out of parts that are intended to run inside servers 24/7/365 for years on end; they are not like your typical Gigabyte or ASUS crap desktops where you can expect them to be dead within ten years. We have lots of members who have successfully run X7 or X8 (advise against) and X9 gear for many years, and I would certainly still consider a used X9 today. I refurb gear commercially and manage a large fleet of X9 and X10 stuff; I can recall only three systems having failed, all at or past the ten year power-on hours mark. The price of used RAM alone makes it insanely attractive to go X9 E5 and then just stuff the thing with as much RAM as you like. Half a terabyte? No problem!

No one here is building NAS's just to "store some old photos". That's the wrong attitude to have. One of the powerful things about a properly designed TrueNAS system is that it is super-easy to replace the mainboard down the road. You unplug the hard drives and maybe HBA and move them to a new board. Then boot.

 

[Ericloewe]

This one I suspect to be a knockoff. This is the one that doesn't work with the SuperMicro motherboard and won't flash. Shows up as having IT firmware.
https://www.newegg.com/p/14G-0006-00159?Item=9SIA4A0C555953

Does yours have a solid slot cover? We recently had a discussion about those and knockoffs: https://www.truenas.com/community/resources/fake-server-cards.198/

I can get it pop up during POST, but the card doesn't exist when you go to flash it.

That sounds fixable and the linked resource has a couple of tips. There are a few options:

• If the card is already in IT mode, you can easily flash it from Windows/Linux/FreeBSD, in addition to DOS or UEFI.
• If UEFI doesn't work, DOS will often work (and vice-versa, system firmware is weird).
  • In DOS, you'll need to also track down the DOS4GW runtime or something similar, since 480K is, in fact, not enough for everyone.

 

[night vision]

This is the weirdest paragraph I've read today. The age of the mainboard has relatively little to do with anything. Supermicro boards are built out of parts that are intended to run inside servers 24/7/365 for years on end; they are not like your typical Gigabyte or ASUS crap desktops where you can expect them to be dead within ten years. We have lots of members who have successfully run X7 or X8 (advise against) and X9 gear for many years, and I would certainly still consider a used X9 today. I refurb gear commercially and manage a large fleet of X9 and X10 stuff; I can recall only three systems having failed, all at or past the ten year power-on hours mark. The price of used RAM alone makes it insanely attractive to go X9 E5 and then just stuff the thing with as much RAM as you like. Half a terabyte? No problem!

No one here is building NAS's just to "store some old photos". That's the wrong attitude to have. One of the powerful things about a properly designed TrueNAS system is that it is super-easy to replace the mainboard down the road. You unplug the hard drives and maybe HBA and move them to a new board. Then boot.

My apologies. I generally write a great deal, but this time figured I'd just get only the details out. Must have fell short there. :)

I'm trying to keep everything relatively new component wise. And I'm sorry, I just didn't want it to seem like my NAS was just a Windows Home Server box from 2007 hosting some cute photos and the instructions to the learning remote... I mean I am, like many here, a power user.

So the weird part is that it doesn't show up on the Supermicro board during power on self tests, will show up on another motherboard (ASUS) but on that same ASUS board the flash utility won't find the card. It's super weird.

Does yours have a solid slot cover? We recently had a discussion about those and knockoffs: https://www.truenas.com/community/resources/fake-server-cards.198/


That sounds fixable and the linked resource has a couple of tips. There are a few options:

• If the card is already in IT mode, you can easily flash it from Windows/Linux/FreeBSD, in addition to DOS or UEFI.
• If UEFI doesn't work, DOS will often work (and vice-versa, system firmware is weird).
  • In DOS, you'll need to also track down the DOS4GW runtime or something similar, since 480K is, in fact, not enough for everyone.

Solid cover. Here are some photos.

I bought many of my components off Newegg, unfortunately, I bought them from random vendors because Newegg has become a clearing house and just isn't what it used to be...loved that place. Five of eight WD drives were DOA. Anyways, sorry -- I watched The Art of Server and while mine looks genuine, I question it because it had a foil sticker over the LSI logo on the front. Not sure why...and of course now I'm paranoid I got some cheap knockoff with no real way to tell. According to the video, mine is supposed to be genuine.


Did you guys ever see this one?

"gpt header corruption has been detected please check sata mode setting in bios"

I got that last night while swapping out the Supermicro mb with a new ASUS board. Couldn't for the life of me work around the problem so I just stuck the Supermicro board back in.


As frustrating as this is, I'm having so much fun doing this again. It takes me back to the years when we pushed our 200 watt power supplies to their limits with all kinds of drives and components...figuring out which jumper configuration worked with all the daughter cards so you had all your IRQ and DMA channels in order.


Thanks for everything, everyone. I do appreciate the help. :)

 

[Ericloewe]

The Chinese stickers are also rather dodgy.

I watched The Art of Server and while mine looks genuine, I question it because it had a foil sticker over the LSI logo on the front. Not sure why...and of course now I'm paranoid I got some cheap knockoff with no real way to tell. According to the video, mine is supposed to be genuine.

It has a few things fakes like to avoid, like the LSI logos. But I guess it could be a grey-market factory reject, which would explain some functional issues.

Did you guys ever see this one?

"gpt header corruption has been detected please check sata mode setting in bios"

Does that come up at the POST screen? I'd give good odds that it's noise from an overenthusiastic system firmware.

Also, can you get us some more model numbers, namely for the Asus motherboard and for the HDDs, that way we can look into what would be worth checking next.

 

[night vision]

The Chinese stickers are also rather dodgy.

It has a few things fakes like to avoid, like the LSI logos. But I guess it could be a grey-market factory reject, which would explain some functional issues.

Does that come up at the POST screen? I'd give good odds that it's noise from an overenthusiastic system firmware.

Also, can you get us some more model numbers, namely for the Asus motherboard and for the HDDs, that way we can look into what would be worth checking next.

Was thinking the same on the dodginess of my existing HBA, so I ordered a replacement LSI card from The Art of Server because they're inexpensive and I wasn't having any luck with flashing the other card, so I'll assume I was first sold a knockoff or just one that kinda worked. Because they're under $100, should I order a spare to have for later if that card goes kaput, or will I be able to use any card and still read the data?


The server motherboard I'm planning on using is a Super Micro Computer X9SCM-F with an Intel Xeon E3-1230 Sandy Bridge 3.2 GHz.

The desktop motherboard I was using as a temporary is a ASUS Prime B660M-A with an Intel Core i5-12400.

The hard drives are from Western Digital 4TB Red NAS WD40EFAX.

Let me know if there is anything else you need. Parts should arrive by the end of the week, maybe a bit later.



Thanks again,


Phil

 

[jgreco]

Because they're under $100, should I order a spare to have for later if that card goes kaput, or will I be able to use any card and still read the data?

Of the cards we recommend, the primary problem would be is if you had SAS drives that you later tried to hook up to SATA. You can pick any of the recommended LSI HBA's which are compatible with each other, although the newer ones use the newer 12Gbps SAS HD connector, so cabling might need to change.

If you have your HBA properly cooled, it is unlikely you will need a spare. They can and will cook themselves if mistreated though.

 

[HoneyBadger]

Western Digital 4TB Red NAS WD40EFAX

Hello,

Sorry to be the bearer of bad news, but these drives have been identified as using SMR (Shingled Magnetic Recording) and have some potential data integrity risks.

Please see the posting here for some further details, including mitigations - but if you have an opportunity to cancel the order and/or exchange the drives for the CMR (Conventional) "WD Red Plus" line, that is strongly recommended.

Update: WD Red SMR Drive Compatibility with ZFS

Thanks to the FreeNAS community, we uncovered and reported on a ZFS compatibility issue with some capacities (6TB and under) of WD Red drives that use SMR (Shingled Magnetic Recording) technology. Most HDDs use CMR (Conventional Magnetic Recording) technology which works well with ZFS. Below is...

www.truenas.com

 

[Ericloewe]

That said, they should at least power up, since they don't claim to have PWDIS. Bad drive cages perhaps?

The desktop motherboard I was using as a temporary is a ASUS Prime B660M-A with an Intel Core i5-12400.

Hmm, I would've expected that to work to flash an LSI SAS controller, but perhaps they're doing something weird with the SMBuses to the PCIe slots. It's worth checking the system firmware setup menu for options related to PCIe SMBus and trying to disable it.

 

[night vision]

Hello,

Sorry to be the bearer of bad news, but these drives have been identified as using SMR (Shingled Magnetic Recording) and have some potential data integrity risks.

Please see the posting here for some further details, including mitigations - but if you have an opportunity to cancel the order and/or exchange the drives for the CMR (Conventional) "WD Red Plus" line, that is strongly recommended.

Update: WD Red SMR Drive Compatibility with ZFS

Thanks to the FreeNAS community, we uncovered and reported on a ZFS compatibility issue with some capacities (6TB and under) of WD Red drives that use SMR (Shingled Magnetic Recording) technology. Most HDDs use CMR (Conventional Magnetic Recording) technology which works well with ZFS. Below is...

www.truenas.com

Oof! Swapped out for the WD Red Plus ones! Thanks! :)

That said, they should at least power up, since they don't claim to have PWDIS. Bad drive cages perhaps?


Hmm, I would've expected that to work to flash an LSI SAS controller, but perhaps they're doing something weird with the SMBuses to the PCIe slots. It's worth checking the system firmware setup menu for options related to PCIe SMBus and trying to disable it.

Same issue with the new cages. They don't power on by themselves. Now if I hook a bunch of SATA cables to the motherboard and shove them on the backplane of the drive cage, they spin up just fine which indicates it may just be a faulty knock off (or genuine) LSI card. Hard to tell at this point without a working spare on hand.

I'll have to look into the bios later when the new(er) card arrives.