Hardware Recommendations Guide Discussion Thread Rev 1e) 2017-05-06

[HoneyBadger]

But what's the reliability on those cheapo SSDs? Any time I've gotten a cheap SSD they've failed more than the USB drives I've gotten.

In the four years I've been using FreeNAS on USB drives, I've killed a total of one, and that one was a used one anyway. Get good quality USB drives, you'll be fine.

Of course, a decent SSD will be better than a USB, but if you'd rather not spend the money, I don't think its a big deal.

I've had similar experience with reliable USB 2.0 drives. A non-Micro Sandisk has been happily running my "home photos" box for years now.

If by "cheapo" you mean any of the questionably branded clones originating out of Shenzhen, then yes, those drives are suitable only for skeet shooting. Drives like the 40GB Intel 320s that are frequently recommended are "inexpensive" but not "cheap" - they'll be fine. I've even started to see DC S3500 80GB drives becoming available for under USD$20. Those will be fine and way, way faster.

 

[dasti]

I had all kind of usb stick brands / formats failing on me since 6 years, sandisk, toshiba, kingston, samsung, ....
- same brand and model seems to fail exactly at the same time, I mean the same day - so a mirror will not be enough
- the most compact ones have the shortest life span (and run very hot)
- I have a couple of usb sticks inside one machine is running flowlesly since 6 years

In conclusion according to what I experienced, if you have no choice but use usb sticks, I'd recommand :
- make sure the system pool is on the harddrives
- make backups of your configuration
- do not use micro ones (like sandisk ultrafit...)
- install them directly on the motherboard if possible so they can benefit from a good airflow (you might need a accessory like this
- as a boot drive use a tripple-way mirror with 3 different brands (only supported at install)
- have 2 key ready to install in your drawer

 

[mpoleski]

I need a clarification regarding IDA/SATA/SAS interface/adapter/controller. The hardware recommendation guide says "Mechanical hard drives barely exceed SATA 1.5Gb/s speeds on a good day, meaning that SATA 3Gb/s is more than adequate for most uses. Naturally, SSDs may benefit from SATA 6Gb/s interfaces, particularly when using 10GbE networking." and also regarding additional SATA/SAS connectivity "If more connectivity than is available from the PCH is desired, or if SAS is required (due to the use of expanders, for instance), the only reliable solution is to add..." - meaning that controllers mentioned in following paragraphs are only necessary if there is not enough ports on-board.

I found a benchmark which shows data suggesting that on-board controllers might be unable to get all of performance from HDDs, even when ports are rated high enough. When you Ctrl+F "All SATA controllers are NOT created equal" you will find a table which summarizes performance of a HDD and a SSD connected to a few different controllers:

Code:

1x 2TB a single drive - 1.8 terabytes - Western Digital Black 2TB (WD2002FAEX)

 Asus Sabertooth 990FX sata6 onboard ( w= 39MB/s , rw= 25MB/s , r= 91MB/s )
 SuperMicro X9SRE sata3 onboard      ( w= 31MB/s , rw= 22MB/s , r= 89MB/s )
 LSI MegaRAID 9265-8i sata6 "JBOD"   ( w=130MB/s , rw= 66MB/s , r=150MB/s )

1x 256GB a single drive - 232 gigabytes - Samsung 840 PRO 256GB (MZ-7PD256BW)

 Asus Sabertooth 990FX sata6 onboard ( w=242MB/s , rw=158MB/s , r=533MB/s )
 LSI MegaRAID 9265-8i sata6 "JBOD"   ( w=438MB/s , rw=233MB/s , r=514MB/s )

As you can see from the table, onboard controllers are unable to get performance of LSI controller from the HDD. Even though one of these onboard controllers is able to get high enough (for HDD) performance from the SSD (242MB/s vs 130MB/s). The explanation of this phenomenon is following:

If you are using the motherboard SATA connectors, they are not going to perform as well as a SATA expander or a dedicated raid card. Just because the SATA port says SATA 6 and comes with fancy cables does not mean the port can move data quickly. The onboard chipsets are normally the cheapest silicon the manufacturer can get away with.

I'd like to understand what's going on here. In particular why SATA controller on-board in Asus Sabertooth 990FX, which is capable of reading 242MB/s from the SSD, is reading only 39MB/s from the HDD. Like it's not capping the speed but rather slowing it down by some factor.
How can I judge what is the actual performance of given SATA controller (either on-board or card which I'm considering to buy)?

 

[Arwen]

@mpoleski,
I can't answer all your questions, but I do have a comment. The LSI MegaRAID controller does contain a read & write cache. I don't know if it's used during JBOD mode, but if so, that can warp speed numbers.

Next, even if the built in SATA ports are of good quality, it maters how they are connected. When using multiple SATA ports at the same time, (like in a RAID array), if the SATA controller chip is wired too few PCIe lanes, (or lower speed lanes), that is a limiting factor. Not your case in the example you used above, which used just 1 disk. Think of the connection like a funnel. Here are some numbers;

PCIe 1.x - 2.5Gbps
PCIe 2.x - 5Gbps
PCIe 3.x - 8Gbps, (but lower overhead than 1.x & 2.x)
PCIe 4.x - 16Gbps

So 4 x SSDs that can saturate 6Gbps SATA needs 24Gbps. Do the math. That means;

10 or 11 lanes of PCIe 1.x, (due to the overhead issue of earlier PCIe)
5 or 6 lanes of PCIe 2.x, (due to the overhead issue of earlier PCIe)
3 lanes of PCIe 3.x
2 lanes PCIe 4.x, (new enough that not many systems have it yet)

So a 4 port SATA chip on a desktop board might only use 2 lanes of PCIe 3.x with the assumption you are not going to use all the disks at the same time. But, for a storage appliance, (like FreeNAS), this MATERS.

Thus, the recommendation to use server style system boards, server HBAs, (Host Bus Adapters for disks), and server style software. Even then, performance requirements sometimes need you to look at the block diagram to determine speed of internal components. Like how many PCIe lanes are wired to the SATA controller chip. And what speed those lanes are.

 

[Ericloewe]

There are a few things here that need to be addressed:
The benchmarks you quote are rather light on details about the actual workload for the disk tests. This alone makes the data useless.
Then there's the RAID controller used: it's biasing the results with its cache and, presumably, prefetching. To what degree naturally depends on the workload, which we don't know.

high enough (for HDD) performance from the SSD (242MB/s vs 130MB/s). The explanation of this phenomenon is following:

It is not an explanation, it is a crummy jedi mind trick. Intel's controllers are more or less on par with LSI, although they have more limited bandwidth - the whole PCH shares four PCIe lanes, whereas the LSI controller typically is connected with eight lanes straight to the CPU. Either way, single disks of any variety cannot saturate the controller's up link.

I'd like to understand what's going on here. In particular why SATA controller on-board in Asus Sabertooth 990FX, which is capable of reading 242MB/s from the SSD, is reading only 39MB/s from the HDD.

Those aren't even read speeds, they're writes, as noted by the "w". But they're pretty miserable for a Samsung 840 Pro, so it sounds to me like a decent amount of random I/O is involved in the workload.

tl;dr - The article is poorly written and lacks rigour, so its conclusions aren't trustworthy.

 

[Johnnie Black]

Just a heads up, WD recently release a couple of SMR RED drives, while I haven't tested them with FreeNAS they will likely suffer from the same poor performance, at least during resilvers, as other similar SMR drives, so it might be a good idea to avoid them with FreeNAS.

2TB - WD20EFAX (single 2TB platter)
6TB - WD60EFAX (three 2TB platters)

AFAIK only these two are SMR, all the other currently available RED models are PMR.

These new SMR drives are also available in the Blue range:

2TB - WD20EZAZ
6TB - WD60EZAZ

 

[mpoleski]

The article mentions that cache is disabled:

Bonnie++ can do asynchronous I/O, which means the local disk cache of the 4k drive is heavily utilized with a flush between ZFS commits once every 30 seconds. Since the disk cache can artificially inflate the results we choose to disable drive caches completely using Bonnie++ in synchronous test mode only. Syncing after each write will result in lower benchmark values, but the numbers will more closely resemble a server which is heavily loaded and using all of its RAM.

I don't know if LSI MegaRAID honors synchronous mode.

The number of PCIe lanes is definitely important to look at, but should not matter for that single-drive test.

My point is that I want HDDs to be the bottleneck, not the controller, CPU, etc. Do you think that cache can explain observed phenomenon on files with sizes of 16GB?
If you know sources where I can learn more about performance of controllers, would you share? I don't want to make decisions based on "brand X is good", "brand Y is bad".

 

[Ericloewe]

The article mentions that cache is disabled:

And I don't believe it without a good explanation. I'm going to go out on a limb and assume it's using system calls and not taking over the OS and drivers like malware. There is no "disable the cache on that LSI SAS controller" system call, for a number of excellent reasons. The most the benchmark can do is issue sync writes. The cache on the RAID controller, if non-volatile, is considered an acceptable intermediate state from which the sync write can be returned, so it will almost certainly be in play here.

Do you think that cache can explain observed phenomenon on files with sizes of 16GB?

Yes, of course. Even just caching ZFS metadata (by coincidence, not integrated design) would provide a considerable boost even for files an order of magnitude or two larger than the cache.

 

[virusbcn]

i've read a lot of hardware recommendations documents about freenas, it's clear to me that the best and most reliable controller is an LSI in IT mode, i start looking on Ebay and find this LSI-9240 /Dell Perc H310 ->

https://www.ebay.com/itm/133105907042?ul_noapp=true

I ask to seller about flash this card in IT mode and this is your answer -> Hello friend, this should not be possible. Because it's beyond his scope.

¿¿???

...then what controllers can I buy that are flashable to IT mode ¿??????

 

[HoneyBadger]

The PERC H310 is perfectly flashable to IT mode; with that said, I would strongly recommend buying a tested used pull from a server refurbisher rather than a "new" part of questionable sources that carries a significant risk of being counterfeit.

The HP H220 is another option as it is based on the newer LSI SAS2308 chipset vs the SAS2008 in the Dell PERC H200/H310.

 

[Chris Moore]

I ask to seller about flash this card in IT mode and this is your answer -> Hello friend, this should not be possible. Because it's beyond his scope.

¿¿???

The seller is probably saying it is beyond their capability. It is very possible and not even all that difficult. I have done six or eight of them myself and that is as someone that would pay extra for a card that is already flashed to IT mode.

It might be a little more difficult to source in Europe, but the cards are readily available here in the US in already flashed condition.

https://www.ebay.com/itm/Dell-H310-...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162834659601

These are just as good:

https://www.ebay.com/itm/Genuine-LS...i-P20-IT-Mode-ZFS-FreeNAS-unRAID/162958581156

 

[Chris Moore]

There is documentation in the forum for how to flash the card yourself if you need to buy one that is not yet flashed to IT mode. It is just a matter of loading different firmware into the card.

 

[virusbcn]

Thank you for your help

my main problem is the choice to buy a card, because there are so many and so many options ...
Buying in USA is problematic because there are large customs costs, I live in Spain, Europe and small purchases to China for example there are no customs fees, however all those who come from USA have to pay enough, so I buy a card in USA I have to buy 3 or 4 in China ...

What do you think about this card? ->

https://www.ebay.com/itm/HP-LSI-SAS...133626?hash=item23c1507fba:g:XUsAAOSwtUBdAlCw

 

[Chris Moore]

I have three of those, two of them in my home NAS systems. They work great, after you get the correct firmware flashed into the card.
If you didn't already find it, this guide should help with the process of updating the firmware:

Detailed newcomers' guide to crossflashing LSI 9211/9300/9311 HBA and variants
https://www.ixsystems.com/community...shing-lsi-9211-9300-9311-hba-and-variants.54/

 

[bsodmike]

With regarding to flashing a 9211-8i controller, I've put everything needed here https://github.com/bsodmike/s5clouds8-lsi9211-8i-IR-to-IT-EFI-bootable-usb

Been running this setup for over a year now, been quite stable.

 

[Chris Moore]

One of the nicest things about FreeNAS is that is does not need and in some cases, it does not even benefit from, the latest generation of hardware. It is often possible to build a perfectly serviceable FreeNAS system using an X8 or X9 generation Supermicro server board. I have a pair of newer X10 boards that I have not even connected and configured yet because my existing system is more than up to the task.

Don't over spend for something you don't need.

 

[Ericloewe]

Small update: Work turned monstrously hectic at the tail end of last year and will remain so through February. The promised update is not forgotten.

 

[diversity]

Could anyone please shed some light on this question regarding the importance of ECC error reporting?

ECC reporting important for FreeNAS and ZFS?

LS, My Setup: FreeNAS 11.3-Release CPU: AMD Ryzen 9 3950x Mobo: ASrock x570 Creator (Latest bios 2.10, ECC enabled in settings) Memory: 4 x Kingston KSM26ED8/16ME (ECC)" (64 GB total) According to AMD the AM4 socket does support ECC but it does not support ECC error reporting function. Is...

www.ixsystems.com

I think this post might have been overlooked by people that can actually know an answer.

 

[Yorick]

I am wondering about stability of em(4). The recommendations guide says "any Intel 1Gbe is fine", yet we have someone over yonder with issues on a desktop adapter: https://www.ixsystems.com/community/threads/network-crashes-but-only-at-night.84211/#post-582435 . That might have been hardware failing as well, though. And then there's this bloke, who has to reset his network stack while on em(4) and i219-LM : https://www.ixsystems.com/community...tion-is-the-only-way.84383/page-2#post-584122 . Curious.

 

[aponomarenko]

A new study to estimate and compare failure rates of desktop-class drives (https://github.com/linuxhw/SMART) and enterprise drives (https://github.com/linuxhw/EnterpriseDrive).

And the growing project to test hard drives under BSD (FreeBSD mostly) https://github.com/bsdhw/SMART