Updating my FreeNAS.... opinions?

Dariusz1989

Contributor
Joined
Aug 22, 2017
Messages
185
Hey

USAGE :
Single human user + 10-20 render nodes. So network rendering, love the L2Arc as all nodes read the same data over and over. Me-user is quite happy with the performance so far... kinda wish the write would be more... when I 1st fired up the NAS it was 1000MB/s write, but then something... glitched... anyway. - Maybe plex?


So Current status

MOBO : MBD-X11SSM-F-O LGA1151
CPU : E3-1260L v5
RAM : 32gb EEC unbuffed - option to push to 64 at some point
HDD : 5x 6TB WD NAS PRO, RAID-Z2
HDD L2 Arc: Samsung 512GB SM961 or 951... can't remember...


NETWORK : Intel x520 da2 sfp+ (I think its Dell-branded)

diskMark via 10gbe sfp+ connection.
Write MB/s Read MB/s
SEQ Q32T1 = 457 1196
SEQ = 338 609


For some reason, I can not increase write speeds no matter what tunable I try. I'm running full jumbo frames - without them, I was getting half the performance above.

In any case...

UPDATE!

8x 10TB WD NAS...
LSI SAS 9207-8i 8 SATA-SAS card - is there any better option? For similar ££ - new ?

Now...

I want to add it as second pool, but not extend my 1st pool - I'm sorry if I name stuff incorrectly, its been a while since I was messing with Freenas. It was working like charm for the past 2 years! :D

In any case ... the second pool, RAID-Z2 or RAID-Z3 ? What do you guys think I should go with? I'm mixed between safety/performance/available disk size... Maybe I should get 1 more 10tb HDD? for 9, Z3?

My MOBO has 8 SATA slots, for some reason 6 white and 2 yellow... I think they might use 2 controllers? So I take I should hock up the 8hdds to LSI for consistency? Or better to MOBO and move the 5REDPRO to LSI instead? Opinions?

Not sure... I'm lost, how are URE at 10tb/8hdd rebuilds?

L2Arc, will my 512gb SSD be shared between the 2 pools or do I need to set it up somehow?

The 10TB drives are RED NAS - no PRO sadly... So I wonder what should I expect of them? Will they be faster/slower than the 5 RED PRO I have? Tricky one...

Also does FreeNAS now allow for extending pool? Say from 8 to 9 drivers? Or it's still a no go?

I think thats it....

TIA !
 
Joined
Oct 18, 2018
Messages
969
HDD L2 Arc: Samsung 512GB SM961 or 951... can't remember...
I'm curious if that is actually negatively affecting you. The index of your L2ARC has to live in main memory and with only 32GB and a massive L2ARC device. I'm not super experience with L2ARC performance but I have seen folks recommend maxing memory before getting an L2ARC and keeping the ratio of sizes lower.
 

lightwave

Explorer
Joined
Jun 14, 2018
Messages
68
My MOBO has 8 SATA slots, for some reason 6 white and 2 yellow... I think they might use 2 controllers?

As far as I can tell, the X11SSM-F has only one SATA controller. The yellow SATA ports on the Supermicro boards are ports with a 5V power pin to be used with SATA DOM / SuperDOM (as an alternative to using a separate power cable)
 

lightwave

Explorer
Joined
Jun 14, 2018
Messages
68
Also does FreeNAS now allow for extending pool? Say from 8 to 9 drivers? Or it's still a no go?

Unfortunately, no. Work on RAIDZ expansion was announced by OpenZFS in 2017, but it still seems to be a work in progress. I suspect it will take some time before it is released*.

Thread about the subject on the forum: https://www.ixsystems.com/community/threads/raidz-expansion-its-happening.58575/

Link to google doc spreadsheet with development status (see row 34): https://docs.google.com/spreadsheets/d/1CFapSYxA5QRFYy5k6ge3FutU7zbAWbaeGN2nKVXgxCI/htmlview

* I have no inside info on the progress of this feature or the release date. My guess that it will take some time is based on the fact that the development status seems to have remained the same for a while. Somebody else on this forum might have more insight into this.
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I’d bump the system RAM to at least 64GB per recommendations set out by jgreco and others. The L2ARC does consume some RAM resources for indexing.

Now, as far as as your past vs. current throughput experience, I suspect it may have to do with how full the drives are. If you want to increase the IOPS of your array, then adding a second VDEV consisting of 6 Z2 drives might be a better option than creating a second pool.

Granted, the new disks rotate slower than the old ones, but I’d suspect that a second vdev will boost performance well beyond the losses due to a slower spinning pool. That will also obviate your question re the L2ARc getting shared or not. I’d always have a dedicated L2ARc, btw. even if partitioning was an option.

I’d take a look at the L2ARc and see how full it is and what the hit rate is. It may make sense to grow the size of the L2ARC pool if the hit rate is high and the pool full.

My use case is very different from yours, I mostly rely on L2ARc to speed browsing and rsync performance. Hence my L2ARc is set to metadata only and consumes about 25GB per TB of pool data. It really boosts rsync tremendously in this mode, especially as the cache gets “hot” (takes a few backups).

Also, keep in mind that the L2ARC is flushed every time you reboot your machine and then needs to be rebuilt.
 

Dariusz1989

Contributor
Joined
Aug 22, 2017
Messages
185
I'm curious if that is actually negatively affecting you. The index of your L2ARC has to live in main memory and with only 32GB and a massive L2ARC device. I'm not super experience with L2ARC performance but I have seen folks recommend maxing memory before getting an L2ARC and keeping the ratio of sizes lower.
Interesting, will look in to it. I don't think I have noticed any large issues yet... So far so good, in case of 2 pools, do I need to do any custom stuff? I read that l2Arc is only per pool so I take I would have to split the arc in to 2 and then assing each half to separate pool ?

As far as I can tell, the X11SSM-F has only one SATA controller. The yellow SATA ports on the Supermicro boards are ports with a 5V power pin to be used with SATA DOM / SuperDOM (as an alternative to using a separate power cable)
Oh I see, so this means I can just connect the sata like normal hdd and use it ? I take if my cable doe not have the pin then it will just work like sata cable ?

I guess the question is, am I better connecting 8hdds to MOBO or to LSI ?

Unfortunately, no. Work on RAIDZ expansion was announced by OpenZFS in 2017, but it still seems to be a work in progress. I suspect it will take some time before it is released*.
Thanks for info! I'm googling this from time to time but I was not sure if I'm up to date. Still fingers cross they add it one day. Would be great feature.

Thanks so much guys for help so far! I'm still tinkering with what to do with rz2/3 options.

EDIT 1 Just spotted another so I'm adding it to reply.

```


I’d bump the system RAM to at least 64GB per recommendations set out by jgreco and others. The L2ARC does consume some RAM resources for indexing.
Its on the TO-BUY list. But not yet.

Now, as far as as your past vs. current throughput experience, I suspect it may have to do with how full the drives are. If you want to increase the IOPS of your array, then adding a second VDEV consisting of 6 Z2 drives might be a better option than creating a second pool.
I don't want to merge the vdevs/have only 1 pool. I prefer to have 2 pools in case one of them fail or something like that. I can always back up some data from the 5hdd pool to 8hdd pool or even place in different nas at a latter date.
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I use z3 because my use case is primarily bit-rot resistant, long term backup. There are performance penalties associated with the additional writes, overhead, etc. In your case, i’d stick to z2 and expand the pool unless there is a very good reason not to.

As for motherboard vs. LSI for connecting to the HDDs, the LSI HBA’s in IT Mode have a bit of a exalted Status here. (FWIW, my pool has a LSI 2116 Controller that came with the motherboard). But with spinning disks, there is unlikely to be a significant difference between intel vs. LSI re SATA performance.

Thus, For the benefit of energy effiency, I’d confirm that the motherboard ports are controlled directly by the CPU (not Marvell controllers). If it’s the cpu, I’d start by using those ports, test, and call it a day if the system works. The LSI units run hot and consume additional power. Why bother unless it’s necessary?
 
Last edited:

Dariusz1989

Contributor
Joined
Aug 22, 2017
Messages
185
I use z3 because my use case is primarily bit-rot resistant, long term backup. There are performance penalties associated with the additional writes, overhead, etc. In your case, i’d stick to z2 and expand the pool unless there is a very good reason not to.
So 8HDD in Z2, at 10Tb should still be "fine" to use despite URE/hardware failures? I looked at my hdds. I have 6 made at the same day + 2 a month early from the labels. I'm bit worried of batching failure. At the moment I'm initializing them 2 at a time and running 1 week before putting all in nas. So that I can delay their "failure" somewhat... if it happens.

As for motherboard vs. LSI for connecting to the HDDs, the LSI HBA’s in IT Mode have a bit of a exalted Status here. (FWIW, my pool has a LSI 2116 Controller that came with the motherboard). But with spinning disks, there is unlikely to be a significant difference between intel vs. LSI re SATA performance.

Thus, For the benefit of energy effiency, I’d confirm that the motherboard ports are controlled directly by the CPU (not Marvell controllers). If it’s the cpu, I’d start by using those ports, test, and call it a day if the system works. The LSI units run hot and consume additional power. Why bother unless it’s necessary?
I'm not sure in what mode it is, I got it off amazon, I need to do a little googling to see what to do with it. Does freenas just give me an option to set it to specific mode or something? Bit lost...

I need LSi tho since I only have 8 ports on board and 5 are already taken.

TIA
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
IT mode is a firmware that the HBA is flashed to. Checking the HBA and updating the firmware to the latest “IT mode” is explained in the resources section, IIRC. The basic idea is to prevent issues between the card and FreeNAS.

I believe you can add a 8-drive z2 vdev to your pool, if that’s what you want to do. But confirm with others and weigh the risks and benefits.

I wouldn’t worry about batch failure (though that can be an issue) as much as the specific model. As flawed as the backblaze published data is, the one takeaway is that while brands May have QC / design / assembly issues in general, specific model numbers may matter more. Some models have very high failure rates.

I use the He10 series, bought them used, have been happy, save for one drive that had a potential infant death issue (SMART errors suggested imminent problems). Replaced that drive, no problems since.
 

Dariusz1989

Contributor
Joined
Aug 22, 2017
Messages
185
IT mode is a firmware that the HBA is flashed to. Checking the HBA and updating the firmware to the latest “IT mode” is explained in the resources section, IIRC. The basic idea is to prevent issues between the card and FreeNAS.

I believe you can add a 8-drive z2 vdev to your pool, if that’s what you want to do. I wouldn’t worry about batch failure as much as the brand / model. Some models have very high failure rates. I use the He10 series, bought them used, have been happy, save for one drive that had a potential infant death issue (SMART errors suggested imminent problems). Replaced that drive, no problems since.

Woah thanks ! Will google flashing the card & try not to break it :D

2 vdevs 2 pools, 1 each. They are WD Red10TB so well see how they go...

By the way does flashing the LSI voids the warranty?

Found this > https://www.ixsystems.com/community...shing-lsi-9211-9300-9311-hba-and-variants.54/ going over it :- )
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
First confirm the mode that the card is in. Should be in the manual or the resource pages here. Don’t update the firmware unless your firmware is not current or the card is not running a “it mode” firmware.

I doubt flashing anything with an official installer from the OEM would ever void a warranty. But anything is possible, I suppose.

Anyhow, you’d download the flash utility and firmware binary from the LSI website at Broadcom or wherever they are now (several acquisitions later). So, not exactly a hack.

LSI wouldn’t publish the stuff on their web site if they didn’t expect their customers to make use of it.
 
Joined
Oct 18, 2018
Messages
969
I don't want to merge the vdevs/have only 1 pool. I prefer to have 2 pools in case one of them fail or something like that. I can always back up some data from the 5hdd pool to 8hdd pool or even place in different nas at a latter date.
Great discussion going on here. This is just a thought, it is worth keeping in mind why pools fail. Pools fail because a vdev fails. Using two vdevs does increase the likelihood of a pool failure over 1 vdev BUT if you do regular scrubs, watch your SMART test results and run them frequently, and attend to any warnings your system gives you the increased risk is only very very minor such that it shouldn't ever impact you. Consider too that a FreeNAS box is not a replacement for a backup. If your PSU catches fire one day and burns your machine it won't matter how many vdevs you have in there. If you have the budget to keep backups then you may feel more free using 1 pool with two vdevs.
 
Last edited:

lightwave

Explorer
Joined
Jun 14, 2018
Messages
68
Oh I see, so this means I can just connect the sata like normal hdd and use it ? I take if my cable doe not have the pin then it will just work like sata cable ?

The yellow connectors should work just like normal SATA connectors if you connect a drive using a SATA cable.

I guess the question is, am I better connecting 8hdds to MOBO or to LSI ?

Don't think it is going to make a whole lot of difference whether you connect the 8 new drives to the MOBO or the LSI. Performance should be the same.
 

Dariusz1989

Contributor
Joined
Aug 22, 2017
Messages
185

lightwave

Explorer
Joined
Jun 14, 2018
Messages
68

lightwave

Explorer
Joined
Jun 14, 2018
Messages
68
Top