Upgrade 4U FreeNAS Server (10Gb etc.)

[Fireball81]

Hey there,

i have build a FreeNAS server several moons ago, which is working as a backup machine for our main production rig.
To give you a brief overview of that the machine is based on, here a quick overview:

Current Server:
- XCase 4U 24bay Case (no expander backplane)
- Supermicro X11SSL-CF
- 32GB DDR4 ECC RAM
- Pentium G4400 2x3,3GHz
- 2xLSI SAS 3008 8i (one embedded and the other as a PCIe card)

I have 2 vdevs, each consisting of 7x8TB (ST8000VN0022) in a RaidZ2 configuration, building a single storage pool.
I am planing for an upgrade of this server, with mainly two goals in mind.

1.) Increase capacity by adding a third vdev of 7x8TB to the storage pool. (or even switching from 2x7x8TB RaidZ2 to 3x8x8 RaidZ2)
2.) Increase performance and overall throughput bandwidth (10GbE)

To increase the capacity i would probably add another SAS 3008, only i can`t, cause i am running out of PCIe slots on my X11SSL-CF, not mentioning the Chelsio T540 (or something equally good) for 10GbE.
Basically that means i either purchase a new chassis with an expander backplane or i upgrade the entire platform to something that offers
more upgrade flexbility.
I tend to do the latter not the former and these are the ideas i have in mind so far:

S2011-3 build:
- XCase 4U 24bay Case (adopt)
- Supermicro X10SRL-F
- 32GB DDR4 ECC RAM (adopt)
- Intel Xeon E5-1620 v4
- 3xLSI SAS 3008 8i (1 of the 3 HBAs will be adopted from the previous configuration)
- Chelsio T540 (hopefully something pops up on ebay)

For a little bit extra upgrade potential i was also considering S3647, e.g:

S3647 build:
- XCase 4U 24bay Case (adopt)
- Supermicro X11SPL-F
- 32GB DDR4 ECC RAM (adopt)
- Intel Xeon Silver 4112, 4x 2.60GHz
- 3xLSI SAS 3008 8i (1 of the 3 HBAs will be adopted from the previous configuration)
- Chelsio T540 (hopefully something pops up on ebay)

The configuration on S3647 would still leave me with a few PCIe slots left for future expansion but on the flipside,
SMB performance would probably favor the E5-1620 v4 over the Xeon Silver 4112. (in theory at least)
What do you guys think? I f you see any (more affordable) alternative, which you would consider a better upgrade path, please let me know.
Thanks for your suggestions and opinions in advance.

Regards
Dennis

 

[Chris Moore]

Increase performance and overall throughput bandwidth (10GbE)

You might be able to do that for sequential but not for random IO and that is a limitation of the mechanical disks. I had that discussion with someone else earlier today.

To increase the capacity i would probably add another SAS 3008

Really, you should sell the second one you have now and use the money from that to buy a SAS expander like this:
https://www.ebay.com/itm/Intel-RES3FV288-SAS-SATA-12Gb-s-RAID-Expander-Controller-8x-Ext-28-Int-Ports/401748144961
You should have used a SAS expander to begin with. Just connect the SAS controller built into the system board to the SAS expander and it splits out the connectivity like a network switch. There is no advantage in having another controller, especially when all you are running is mechanical disks.

 

[Fireball81]

Really, you should sell the second one you have now and use the money from that to buy a SAS expander like this:
https://www.ebay.com/itm/Intel-RES3FV288-SAS-SATA-12Gb-s-RAID-Expander-Controller-8x-Ext-28-Int-Ports/401748144961
You should have used a SAS expander to begin with. Just connect the SAS controller built into the system board to the SAS expander and it splits out the connectivity like a network switch. There is no advantage in having another controller, especially when all you are running is mechanical disks.

Thanks for your reply Chris.
Yeah you`re right. Back in the day when i decided to purchase the additional HBA instead of a SAS expander, it was more a money decision less a performance one. Where i come from (germany) the RES3FV288 wasn`t available back then and even now the cheapest option i can find on ebay.de is 400€. I can ask the seller from your link if international shipping to germany is a possibility.

The only drawback i can see with the RES3FV288 expander is, that i could only connect to the expander with one MiniSAS cable from the LSI 3008, but i guess it probably wouldnt impact my storage pool bandwidth-wise with mechanical disk only.

 

[Stevie_1der]

The only drawback i can see with the RES3FV288 expander is, that i could only connect to the expander with one MiniSAS cable from the LSI 3008, but i guess it probably wouldnt impact my storage pool bandwidth-wise with mechanical disk only.

Why?
You have 2x SFF-8643 connectors on the mainboard, giving you 8x SAS ports.
The expander has 36 ports, 8 input and 28 output (20 internal and 8 external).
If you go with 3x8 disks, you need 24 ports, 20 come from the expander, and 4 from the mainboard SATA ports, and you still have 2 SATA ports for mirrored boot disks or something else.
The 8 external SAS ports can either be connected to another chassis with built-in expander for a later upgrade, or you can route them back into the chassis.

 

[blanchet]

A single SFF-8087 has far enough bandwidth for 24 disks, because it has 4 SAS3 lines (4x12 Gbps = 48 Gbps).
iXsystems uses this setup on their 4U Certified FreeNAS servers

 

[Fireball81]

@Stevie
Yeah thats right, when you include the onboard SATA ports of the X11SSL-CF but if i want to connect 24 disk entirely over the Intel SAS expander
(by using the internal ports) i could only connect one MiniSAS cable to the LSI 3008 but as blanchet already stated i wouldn`t starve my pool in terms of bandwidth, i just thought it could be considered bad practice because the whole storage pool would depend on one SFF-8643 port on the expander.

@blanchet
I didn`t know that this is the configuration iX uses for their certified systems, thanks for sharing.
I already found a good price for the Intel SAS expander from a retailer in my region.
I will probably try to find a Intel Xeon E3-1230 v3 and a Chelsio T520-CR as well and will leave the X11SSL-CF in the system.

 

[Chris Moore]

The only drawback i can see with the RES3FV288 expander is, that i could only connect to the expander with one MiniSAS cable from the LSI 3008, but i guess it probably wouldn't impact my storage pool bandwidth-wise with mechanical disk only.

With 24 drives, it would give you, very roughly, 200MB/s of bandwidth per drive but the thing to keep in mind is that not all drives are delivering data back to the system at full speed all the time and that bandwidth can dynamically adjust as needed. It is really very efficient.

Mechanical drives usually max out at 250MB/s for long sequential reads but work much more slowly for any random access work.

 

[Fireball81]

Thanks Chris.

With that out of the way, i found a few sellers for the Chelsio T520-CR on eBay , most of them seem to be refurbished cards located in chinaq. (dont seem to have the original heatsink on) Can you guys recommend SFP+ modules which from your experience work fine with the T520-Cr and are compatible to the SM10G-SR cause those are very difficult to get for me.

This is what i found by looking a littlbe bit around the web:
https://www.optcore.net/product/chelsio-sm10g-sr-c/

Maybe someone here has experience with those.

Thanks

 

[danb35]

Can you guys recommend SFP+ modules which from your experience work fine with the T520-Cr

I don't have experience with the T520, but the modules from fs.com work perfectly with my T420 cards. About $15 each.

 

[melloa]

I don't have experience with the T520, but the modules from fs.com work perfectly with my T420 cards. About $15 each.

I would 2nd that. I even have some running T310 with a good performance. I'd go with the T420.

 

[blanchet]

The Chelsio T520 NIC is very well supported by FreeNAS (iXsystems uses them in 4U systems).
If you have difficulties to purchase a Chelsio NIC, you can also get an Intel 10GbE NICs. Despite the hardware guide says, they work also very well with FreeNAS.

Warning: many Intel 10GbE NICs for servers are DA version. DA means Direct Attach.
It means that they support only SFP+ Direct Attach Copper Twinax cables.
It is ok for inner rack communication, but they do not work wit an optical transceiver.

 

[Fireball81]

Today i bought the Intel RES3FV288 10xST8000VN0022, the Xeon E3-1230v3 and a Chelsio T520.
I got a bunch of SFF8643 to SFF8087 cables flying aroundf here to use them for the Expander -> Backplane connection.
What i am still missing is the SFP+ modules for the Chelsio NIC, i will probably try those here (https://www.fs.com/products/74668.html) or will try to fetch original modules on ebay.