IScsi Game Libary

[Clintel]

Hey Guys, just trying to figure out my bottle neck is


Network Card: Intel X520-DA1 E10G41BTDA 10GbE SFP 10GB PCI-E Ethernet Adaptor
Storage: 2 x Sabrent 1TB NVME Striped (Cache/Compression all turned off on pool)

iScsi to Windows 10


Crystal mark of my current single system NVME

Crystal Mark of the iScsi

I know its not going to match, but seems very far off

 

[Samuel Tai]

That's actually not bad. With 10 Gbps Ethernet, the max transfer rate is going to be around 900 MiB/s. You can get closer to the max by enabling jumbo frames. If you want something closer to direct-connected SSDs, you're going to need a faster transport, like 40Gbps or 100Gbps Ethernet.

 

[Clintel]

Thank you @Samuel Tai that has made a big difference!!! :)

can you recommend a card that will run 40Gbps on windows and freenas?
I figure I might as well I've committed so far

 

[Samuel Tai]

Any of the Chelsios have good support in FreeNAS.

 

[Samuel Tai]

Note, you should rebuild your 2x NVMe pool to be a mirror of 2x vdevs, instead of a stripe. As it currently is, you'll lose your pool if a single device fails. You should also enable cache and compression, which will help the mirror get close to the performance of a stripe.

 

[Clintel]

I have a multitude of pools setup the nvme stripe is purely my game libary, so I'm after pure brutal speed


Intel 100hfa016ls could this work? Qsfp 100gb

I can get them at reasonable price, and think even 40gbs would struggle to saturate 2 x nvme

The price is about the same.... so seems logical to go 100 vs 40?

Thank you

 

[Samuel Tai]

Did you check if there's a FreeBSD driver for this card? FreeNAS only comes with 100Gb drivers for Chelsio cxgbe. For Intel, the only driver I could see on my system is the 40 Gb ixl driver.

 

[Clintel]

I appreciate that thank you @Samuel Tai

These are the results I have so far on the 10GB/s with iSCSI

 

[Clintel]

is it possible to add an NVMe cache to multiple pools?

I'm struggling to find anything that says yes! :P but just in case i've missed something

 

[Samuel Tai]

Yes, you can add cache devices to a pool after it's up and running. Under Storage->Pools, locate the relevant pool, and click the gear icon. Select Extend, and at the bottom will be a button to Add Cache.

 

[Clintel]

this would be for a single pool though?

I have 3 pools, is it possible to setup 1 cache for all of them?

 

[Samuel Tai]

Yes, if you have the drives and the connections.

 

[subhuman]

I think there's some miscommunication here. (or maybe I'm the one mis-reading it!)

I think Clintel is asking if he can use a single NVMe SSD as a cache on multiple pools at the same time.

 

[Samuel Tai]

Thanks for the clarification. No, any drive can only be assigned to a single pool in the GUI. It’s possible, using the command line to partition a drive, and then assign the partitions to separate pools, but this isn’t recommended, as this will triple the PCI-E bandwidth consumed, and also shorten the lifespan of the cache drive accordingly.

 

[Clintel]

I think I've settled for now! :)

I've just put the NVMe's in my machine, and put a SSD's in the NAS, this way i can easily saturate the speed with 10GB/s... this is all perfectly suitable for my needs, I just have the mindset of "It can be better"

Thank you for all the help! :)

I'm loving Freenas!!!