My FreeNAS sucks! Slow speed, confused on what to do.

[JenWest1919]

Hello everyone,

I built my FreeNas system last year (2017) with the following specs:

6x 8TB (5400 WD REDS - WD80EFZX) - Not sure why I can't find where (in the GUI) it says how I formatted the discs but I'm assuming it was in a 2 way mirror... (4x4)

SUPERMICRO MBD-X10SRI-F Server Motherboard LGA 2011 R3
https://www.newegg.com/Product/Product.aspx?Item=N82E16813182928

Intel Xeon E5-2630 V4 Broadwell-EP 2.2 GHz 10 x 256KB L2 Cache 25MB L3 Cache LGA 2011-3 85W BX80660E52630V4 Server Processor
https://www.newegg.com/Product/Product.aspx?Item=N82E16819117630

Kingston ValueRAM 32GB (1 x 32GB) DDR4 2400 RAM (Server Memory) ECC Load Reduced DIMM (288-Pin) KVR24L17Q4/32I (Intel Validated)
https://www.newegg.com/Product/Product.aspx?Item=9SIA98C5DB9343

Problem:
Wired connection maxes out around 50 - 100Mbps.

Confusion:
I don't understand what good upgrading to a 10gbe network will do for me if it's already not hitting 1gbe speeds. Did I not buy the right discs? I see read speeds for these drives top out around ~130Mbps.

What to do? New drives? Mobo doesn't have 10gbe support, get a new mobo that does? Instead of a 10gbe card?

Any help is greatly appreciated.

Thank you!

 

[garm]

First of all, no; the read speed of those drives should be well over 1Gbps as they are rated for 178 MBps internal transfer, which is 1400 Mbps. If you have three pairs you should have read speeds of more then 4 Gbps. Are you sure you are seeing speeds of 100 Mbps and not MBps?

 

[SweetAndLow]

Post screen shots of your speeds. Also run a iperf test between your client and server then sever to client. Are you using WiFi?

 

[danb35]

Not sure why I can't find where (in the GUI) it says how I formatted the discs

On the storage screen (what you posted the screen shot of), click on the first entry for "FreeNAS", then click the Volume Status button at the bottom--it looks like a blank sheet of notebook paper.

Mobo doesn't have 10gbe support, get a new mobo that does?

No, a 10G card would make much more sense.

 

[Elliot Dierksen]

Remember that you system will only be as fast as the slowest component. Using iperf/iperf3 to verify that networking is performing well (as has already been mentioned) is key. Otherwise you might interpret lack of speed as a network issue when the problem resides somewhere else. You didn't mention your use case use, or how you access the data. With higher speed networking, have an SLOG becomes really important to get good write speeds if the data being written with synch enabled (like NFS from an ESXi host).

 

[KrisBee]

@JenWest1919 You've said nothing about how you're accessing your data - e.g via iSCSI, Windows shares, etc. Nor how the transfer speeds you are seeing

"Wired connection maxes out around 50 - 100Mbps."

differ say between read and write. Top theoretical speed on a 1gbe network is 125Mbps, so the top of the range you quoted is in line with this - for reads anyway.

It would be worth posting your pool layout, as shown via the GUI as described by @danb35, or at the shell using zpool status

 

[SweetAndLow]

@JenWest1919 You've said nothing about how you're accessing your data - e.g via iSCSI, Windows shares, etc. Nor how the transfer speeds you are seeing differ say between read and write. Top theoretical speed on a 1gbe network is 125Mbps, so the top of the range you quoted is in line with this - for reads anyway.

It would be worth posting your pool layout, as shown via the GUI as described by @danb35, or at the shell using zpool status

Please make sure you understand the difference between 100mbps and 100MB/s. Your previous post shows you don't know the difference. And that's just fine to cause lots of confusion when people have questions like this.

 

[joeschmuck]

Problem:
Wired connection maxes out around 50 - 100Mbps.

I'm onboard with everyone else here. I suspect that you are not reading the results correctly or you could be transferring a ton of small files which can also give the appearance of slow performance. Run some sort of benchmarking tool and tell us what it is and how you performed the test if your results seem slow. I peak out at ~109MB/s in transferring large files, when I transfer lots of small files I can drop below 50MB/s easily.

6x 8TB (5400 WD REDS - WD80EFZX) - Not sure why I can't find where (in the GUI) it says how I formatted the discs but I'm assuming it was in a 2 way mirror... (4x4)

Hum... You stated both that you have six 8TB drives and then state a 2-way mirror of 4x4. You would be short by two hard drives. It seems more likely that you have a RAIDZ2 of the six drives based on capacity. @danb35 has provided you the instructions to figure out the configuration via the GUI. Your answer will doubtfully impact the troubleshooting of your perceived slow network speed.

And a question... Has it always been this slow (assuming it is actually slow)?
If NO then what has changed?

Have you tried a new Ethernet cable? Direct Connections? Cables do go bad as well.

Good Luck

 

[JenWest1919]

Thank you so much for all of your responses!

First of all, no; the read speed of those drives should be well over 1Gbps as they are rated for 178 MBps internal transfer, which is 1400 Mbps. If you have three pairs you should have read speeds of more then 4 Gbps. Are you sure you are seeing speeds of 100 Mbps and not MBps?

Confirming I have three mirrors:

My bad, my speed is in MB/s (I always get this mixed up). I use both on wifi and wired, and my speed on wifi right now caps out at about 20MB/s. It took 2 minutes 58 seconds to transfer a 2.93 GB file. Speed on wired is ~100MB/s.

@JenWest1919 You've said nothing about how you're accessing your data - e.g via iSCSI, Windows shares, etc. Nor how the transfer speeds you are seeing differ say between read and write. Top theoretical speed on a 1gbe network is 125Mbps, so the top of the range you quoted is in line with this - for reads anyway.

It would be worth posting your pool layout, as shown via the GUI as described by @danb35, or at the shell using zpool status

The NAS is being used (mainly) to store raw video, and I'm accessing it via a Windows SMB share through Windows 10.

I swapped out all cables to Cat6a (SSTP/SFTP). However, I'm quite sure the cable in my media channel from my basement through out the house is 5e (which I'm assuming I'll need to replace if I go up to a 10gbe). My NAS is in my basement, next to my router/modem...

Anyway, I'm assuming my speeds are normal now, and that if I want higher speeds I will need to get a new network adapter + swap out the 5e cable? My desktop mobo supports 10gbe, and I ordered the Ubiquiti EdgeRouter 12.

My only confusion now is which network adapter would be appropriate. I read the 10gbe primer but I'm a little confused because the Intel cards talk about being 10BASET and the EdgeRouter uses SFP+ ports...

Thanks again for all of your responses @garm @SweetAndLow @joeschmuck @KrisBee @Elliot Dierksen @danb35 !

 

[SweetAndLow]

So your wired connection is maxing out, which is good. Your WiFi is slow but that's normal. If you get a 10gig nic for your sever will you also get one for your client computer? For the sfp+ pretty you're going to want a chelsio t420/t520 or a mellonox connect x3. These use fsp+ connections the Intel ones use cat7 with rj45 connectors. The 10gig rj45 is more expensive, uses more power so most people use sfp+ stuff.

 

[danb35]

I'm assuming my speeds are normal now, and that if I want higher speeds I will need to get a new network adapter + swap out the 5e cable?

Pretty much--you can't get much over 100 MBytes/sec over GbE. If you're looking to go 10G, you have options; my 10G infrastructure is fiber rather than copper. There's a resource on the subject that would be worth your reading.

 

[JenWest1919]

So your wired connection is maxing out, which is good. Your WiFi is slow but that's normal. If you get a 10gig nic for your sever will you also get one for your client computer? For the sfp+ pretty you're going to want a chelsio t420/t520 or a mellonox connect x3. These use fsp+ connections the Intel ones use cat7 with rj45 connectors. The 10gig rj45 is more expensive, uses more power so most people use sfp+ stuff.

I don't think I'll need one for my client computer, as it has a AQTION AQC107

The Aquantia AQtion AQC107 is a high-performance, 5-speed, 10 G PCI Express (PCIe)–to–Multi-Gig Ethernet controller that integrates PCIe, MAC, and PHY to provide power and space-efficient connectivity for client systems. An x4/x2/x1 PCIe interface enables the AQC107 to easily handle 10 Gbps line-rate performance while delivering maximum flexibility.

I will check out the ones you mentioned. Thanks so much!

 

[joeschmuck]

Glad to hear that things are actually normal.

Cat 6 and Cat 5e will generally provide the same results on a 1Gbit network, no need to replace the Cat5e cables in the house. My FreeNAS machine is also in my basement and I have Cat5e cable throughout.

So if you really desire to upgrade your home to 10gbe then you should expect to replace all your Ethernet cables with CAT6a or CAT7 cable.

I read the 10gbe primer but I'm a little confused because the Intel cards talk about being 10BASET and the EdgeRouter uses SFP+ ports...

The EdgeRouter 12 has RJ45 connections for your 10BASET connections and only two SFP+ ports in which you can plug in any form factor connection you desire, great for adding Fiber connections if you desire.

 

[joeschmuck]

P.S Can you buy some extras and ship them my way :D

 

[JenWest1919]

Glad to hear that things are actually normal.

Cat 6 and Cat 5e will generally provide the same results on a 1Gbit network, no need to replace the Cat5e cables in the house. My FreeNAS machine is also in my basement and I have Cat5e cable throughout.

So if you really desire to upgrade your home to 10gbe then you should expect to replace all your Ethernet cables with CAT6a or CAT7 cable.

The EdgeRouter 12 has RJ45 connections for your 10BASET connections and only two SFP+ ports in which you can plug in any form factor connection you desire, great for adding Fiber connections if you desire.

I'm assuming with the Chelsio I just ordered, I'll be only using 1 SFP+ port (the NAS to the EdgeRouter 12) (no need for second one).

P.S Can you buy some extras and ship them my way :D

It will be a top priority :D

 

[danb35]

I ordered the Ubiquiti EdgeRouter 12.

From what I can tell (i.e., from the datasheet), the EdgeRouter 12 doesn't do 10G. It has ten RJ45 GbE ports, and two SFP (not SFP+) ports. Is there a different model of EdgeRouter 12 you're looking at?

In any event, you don't need a new router just to run a couple of devices on 10G; a switch will do (if you only have two machines you want on 10G, you could even wire them point-to-point). I ran across a small one that could do the job for you nicely.

 

[JenWest1919]

From what I can tell (i.e., from the datasheet), the EdgeRouter 12 doesn't do 10G. It has ten RJ45 GbE ports, and two SFP (not SFP+) ports. Is there a different model of EdgeRouter 12 you're looking at?

In any event, you don't need a new router just to run a couple of devices on 10G; a switch will do (if you only have two machines you want on 10G, you could even wire them point-to-point). I ran across a small one that could do the job for you nicely.

Wow, nice catch.

Welp, looks like I read this the wrong way:

 

[joeschmuck]

I took a quick look at the Ubiquiti website and I too read 10 Gigabit RJ45 ports and to me it was 10GbE, not Ten 1GbE ports. I'd complain against the marketing of this company and see if you could get the switch replaced with the proper 10GbE ports you thought you were purchasing. At a minimum get a full refund so you can afford to buy what you want.

 

[JenWest1919]

I took a quick look at the Ubiquiti website and I too read 10 Gigabit RJ45 ports and to me it was 10GbE, not Ten 1GbE ports. I'd complain against the marketing of this company and see if you could get the switch replaced with the proper 10GbE ports you thought you were purchasing. At a minimum get a full refund so you can afford to buy what you want.

Ubiquiti has no order support whatsoever. I ended up having to send an email to their headquarters, who then emailed someone at the Ubiquiti "store", who then emailed me. I ultimately ended up getting a refund. Just annoying how they have no order support or self order management. I don't think I'd ever order directly from them again.

 

[joeschmuck]

I can't blame you.