Fresh install - slow transfer speeds

[Dariusz1989]

Hey

I was battling with freenas transfer speeds and today I decided to do a "Raw" test.

So I unplug all my hdds + my OS USB.

I plug in m.2 512gb hdd 2GB/s read/1GB/s write more or less.
I use 10gb intel 520x-da2 nics on both my pc and nas.

I installed U2 as that version used to work for me.

So I went ahead and started a transfer test. Before on old setup, I was getting 300mb read/write. Now I'm getting 150Mb/s writes and 300Mb/s read. I'm out of ideas. What to do to get my freenas to utilize full power it has?

When I 1st started with freenas 1-2 months ago On my 1st install using 4 HDDs - standard spining ones wd red pro, I was getting 800Mb/s read and 500Mb/s write. Now on m.2 test I'm getting fraction of what I used to get :/

I don't know what to do/how to troubleshoot it anymore:- (

The 1st test I did before was direct connection PC>NAS. Then I added switch and transfer speed dropped. Now when I did PC>NAS connection again I still have slow speeds... its as if something got damaged...

Supermicro X11SSM-F-O
16gb ECC ram
Xeon E3 1260L - I think
M.2 sm951 512gb
Intel X520-da2


Just to recap
I installed freenas 2 months ago on 4hhds PC>NAS 10gbe connection = 800Mb/s read/500Mb/s write
I updated to be PC>Switch>NAS > 300Mb/s cap
I went back to PC>NAS 300Mb/s cap speed remain
I formated/reinstalled with m.2 > 150Mb/s write/ 300Mb/s read.

Regards
Dariusz

 

[nightshade00013]

Hardware information is extremely important when diagnosing problems. This includes:

• motherboard make and model
• CPU make and model
• RAM quantity
• hard drives, quantity, model numbers, and RAID configuration
• hard disk controllers
• network cards

 

[Ericloewe]

You have to narrow this down before anyone can provide more than generic advice.

Use iperf to measure network performance.

 

[Dariusz1989]

Hey

I'm more than happy to provide more details, Just let me know what those are.

On last install I was getting 300 Mb/s read/write. Now I'm down to 150/300Mb/s...gonna cry here... Anyway attached Iperf screnschot.
iperf with 1 connection = 150Mb/s cap.
iperf with 6 connection = 150Mb/s cap per connection but total 1GB/s...


Jperf in dual mode:

 

[Ericloewe]

Okay, that seems lower than it should be, but it's still nearly an order of magnitude more than 150Mb/s.

Start by going through your previous posts and applying proper units to all the numbers (MB/s vs. Mb/s) so we have a better idea of where to look.

 

[Dariusz1989]

Okay, that seems lower than it should be, but it's still nearly an order of magnitude more than 150Mb/s.

Start by going through your previous posts and applying proper units to all the numbers (MB/s vs. Mb/s) so we have a better idea of where to look.

Hey

Updated posts. Thanks.

 

[Dariusz1989]

Ok some news.... Very exciting!

I found these tunables:
hw.ix.max_interrupt_rate=65536
hw.ix.enable_aim=0
kern.ipc.soacceptqueue=256
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=4194304
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=4194304
net.inet.tcp.sendbuf_inc=4194304

From this post https://forums.freenas.org/index.ph...d-on-fast-hardware-smb-cifs-bottleneck.42641/. I also enabled autotunables as well... The result is PC>NAS transfer 900MB/s-1000MB/s speed (YAY) However NAS>PC is still locaked at 300MB/s... I guess I need more tunables?

Help :- )

Here are my current tunables:


I changed all my user added tunables to be sysctl that seem to fix some of my transfer issues...

Ok this make no sense AT ALL- I restarted nas and now I'm down to 350MB/s/300MB/s write/read transfers! :[

Ok Auto Tune broke it! I'm down to the tunes listed above... that give me 900-1000MB/s write speed to NAS, what tunes do I need to get that same read speed from nas? :- D

Regards
Dariusz

 

[Ericloewe]

You definitely do not want autotune. Get rid of those.

As for the rest, your performance is significantly below where it should be without the tunables, so something weird is going on. Maybe the low-power Xeons have some aggressive throttling that is messing with something that doesn't expect it?

 

[Dariusz1989]

You definitely do not want autotune. Get rid of those.

As for the rest, your performance is significantly below where it should be without the tunables, so something weird is going on. Maybe the low-power Xeons have some aggressive throttling that is messing with something that doesn't expect it?

Hey I was tuning all day today.

So far these are the relevant figures>
kern.ipc.soacceptqueue=2048
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=4194304

Once I put that to this I was getting 600MB/s write speed TO nas. Then I went to my NIC settings on workstation and set the Receive Buffers to 4096 and Transmit Buffers to 16384 That pushed my Write speed to nas to 1000MB/s +.

However I'm still stuck at 300-350MB/s read from NAS. Which is more important to me than write.

My current Tunables are :

kern.ipc.soacceptqueue=2048
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=4194304

net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=4194304
net.inet.tcp.sendbuf_inc=4194304

net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1

net.inet.udp.recvspace=4194304
net.inet.udp.sendspace=4194304

NIC settings > rxcsum txcsum tso4 lro

I will look in to CPU, but last I checked it was set for full turbo.
So far I have around 20 tabs with topics open on this topic. The latest info I take from here > https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning

I wonder if the read NAS>PC speed of 300-350MB/s is an issue on workstation that I somehow have to set windows recv buffers to be higher...

 

[Dariusz1989]

Any1 any suggestions? I'm stick at 300mb read from nas :- (

 

[Dariusz1989]

Bump...