Big problem with 10Gb network system

[skyyxy]

What chassis do you have? Something is still messed up between your disks and controller.

Sent from my Nexus 5X using Tapatalk

Its the Chinese brand called guoxing. its 24 disks slots with a LSI chipset for detect 24hdds only 1 sas port onboard. the mean is I just need 1 sas cable connect between lsi9211-8i with the chassis pannel.
Or I need to flash the seagate firmwear or change the disks to WD? or else? Or can suggest me another chassis? Thanks a lot.

 

[snaptec]

Never heard of this.
Supermicro 846 are proven to work. They also have 24 slots and an expander.

No you don't need to flash the hdds

 

[skyyxy]

Never heard of this.
Supermicro 846 are proven to work. They also have 24 slots and an expander.

No you don't need to flash the hdds

Thanks a lot, I will make try todo ,but its quite of diffcult because in China Supermicro dosnt offer pretest service The mean is I need pay and never can back money :-(

 

[Robert Trevellyan]

When I transfer data from server to clients the freenas alert me the errors

Other things to check include the cables, and whether the power supply is stable.

 

[skyyxy]

Other things to check include the cables, and whether the power supply is stable.

Thanks Robert. I checked cable (I have 2 new highpoint sas cable), and the powersupply is DELTA 800W 2 hotbackup model( I dont know how to say in english, its mean the model has 2 powersupply inside, when I disconnect any 1 of them not shutdown the server ).
Now I took 8 of 24hdds outside and use MINI SAS to 4*sata3 cable to test direct link to the lsi9211-8i IT. If this no problem that can prove is about the chassis back pannel(the pannel use LSI 2x28 expander chipset I asked the supplier same as supermicro 846), also I bought a new 9211-8i can try to flash to 9200-8e IT ROM, may be about the Firmwear version?

 

[skyyxy]

Now I quite sure its about Freenas system problem, if I'm wrong please correct me. Thanks advance.
PS:I use Freenas 9.10.1 U4 stable offical version.

Server's’ hardwear：
Intel E5-1620v3
Supermicro X10srl-f
120gb ddr4 ecc
lsi 9211-8i it mode
intel x540 rj45 10GB network
24*st 2tb desktop hdds with lsi 2x28 expander chipset(the 24 hdd docks chassis include)
DELTA 800w powersupply
sandisk 8gb usb stick for Freenas system 9.10.1 U4

Client's hardwear:
Intel e5-2670v1
Supermicro x9sra
16gb ddr3 ecc
samsung 120g ssd for win7 or 10 system
Highpoint 4520,2720, LSI 9260 Raid adapter
8*st 2tb desktop hdds , build with raid0 just for test
Nvidia Q4000 GraphicsCard
Intel X540 rj45 10GB network adapter
Acbel 600w powersupply

The problem is the third point:
1:As I said, At server I took 8 hdds (ST 2tb desktop with 64mb cache) out and use the sas to 4xsata3 cable to connect 9211-8i it mode adapter, that can be sure not use the chassis expander chipset anymore.
2:First:at client, I copy the huge folder (2.7TB, most of it is R3D RAW media file, so its very big for each file) to Freenas server, the speed is very good around 900mb-1.1gb/s (I did some performance tuning in system-tuning option and set network mtu=9014)
3:Second:at client,I copy the same foler from Freenas Server to client, at begining time the speed also very fast around 900mb-1.1gb/s but just few seconds, then rise down to 3xxmb or lower even Zero and very unstable between Zero to 3xxmb/s and never higher again.

Something I did but no change:
1:Off all parameters as I did in System-Tuning options and off mtu 9014 option, even reinstall the whole new system for Freenas 9.10.1 U4
2:Flash the 9211-8i firmwear to offical 9211-8i it again and flash to 9200-8e it.Both of them are P20 version.
3:Change the client's Raid adapter from highpoint 4520 to 2720 and lsi 9260 (all for their Raid0 are build in their webgui or card's bios setting), even I connect the 8 hdds(in client's machine) connect to sata slots onboard (supermicro x9sra)to build Intel ICH softRaid0.
4:Change the client's OS from Windows7 64bit to 10 64bit
5:Rebuild server's and client's Raid0 and check all hdds be sure no bad secotrs.

Now I really dont know what I can do else, somebody can help me? Thanks a lot.

 

[snaptec]

So if you copy from client to freenas everything is ok and if you copy from freenas to client it will drop to 300mb/s ?
Did you make a write speed test if the client is capable writing 1gb/s for 2,7TB ?
What about CPU usage on client and freenas during copy?

 

[skyyxy]

So if you copy from client to freenas everything is ok and if you copy from freenas to client it will drop to 300mb/s ?
Did you make a write speed test if the client is capable writing 1gb/s for 2,7TB ?
What about CPU usage on client and freenas during copy?

Hi Snaptec
1:yes. drop to 300mb or lower each second when I copy huge file from server to client.
2:yes, I tested write speed in client use AJA system test 2.1 offical verion. read and write all for 1.1GB/s.
3:Client's cpu just 9% usage (E5-2670V1 8cores 2.6ghz) and 10-15% usage for server (E5-1620V3 4cores 3.5ghz)
Thanks

 

[snaptec]

Can you please try Iperf for a couple of minutes (see the man page for flags) in both directions?

 

[skyyxy]

Can you please try Iperf for a couple of minutes (see the man page for flags) in both directions?

Sure, I will try and post the result here. Now i'm trying to downgrade the system to U3 U2 U1 or non U :) to check if its about the system version. Thanks

 

[Robert Trevellyan]

The way I see it, when you have a lot of read errors occurring, you should expect poor read performance. Fix the read errors, and the read performance will recover.

 

[skyyxy]

Can you please try Iperf for a couple of minutes (see the man page for flags) in both directions?

Hi Snaptec. Here is the results of write and read performace, but I used dd command because I dont know how to use iperf but I think its same:),please check the attatchment pictures

PS:
444 is write performance
555 if read performance

Now I using 6 st 2tb hdd for raid0, because the office need 2x2tb for copy data.

 

[skyyxy]

The way I see it, when you have a lot of read errors occurring, you should expect poor read performance. Fix the read errors, and the read performance will recover.

Yes, so I was replace all hdds to new and now no read errors but same problem. Is it about the file format what I am copying? what do you think? because When I'm using mac osx and test the performance with AJA system test v2.1 the read and write all good (8xx-9xxMB/s) but for the real read just 3xxmb/s but write still good.

 

[SweetAndLow]

You are running your dd tests against your boot device. You need to run it on your pool and a dataset with no compression. You also need to test using iperf to check your network connection.

Sent from my Nexus 5X using Tapatalk

 

[snaptec]

Dd is local. Iperf is for testing the network line speed

Anyway 2,9 sec dd time is way too short And not a good way to measure speed on a zfs system.

Please read some how to's and do an iperf test over some minutes

Gesendet von iPhone mit Tapatalk

 

[skyyxy]

You are running your dd tests against your boot device. You need to run it on your pool and a dataset with no compression. You also need to test using iperf to check your network connection.

Sent from my Nexus 5X using Tapatalk

thanks alot for that. i will learn and do.

 

[skyyxy]

Dd is local. Iperf is for testing the network line speed

Anyway 2,9 sec dd time is way too short And not a good way to measure speed on a zfs system.

Please read some how to's and do an iperf test over some minutes

Gesendet von iPhone mit Tapatalk

thank a lot. i will try to learn and do test again.

 

[skyyxy]

You are running your dd tests against your boot device. You need to run it on your pool and a dataset with no compression. You also need to test using iperf to check your network connection.

Sent from my Nexus 5X using Tapatalk

I think jperf is much more easy than iperf, Please check the picture in the attachment.

 

[skyyxy]

Dd is local. Iperf is for testing the network line speed

Anyway 2,9 sec dd time is way too short And not a good way to measure speed on a zfs system.

Please read some how to's and do an iperf test over some minutes

Gesendet von iPhone mit Tapatalk

This is the result of jperf, please check it :) thanks

 

[SweetAndLow]

Looks like 4gbps which is way less than 10gbps.

Sent from my Nexus 5X using Tapatalk