Big problem with 10Gb network system

[skyyxy]

Hi guys, Sorry for my bad english , but I really need help with it.
I build my own Freenas server in the office. The base hardwear and system environment is:
Hardwear (Server):
E5-1620 V3
128GB DDR4 RAM
SuperMicro X10srl
Usb dom stick for system
Highpoint 2720 just for HBA
24* Seagate 4TB Sata3 Enterprise HDD with Raid0 (just for test) and build in Freenas
Intel X520(spf+) or X540 10Gb(rj45) network adapter

Hardwear (Client):
Intel I7 3930K
32GB DDR3 Ram
Asus P9X79
1TB hdd for system windows7
Highpoint 2720 Build Raid0 in WebGUI
Seagate 2TB Enterprice HDDS*8 with Raid0 for test

System:
Freenas 9.10 (U1 U2 U4) tested.

Problem:
My office users usually will copy huge data from the client's PC or Mac to the Freenas Server like 4tb - 8tb eachtime.When I try to test transfer date less than 500GB and eveything is perfect the speed is 900-1.1GB/s its really cool.
But When I try to transfer more than 1TB data(acturlly I took 4.44TB data together todo) eachtime together to Freenas server, at first 5XXGB still good around 900-1.1GB/s then rise down to 1xx-2xx MB/s and never rise up again until transfer finish.

I check the report in Freenas:
128GB memory is full and most of RAM is used by system wired
CPU just less 20% uesed
ARC cache up to 120GB and never down
ARC hit ratio high than 85% most of time is high than 95%

Something I tried but sitll same:
Clean all parameters I did in system-tuning option
On or off mtu 9000 in netwrok option
Change the Freenas version: 9.3.1 9.10 9.10.1 u1 u2 u4 but except 10 beta2
Add L2ARC or ZIL cache with 4 SSDs with Raid0
Change the HBA card from highpoint 2720 to lsi 9211-8i
Change the HBA to RAID card(Highpoint 4520 and LSI 9260-8i both with 512MB cache and build Raid0 for 24hdds)
Change the clients system from windows7(smb) to 10 and OSX 10.10.5-10.11.6(afp)

Anybody has experience the it? or something I need to do or else even I can pay for it.
Thanks advance and waiting.

 

[snaptec]

What Switch do you use?
Iperf is constant?
Tried nfs?
Where are you from?
Regards



Gesendet von iPhone mit Tapatalk

 

[skyyxy]

What Switch do you use?
Iperf is constant?
Tried nfs?
Where are you from?
Regards



Gesendet von iPhone mit Tapatalk

Thanks for reply
I'm not use any swith just via directlly by SFP+ or RJ45(CAT7) cables
Doesnt tried nfs because not using FCP7 in mac osx or Unix system
I'm from Shanghai
Doesnt tired Iperf yet but I will try and post the result here
Thank a lot again.

 

[Robert Trevellyan]

Highpoint 2720

Not sure if this is the source of your trouble, but it doesn't looks like a good choice for FreeNAS.

 

[skyyxy]

Not sure if this is the source of your trouble, but it doesn't looks like a good choice for FreeNAS.

Thank you so much Robert
I will try to replace back to lsi 9211-8i and do test again, the reason for HPT 2720 is I can install the HPT HBA webgui in Freenas and its looks much more friendlly than Freenas internal.

 

[Robert Trevellyan]

I can install the HPT HBA webgui in Freenas

Why would you want that?

 

[Ericloewe]

the reason for HPT 2720 is I can install the HPT HBA webgui in Freenas and its looks much more friendlly than Freenas internal.

I'm not even sure how to approach this. That just screams "bad idea" from all angles, in every conceivable detail.

 

[bigphil]

How are your disks laid out in your pool and vdev setup? Have you flashed the 9211-8i to the P20 IT firmware? It almost sounds like you have it in RAID mode...not good if it is. Firmware downloads for the LSI card: https://www.broadcom.com/cs/Satelli...childpagename&pagename=AVG2_Wrapper#downloads

 

[skyyxy]

I'm not even sure how to approach this. That just screams "bad idea" from all angles, in every conceivable detail.

Thanks for your reply, for the hpt webgui just want to test if its working in freenas, if it make freenas unstable that I can remove it in anytime, I will test it.

 

[skyyxy]

How are your disks laid out in your pool and vdev setup? Have you flashed the 9211-8i to the P20 IT firmware? It almost sounds like you have it in RAID mode...not good if it is. Firmware downloads for the LSI card: https://www.broadcom.com/cs/Satellite?c=AVG_Product_C&childpagename=AVG2/AVG2_Layout&cid=1211131586601&d=Touch&locale=avg_en&packedargs=childpagename&pagename=AVG2_Wrapper#downloads

I think forgot do this step, will do it afternoon and test again, Thanks for your advise.

 

[SweetAndLow]

Use only the 9211 for the time being until things are working. All those other cards will cause problems.

Sent from my Nexus 5X using Tapatalk

 

[skyyxy]

Use only the 9211 for the time being until things are working. All those other cards will cause problems.

Sent from my Nexus 5X using Tapatalk

Really? i will try for 9211 to it mode and test. thank you so much.

 

[SweetAndLow]

Really? i will try for 9211 to it mode and test. thank you so much.

Yes, anything that does raid, has a cache or doesn't allow smart data access is not a good choice for using with zfs.

Sent from my Nexus 5X using Tapatalk

 

[skyyxy]

Yes, anything that does raid, has a cache or doesn't allow smart data access is not a good choice for using with zfs.

Sent from my Nexus 5X using Tapatalk

Something is Bad:), I brocken my 9211-8I at last night, but bought the new one with IT mode rom and already on the way, should be get it in 2 days, then I can test your suggest at all. thanks a lot.

 

[skyyxy]

The new LSI 9211-8I has arrived, stating test. Hope working. Thanks a lot everybody.

 

[skyyxy]

Unfortunately same problem like my question. Was replace the HBA card to 9211-8i it mode. anything else I can do? thanks.

 

[snaptec]

Which question do you mean?


Gesendet von iPhone mit Tapatalk

 

[skyyxy]

Which question do you mean?


Gesendet von iPhone mit Tapatalk

Sorry, Its my fault. The question solved. Absoulutly about the HBA card rom. When I flash to IT mode that is working smooth, Thanks a lot everybody

 

[skyyxy]

Hi all, Merry Xmas:) I have a new problem. When I transfer data from server to clients the freenas alert me the errors like the pictures in the attachments, then the copy speed will rise down to 0mb/s or very slow , then few seconds later will up again to normal 900-1000mb/s. Anybody can help? Thanks a lot.

 

[SweetAndLow]

What chassis do you have? Something is still messed up between your disks and controller.

Sent from my Nexus 5X using Tapatalk