What speeds should I be getting with 2 WD Red 6 Terr in Mirror VDEV

[Cytomax]

What speeds should i be getting with 2 WD Red 6 Terr in Mirror VDEV

So i have 2 WD Red 6 terrabyte drives in mirror vdev and i am only getting

70 MegaBYTES read

and

90 MegaBYTES write

when transferring files to and from FreeNAS server to my client desktop using SMB which are sitting a few feet apart.

I am transferring 3 separate 2 gig files at the same to get the data.

I removed my mirror vdev and put it in another FreeNAS server and i got the same speeds transferring to a client.

Are my drives my bottle neck or is the bottle neck the hardware for my 2 FreeNAS servers.

I am able to transfer data at a full gigabit between 2 clients in my home network both running Kubuntu 18.04.

Speeds are measured using iptraf-ng which correlates with the transfer d0ialog box in Kubuntu 1804.

Thanks in Advance

Network Gear

Switch Netgear GS108

https://www.netgear.com/support/product/GS108.aspx

Cabling

Monoprice preterminated cat 5e cables 6 feet

FreeNAS Server #1

OS: FreeNAS-11.1-U5 (Fresh install up to date)

CPU: AMD Phenom II X6 1045T

https://products.amd.com/en-us/search/CPU/AMD-Phenom™/AMD-Phenom™-II-X6/1045T/104

Motherboard: ASUS M4A78LT-M LE

https://www.asus.com/Motherboards/M4A78LTM_LE/

RAM: Crucial 8GB Kit (2 x 4GB) DDR3-1600 ECC UDIMM

http://www.crucial.com/usa/en/m4a78lt-m-le/CT5193976

HBA: LSI 9211 8i (updated the latest firmware and set to IT mode)

https://docs.broadcom.com/docs/12353333

Hard Drive: 6 TB RED WD Drives X 2 in RAID 1 ..... 2.5 Terrabytes used

https://www.wdc.com/products/internal-storage/wd-red.html

https://www.wdc.com/content/dam/wdc/website/downloadable_assets/eng/spec_data_sheet/2879-800002.pdf

Network Card: Intel PWLA8391GT Desktop Adapter PRO/1000 GT 10/100/1000Mbps PCI 1 x RJ45

https://www.intel.com/content/www/us/en/desktops/intel-pro-1000-gt-desktop-adapter-datasheet.html

FreeNAS Server #2

OS: FreeNAS-11.1-U5 (Fresh install up to date)

CPU: AMD Phenom™ II X4 980

https://products.amd.com/en-us/search/CPU/AMD-Phenom™/AMD-Phenom™-II-X4-Black/980/74

Motherboard: ASUS M4A78T-E

https://www.asus.com/us/Motherboards/M4A78TE/

RAM: Kingston KTH-PL313E/4G 4 x 4 Gig sticks totaling 16 Gigs of ECC RAM @ DDR-1333

http://www.ec.kingston.com/ecom/hyperx_us/partsinfo.asp?root=&ktcpartno=KTH-PL313E/4G

HBA: LSI 9211 8i (updated the latest firmware and set to IT mode)

https://docs.broadcom.com/docs/12353333

Hard Drive: 6 TB RED WD Drives X 2 in RAID 1 ..... 2.5 Terrabytes used

https://www.wdc.com/products/internal-storage/wd-red.html

https://www.wdc.com/content/dam/wdc/website/downloadable_assets/eng/spec_data_sheet/2879-800002.pdf

Network Card: Intel PWLA8391GT Desktop Adapter PRO/1000 GT 10/100/1000Mbps PCI 1 x RJ45

https://www.intel.com/content/www/us/en/desktops/intel-pro-1000-gt-desktop-adapter-datasheet.html

The Client Computer

OS: Kubuntu 18.04 up to date

CPU: AMD Ryzen™ 5 1600X

https://www.amd.com/en/products/cpu/amd-ryzen-5-1600x

Motherboard: ASRock AB350 Pro4

http://www.asrock.com/mb/AMD/AB350 Pro4/

RAM: GSkill F4-3600C16D-16GVK

https://www.gskill.com/en/product/f4-3600c16d-16gvk

Hard Drive: Crucial MX200 Solid State Drive

http://www.crucial.com/wcsstore/Cru...crucial-mx200-ssd-product-flyer-letter-en.pdf

Network Card: Intel PWLA8391GT Desktop Adapter PRO/1000 GT 10/100/1000Mbps PCI 1 x RJ45

https://www.intel.com/content/www/us/en/desktops/intel-pro-1000-gt-desktop-adapter-datasheet.html

Update 7-10-18

Mellanox ConnectX-2 EN Single Port 10GbE MNPA19-XTR

So i made a direct connection between my client computer and FreeNAS #2 using Mellanox ConnectX-2 EN Single Port 10GbE MNPA19-XTR

Read speeds maxing out at 3 Gigs

and

Write speeds maxing out at 2 Gigs

so definitely a MUCH better transfer speed.

Im wondering why i could not max out 1 Gigabit when using a 6 ' Cat 5e cables that were factory terminated by monoprice.

 

[Chris Moore]

You need to tell what protocol is used, at the very least, SMB or NFS, etc.
If you don't give up some details, it is impossible to get answers.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Cytomax]

im using SMB... i just want to know what the theoretical max from these 2 HD in mirror vdev and what a decent speed in the real world would be

 

[Chris Moore]

With SMB it should be faster than that.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Cytomax]

thank you for the response.. i updated my original post.. let me know if there is anything else you need

 

[Chris Moore]

So i have 2 WD Red 6 terrabyte drives in mirror vdev and i am only getting
70 MegaBYTES read
and
90 MegaBYTES write
when transferring files to and from FreeNAS server to my client desktop using SMB which are sitting a few feet apart.

Are my drives my bottle neck or is the bottle neck the hardware for my 2 FreeNAS servers.

I have a WD 6 TB drive as the data drive in my workstation and a 1 TB Samsung SSD for the boot drive. I can move files from the SSD to WD 6TB drive at around 230 MB/s which is enough faster than 1Gb network speed that the drive should not be the problem. I have noticed that my Linux system has slower connectivity to the FreeNAS than my Windows system over the same network. Often times it is 20 to 30 MB/s slower transferring the same file.
I am not sure why but I think it is a conflict with the configuration between the two systems.


Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[KrisBee]

As a comparison, I can xfer large files, e.g. >= 1GB, from linux box to FreeNAS box at around 117MB/s as shown by systat -if 1 at FreeNAS CLI. Speed drops when transferring data with large numbers of dirs and small files. Linux box has single 7200rpm HDD, FreeNAS box had a 4TB mirror. CPU on FreeNAS box running at circa 3GHz. Realtek NIC on linux box, intel NIC on FreeNAS box and a cheap gigabit switch in-between.

I've seen comments from ixSystems that using "Auxillary Parameters" in the SMB service is voodoo, not needed and/or is counter productive ( e.g: https://redmine.ixsystems.com/issues/24210 )

But I did add this to my SMB service "socket options = TCP_NODELAY IPTOS_LOWDELAY"

The SMB shares are defined with the default vfs objects: "zfs_space,zfsacl,streams_xattr"

Linux mount options: mount -t cifs -o vers=2.1,user=fred,uid=xxxx,gid=xxxx //192.168.0.xx/data /home/fred/CIFS

The dataset in FreeNAS and mount point in linux are both owned by the user fred.

 

[Chris Moore]

im using SMB... i just want to know what the theoretical max from these 2 HD in mirror vdev and what a decent speed in the real world would be

The drives should be able to keep up with 1Gb network line speed which would theoretically be 125MB/s but you loose some speed to overhead, errors, etc. The speeds I usually see on my Windows systems are between 107 and 117MB/s but I never get that good on my Linux systems and I don't know if the problem is on the Linux side or on the FreeNAS side.

As a comparison, I can xfer large files, e.g. >= 1GB, from linux box to FreeNAS box at around 117MB/s

This makes me think there is a simple fix that might make my Linux system faster, but my FreeNAS is connected on a 10Gb port and my Linux system is connected to a 1Gb port, so I would have thought the problem was on the Linux side.

 

[Cytomax]

Well i guess i am going to try 2 different things
I have a couple 10 Gig cards i can use and direct connect the systems together
I have a spare SSD i can install windows 10 on experiment from there

 

[Cytomax]

Update 7-10-18

Mellanox ConnectX-2 EN Single Port 10GbE MNPA19-XTR

So i made a direct connection between my client computer and FreeNAS #2 using Mellanox ConnectX-2 EN Single Port 10GbE MNPA19-XTR

Read speeds maxing out at 3 Gigs

and

Write speeds maxing out at 2 Gigs

so definitely a MUCH better transfer speed.

Im wondering why i could not max out 1 Gigabit when using a 6 ' Cat 5e cables that were factory terminated by monoprice.