Slow read performance raid 10

[mmplpri]

Hi I got raid 10 from 6x HGST HUH721212AL5200 on 10 gb nic intel 540 read is only between 200-300 mb/s(copying 20 gb movie files).My other pool from 2 ssd got full 1GB/s transfer no problem(bot connected to the same backplane) thats why I know its not smb or network issue before that .I saw in reporting that on transfer only 40% of each hdd is used in busy stats.Scrub is fast as well (4.45T scanned at 2.46G/s, 1.11T issued at 628M/s, 6.39T total).Hdds are 25% full.Server got 64 ddr4 ecc with xeon D1541.Can someone explain me why i only get aroung 250 MB read from them?And the pool got slog device optane 900p pcie.
After Further tests write to this raid 10 is much quicker around 500-600mb and taht use almoust 100% busy in the stats.

 

[Evertb1]

You are better of if you offer some more information about your systems configuration. Especialy about the configuration of your storage. Because if you realy have raid 10 going on you are in for some trouble with your storage.

 

[mmplpri]

Hi my hardware is
ASRock Rack D1540D4U-2T8R with lsi 3008 flashed to it newest firmware
64 DDR 4 2133 ECC
6x 12TB HGST Sas HDD HUH721212AL5200 (512e) with optane 900p pcie as slog and 1 extra hgst as spare
thats setup for them
mirror-0 ONLINE 0 0 0
gptid/79a56c47-03bc-11e9-a3c7-d05099c25b71 ONLINE 0 0 0
gptid/7a7c2589-03bc-11e9-a3c7-d05099c25b71 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/7b62704a-03bc-11e9-a3c7-d05099c25b71 ONLINE 0 0 0
gptid/7c35dcd3-03bc-11e9-a3c7-d05099c25b71 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/7d22cca1-03bc-11e9-a3c7-d05099c25b71 ONLINE 0 0 0
gptid/7dff49b8-03bc-11e9-a3c7-d05099c25b71 ONLINE 0 0 0
logs
gptid/bc49b965-1e4b-11e9-a798-d05099c25b71 ONLINE 0 0 0
spares
gptid/f8604d63-0848-11e9-b9c4-d05099c25b71 AVAIL
2x Seagate 4TB 1200.3 ssd ST3840FM0043 in raid 0
Intel x540 with jumbo frames
Case is RM23612 with sas 12GBs backplane with expander

Ofc all smart tests are ok long and short and scrub 0 errors all hgst hdds got around 1265 hours power on time

DD tests
Write
dd if=/dev/zero of=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 98.526636 secs (638553825 bytes/sec)
Read
dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 181.292199 secs (347034016 bytes/sec)

 

[mmplpri]

I tested it further removed raid 10 and created 3 pairs of mirror on its own and they works as intended
Thats 1 Mirror
62914560000 bytes transferred in 249.303043 secs (252361781 bytes/sec)
root@freenas:/mnt/M1/test # dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 186.721765 secs (336942830 bytes/sec)
Thats 2 Mirror
root@freenas:/mnt/M2/test # dd if=/dev/zero of=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 238.096736 secs (264239490 bytes/sec)
root@freenas:/mnt/M2/test # dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 181.861459 secs (345947736 bytes/sec)
Thats 3 Mirror
root@freenas:/mnt/M3/test # dd if=/dev/zero of=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 258.467904 secs (243413433 bytes/sec)
root@freenas:/mnt/M3/test # dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 195.182976 secs (322336308 bytes/sec)

They around the same speeds but when I created raid 0 from 6 HDDS I got this

root@freenas:/mnt/M1/test # dd if=/dev/zero of=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 39.090262 secs (1609468873 bytes/sec)
root@freenas:/mnt/M1/test # dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 68.098571 secs (923874896 bytes/sec)

Could someone explain why this is happening its really wierd

I check g stat as well and when i write files its use 100% hdds on read hdds are used between 50-98 % its look like they w8 for each other...

 

[dlavigne]

Were you able to figure out the cause for this?

 

[mmplpri]

No Still fight with it I will try to test it with lsi 9400 and see if it make a difference also I recreated raid 10 again and I used 256 sector size it's make it around 720 MB read and write..... so from what i understand it I should got around 1300mb read..... If this lsi won't fix it I will try on another server without backplane just sas to 4xsas cables.

 

[mmplpri]

Now I tried 4 of that 12tb sas hgst hdd on lsi 9400 controler on another server supermicro xeon d1521 with 32 gb ramu without backplane but with cable Mini SAS HD SFF-8643 to 4x SAS 29 Pin Female SFF-8482 (fresh freenas install)and thats results I got from raid 0

root@freenas[/mnt/Bench/test]# dd if=/dev/zero of=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 63.805183 secs (986041521 bytes/sec)
root@freenas[/mnt/Bench/test]# dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 92.596178 secs (679450938 bytes/sec)
Again writes perfectly fine and reads really slow.....And again on Disk usage graphs when on writes shows 100% busy on all disks and on reads only 60% busy... Now i will probably try with others hdds I dont have even slightest clue why this is happening the last thing i will try is another hdds probably...What I observe as well more when is more hdds in array more then difference betwene reads and writes grow...

 

[mmplpri]

Hi anyone got any ideas how to solve my problem im still fighitng with it after many test i didnt get any closer to resolve it now im testing raid 10 with 6 hdds (3x mirrors)
Thats the scores

root@freenas:/mnt/MEDIA/VM_BACKUP # dd if=/dev/zero of=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 120.856929 secs (520570567 bytes/sec)
root@freenas:/mnt/MEDIA/VM_BACKUP # dd of=/dev/zero if=testfile bs=4M count=15000
15000+0 records in
15000+0 records out
62914560000 bytes transferred in 175.326447 secs (358842383 bytes/sec)

W would be really thank full for any tips what else I can try.

 

[MikeyG]

@mmplpri I had a similar issue with 4x mirrors with 8X8TB drives. Read and write performance was stuck at around 500MBps for me which made no sense. Switched to RAIDZ2 and it double to nearl 1GBps- decided I'd rather have the streaming bandwidth than the IOPs. I'd be curious if you find anything. It seems like in terms of streaming performance, mirror vdevs don't work how we'd expect, unless they are SSD drives. I've got 4X500MB SSDs as 2X mirrors in the same system and I easly get 1GBps from those read and write.