RAID is not faster than single disk

shou

Cadet
Joined
Sep 29, 2019
Messages
4
Hi, I recently get a HP Gen10 ProLiant Microserver to build a NAS. I bought 4 HDD to build a ZFS pool using RAIDZ, however the write performance is lower than expected. I also tried mirror/stripe, or RAID10, they all seem to have the similar performance, so I wonder the bottleneck might not be ZFS or RAID. (benchmark stats at the end)

CPU usage/load is low during during the IO operations, also the memory usage or ARC usage were high either.

Here is the hardware:

- CPU: AMD Opteron X3216
- Memory: 8GB DDR4 ECC
- Storage Controller: Embedded Marvell 88SE9230 PCIe to SATA 6Gb/s Controller
- Hard Drives: TOSHIBA MG04ACA400N (4TB) x 4 (ada0,ada1,ada2,ada3)
- Boot Drive: SanDisk Extreme USB Disk (32GB) (da0)
- Sytem Drive: TOSHIBA- TR200 SSD (240GB) (da1)

And here is the zpool status:
Code:
$ zpool status
  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: storage
 state: ONLINE
  scan: none requested
config:

    NAME                                          STATE     READ WRITE CKSUM
    storage                                       ONLINE       0     0     0
      gptid/7aae6b35-e327-11e9-9bc5-5480284903b2  ONLINE       0     0     0
      gptid/7c1218b6-e327-11e9-9bc5-5480284903b2  ONLINE       0     0     0
      gptid/7d5d2fb3-e327-11e9-9bc5-5480284903b2  ONLINE       0     0     0
      gptid/7eac58ef-e327-11e9-9bc5-5480284903b2  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: ONLINE
  scan: none requested
config:

    NAME                                          STATE     READ WRITE CKSUM
    system                                        ONLINE       0     0     0
      gptid/33c5fbcb-e294-11e9-9ddf-5480284903b2  ONLINE       0     0     0

errors: No known data errors


- Here is how I benchmark for write performance:

Code:
$ sudo dd if=/dev/urandom of=/mnt/storage/test bs=1M count=3000
3000+0 records in
3000+0 records out
3145728000 bytes transferred in 25.706836 secs (122369319 bytes/sec)

(on another terminal)$ sudo zpool iostat -v 5
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                             767M  27.8G      0      0      0      0
  da0p2                                  767M  27.8G      0      0      0      0
--------------------------------------  -----  -----  -----  -----  -----  -----
storage                                  826M  14.5T      0    418      0   130M
  gptid/7aae6b35-e327-11e9-9bc5-5480284903b2   203M  3.62T      0    107      0  32.2M
  gptid/7c1218b6-e327-11e9-9bc5-5480284903b2   208M  3.62T      0    106      0  33.0M
  gptid/7d5d2fb3-e327-11e9-9bc5-5480284903b2   211M  3.62T      0     94      0  33.2M
  gptid/7eac58ef-e327-11e9-9bc5-5480284903b2   205M  3.62T      0    109      0  31.8M
--------------------------------------  -----  -----  -----  -----  -----  -----
system                                  13.8G   206G      0      0      0      0
  gptid/33c5fbcb-e294-11e9-9ddf-5480284903b2  13.8G   206G      0      0      0      0
--------------------------------------  -----  -----  -----  -----  -----  -----


As you can see here the total write speed on each drive is roughly 30M/s. Then I tried creating UFS directly on each drive and here's the benchmark result:

Code:
$ sudo zpool destroy storage
$ sudo mkdir /mnt/ada{0,1,2,3}
$ for i in {0,1,2,3}; do sudo newfs -U "/dev/ada${i}p2"; done
(output omitted)
$ for i in {0,1,2,3}; do sudo mount /dev/ada${i}p2 /mnt/ada${i}; done
$ for i in {0,1,2,3}; do sudo dd if=/dev/urandom of=/mnt/ada$i/test bs=1M count=2048; done
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 16.578687 secs (129532796 bytes/sec)
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 16.758181 secs (128145390 bytes/sec)
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 16.818135 secs (127688572 bytes/sec)
2048+0 records in
2048+0 records out
2147483648 bytes transferred in 16.899451 secs (127074163 bytes/sec)


Here is more information if needed:

Code:
$ sudo lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Complex
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Wani [Radeon R5/R6/R7 Graphics] (rev 84)
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Host Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Port
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Port
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Host Bridge
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1578
00:09.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 157d
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 20)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 49)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 49)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 4a)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 4a)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 0
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 1
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 2
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 3
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 4
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 5
01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11)
02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe

$ sudo geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Mode: r2w2e4
   descr: TOSHIBA MG04ACA400N
   lunid: 500003984ba82788
   ident: 18R8K6QUFSYC
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Mode: r2w2e4
   descr: TOSHIBA MG04ACA400N
   lunid: 500003984ba82756
   ident: 18R8K6PTFSYC
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Mode: r2w2e4
   descr: TOSHIBA MG04ACA400N
   lunid: 50000397fbb021a6
   ident: 87Q9K4QLFSYC
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: ada3
Providers:
1. Name: ada3
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Mode: r2w2e4
   descr: TOSHIBA MG04ACA400N
   lunid: 500003984b802ff8
   ident: 18R3K6J5FSYC
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: da0
Providers:
1. Name: da0
   Mediasize: 31376707072 (29G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: SanDisk Extreme
   lunid: 6001b4403fc164390000000000000000
   ident: AA010119161727160443
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 240057409536 (224G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: TOSHIBA- TR200
   lunname: TOSHIBA-TR200
   lunid: 50001b8c00050609
   ident: 000012345678
   rotationrate: 0
   fwsectors: 63
   fwheads: 255

Geom name: da2
Providers:
1. Name: da2
   Mediasize: 1000170586112 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: WD Elements 25A2
   lunname: WD      Elements 25A2   WXK1A27DJJCS
   lunid: WD      Elements 25A2   WXK1A27DJJCS
   ident: 57584B31413237444A4A4353
   rotationrate: 5400
   fwsectors: 63
   fwheads: 255


A lot of new things to me, RAID, ZFS, FreeBSD - so if I missed or mistaken anything please kindly point me out :smile:
 

blueether

Patron
Joined
Aug 6, 2018
Messages
259
I'm not going to comment on your testing methods but raidz(x) will have write speeds of about the speed of the worst disk in the array, an array of striped mirrors should be faster, I have 8 SAS 10k rmp drives that in mirrored pairs that preform very well for my needs
 
Last edited:

shou

Cadet
Joined
Sep 29, 2019
Messages
4
Thanks blueether! The benchmark I posted there is the performance for striped four disk array. Yet the write speed is still even worse than any single disk. There must be something wrong isn't it?

Speaking of benchmarking method, I actually tried multiple ways to bench mark it, including:

- Copying a very large file from SSD
- Filling a large file with zero (with compression turned off)
- iozone -a
- Bonnie++

The result are consistent - all around 110MB/s seq write speed for the striped zpool.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742

shou

Cadet
Joined
Sep 29, 2019
Messages
4
I tried reinstalling FreeNAS and the problem is gone. I now receive well over 350MiB/s seq write speed on a striped pool which matches my expectation. I still didn't figure out why though, I'm quite satisfied with the current status. Thanks again - and @JaimieV the links you provided are very informative :tongue:
 
Top