Extremely slow throughput with aoc-sat2-mv8

[kaltag]

Good evening. I have a NAS/SAN build I've been working on for my home lab. I have used freenas for work before and it has performed well and reliably. I seem to have to run into something goofy with this one though. I have 2 aoc-sat2-mv8 PCI-X cards, 8x1TB Seagate hard drives, 8GB ECC DDR2 , Xeon 5120 1.86GHZ, and Supermicro motherboard booting off a USB flash drive. With a clean install of the latest Freenas and 9.1 I am getting extremely slow throughput from the drives on the HBAs. dd reports 5-7MB/sec read and write speeds on any drive. Using the motherboard's onboard SATA controller I get around 100-103MB/sec. Everything seems to be functional though. I built a couple volumes and made them iSCSI targets which have been working, but the speed is terrible. I tried booting off a couple Linux live CDs and the dd speeds were about right at 102-103MB/sec with the onboard and the HBA so this seems like some sort of FreeBSD/driver problem with the HBA. I'm curious if anyone has seen something similar before.

EDIT: I just tried nas4free and got the same slow transfer speeds on the HBA and the onboard controller in AHCI mode. 13MB/sec off the SSD :(. Booted back into freenas and now onboard controller is showing slow speed as well. Maybe I was mistaken before. The onboard control is an Intel 631xESB/632xESB SATA AHCI controller.

 

[kaltag]

Got some more info here. The controller looks like it's being detected properly.

Code:

[root@freenas] ~# dmesg | grep mvs | less
mvs0: <Marvell 88SX6081 SATA controller> port 0x3000-0x30ff mem 0xd8400000-0xd84fffff irq 24 at device 1.0 on pci5
mvs0: Gen-II, 8 3Gbps ports, Port Multiplier supported
mvsch0: <Marvell SATA channel> at channel 0 on mvs0
mvsch1: <Marvell SATA channel> at channel 1 on mvs0
mvsch2: <Marvell SATA channel> at channel 2 on mvs0
mvsch3: <Marvell SATA channel> at channel 3 on mvs0
mvsch4: <Marvell SATA channel> at channel 4 on mvs0
mvsch5: <Marvell SATA channel> at channel 5 on mvs0
mvsch6: <Marvell SATA channel> at channel 6 on mvs0
mvsch7: <Marvell SATA channel> at channel 7 on mvs0
mvs1: <Marvell 88SX6081 SATA controller> port 0x4000-0x40ff mem 0xd8700000-0xd87fffff,0xd8800000-0xd8bfffff irq 16 at device 1.0 on pci8
mvs1: Gen-II, 8 3Gbps ports, Port Multiplier supported
mvsch8: <Marvell SATA channel> at channel 0 on mvs1
mvsch9: <Marvell SATA channel> at channel 1 on mvs1
mvsch10: <Marvell SATA channel> at channel 2 on mvs1
mvsch11: <Marvell SATA channel> at channel 3 on mvs1
mvsch12: <Marvell SATA channel> at channel 4 on mvs1
mvsch13: <Marvell SATA channel> at channel 5 on mvs1
mvsch14: <Marvell SATA channel> at channel 6 on mvs1
mvsch15: <Marvell SATA channel> at channel 7 on mvs1
ada0 at mvsch0 bus 0 scbus0 target 0 lun 0
ada1 at mvsch1 bus 0 scbus1 target 0 lun 0
ada2 at mvsch2 bus 0 scbus2 target 0 lun 0
ada3 at mvsch3 bus 0 scbus3 target 0 lun 0
ada4 at mvsch5 bus 0 scbus5 target 0 lun 0
ada5 at mvsch6 bus 0 scbus6 target 0 lun 0
ada6 at mvsch7 bus 0 scbus7 target 0 lun 0
ada7 at mvsch8 bus 0 scbus8 target 0 lun 0
ada8 at mvsch10 bus 0 scbus10 target 0 lun 0
ada9 at mvsch11 bus 0 scbus11 target 0 lun 0
ada10 at mvsch12 bus 0 scbus12 target 0 lun 0
ada11 at mvsch13 bus 0 scbus13 target 0 lun 0
ada12 at mvsch15 bus 0 scbus15 target 0 lun 0

Drive information.

Code:

ada1 at mvsch1 bus 0 scbus1 target 0 lun 0
ada1: <ST3500320NS SN06> ATA-8 SATA 1.x device
ada1: Serial Number 9QMBANXR
ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 2048bytes)
ada1: Command Queueing enabled
ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)

dd read test from drive.

Code:

[root@freenas] ~# dd if=/dev/ada1 of=/dev/null bs=512
1334622+0 records in
1334622+0 records out
683326464 bytes transferred in 122.443023 secs (5580771 bytes/sec)

gstat during dd.

Code:

dT: 1.001s  w: 1.000s
L(q)  ops/s    r/s  kBps  ms/r    w/s  kBps  ms/w  %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p2
    1  6563  6557  3279    0.2      4    16    1.8  99.7| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ada4
    0      0      0      0    0.0      0      0    0.0    0.0| ada5
    0      5      0      0    0.0      4    16    0.2    4.2| ada6
    0      0      0      0    0.0      0      0    0.0    0.0| ada7
    0      0      0      0    0.0      0      0    0.0    0.0| ada8
    0      5      0      0    0.0      4    16    0.2    4.9| ada9
    0      0      0      0    0.0      0      0    0.0    0.0| ada10
    0      0      0      0    0.0      0      0    0.0    0.0| ada11
    0      5      0      0    0.0      4    16    1.2    3.9| ada12
    0      2      0      0    0.0      2    12    0.1    0.0| ada13
    0      0      0      0    0.0      0      0    0.0    0.0| ada7p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/baf63647-a1
9d-11e3-868f-0030487c5144

diskinfo.

Code:

[root@freenas] ~# diskinfo -ct /dev/ada1
/dev/ada1
        512            # sectorsize
        500107862016    # mediasize in bytes (465G)
        976773168      # mediasize in sectors
        0              # stripesize
        0              # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9QMBANXR        # Disk ident.
I/O command overhead:
        time to read 10MB block      0.101376 sec      =    0.005 msec/sector
        time to read 20480 sectors  5.847401 sec      =    0.286 msec/sector
        calculated command overhead                    =    0.281 msec/sector
Seek times:
        Full stroke:      250 iter in  5.647135 sec =  22.589 msec
        Half stroke:      250 iter in  4.390876 sec =  17.564 msec
        Quarter stroke:  500 iter in  8.783410 sec =  17.567 msec
        Short forward:    400 iter in  2.349221 sec =    5.873 msec
        Short backward:  400 iter in  2.402341 sec =    6.006 msec
        Seq outer:      2048 iter in  0.552527 sec =    0.270 msec
        Seq inner:      2048 iter in  0.575835 sec =    0.281 msec
Transfer rates:
        outside:      102400 kbytes in  1.900363 sec =    53884 kbytes/sec
        middle:        102400 kbytes in  1.203713 sec =    85070 kbytes/sec
        inside:        102400 kbytes in  2.004294 sec =    51090 kbytes/sec

 

[cyberjock]

I'd get a different controller.. it's no secret that Marvells aren't exactly great for FreeBSD.

 

[Durandal]

Could you try Windows/Linux? It seems like the performance is very bad, but it could as cyberjock pointed out, that the controller does not have good support in FreeBSD.

 

[kaltag]

I have tested CentOS and Gentoo. Both show 100-103MB/sec transfer which is what those drives should be getting. I'm not gonna trust that much data to a Windows box. ZFS on Linux is an option I suppose but I am still leery of it. For kicks I tried FreeBSD 10.0 last night and got the same slow result. Gonna try OpenIndiana tonight and see if there is any luck there.

 

[cyberjock]

I have tested CentOS and Gentoo. Both show 100-103MB/sec transfer which is what those drives should be getting. I'm not gonna trust that much data to a Windows box. ZFS on Linux is an option I suppose but I am still leery of it. For kicks I tried FreeBSD 10.0 last night and got the same slow result. Gonna try OpenIndiana tonight and see if there is any luck there.

Sounds like you just validated what I said above.... Marvell support in FreeBSD is crap. ;)

 

[leenux_tux]

I purchased a SuperMicro HBA for my FreeNAS system, which is running a SuperMicro Motherboard. My thinking was "both same manufacturers, will be compatible and I will be good-to-go". I should have done some research and read up on how bad support is for the Marvel controller. I couldn't even get my card to be seen by FreeNAS.

Get yourself an IBM M1015 and flash it to IT mode, plane ind simple. There's lots of information on this forum about them and all over the Internet. Was easy to flash to IT mode, is reasonably priced (I got mine off ebay for £118.00 GBP) and (so far) is rock solid