Is it possible to test this from a directly attached SATA port? I have a feeling the HBA is interfering somehow. It will also take the PCIe/SAS HBA latency out of the picture.
OK, let's replace PCIe/SAS controller latency with QPI/PCH latency.
So I offlined one of my boot drives and put an s3700 in its place in the ODD slot. Like I did via the SAS controller, I ran two tests with each of the 512 and 4096 bytes physical sector sizes.
Bear in mind that my PCH is crippled with 1.5Gbps speed by IBM.
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model: INTEL SSDSC2BA100G3T
Serial Number: BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id: DELL(tm)
Firmware Version: 5DV1DL06
User Capacity: 20,006,048,768 bytes [20.0 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Sun May 5 10:39:20 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
/dev/ada0
512 # sectorsize
20006048768 # mediasize in bytes (19G)
39074314 # mediasize in sectors
0 # stripesize
0 # stripeoffset
38764 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
INTEL SSDSC2BA100G3T # Disk descr.
BTTV428406CP100FGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Not_Zoned # Zone Mode
Synchronous random writes:
0.5 kbytes: 160.2 usec/IO = 3.0 Mbytes/s
1 kbytes: 164.1 usec/IO = 6.0 Mbytes/s
2 kbytes: 170.5 usec/IO = 11.5 Mbytes/s
4 kbytes: 181.0 usec/IO = 21.6 Mbytes/s
8 kbytes: 212.6 usec/IO = 36.7 Mbytes/s
16 kbytes: 287.8 usec/IO = 54.3 Mbytes/s
32 kbytes: 417.0 usec/IO = 74.9 Mbytes/s
64 kbytes: 670.4 usec/IO = 93.2 Mbytes/s
128 kbytes: 1156.9 usec/IO = 108.0 Mbytes/s
256 kbytes: 2182.7 usec/IO = 114.5 Mbytes/s
512 kbytes: 4137.1 usec/IO = 120.9 Mbytes/s
1024 kbytes: 8043.9 usec/IO = 124.3 Mbytes/s
2048 kbytes: 15808.0 usec/IO = 126.5 Mbytes/s
4096 kbytes: 31369.0 usec/IO = 127.5 Mbytes/s
8192 kbytes: 62530.4 usec/IO = 127.9 Mbytes/s
/dev/ada0
512 # sectorsize
20006048768 # mediasize in bytes (19G)
39074314 # mediasize in sectors
0 # stripesize
0 # stripeoffset
38764 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
INTEL SSDSC2BA100G3T # Disk descr.
BTTV428406CP100FGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Not_Zoned # Zone Mode
Synchronous random writes:
0.5 kbytes: 156.5 usec/IO = 3.1 Mbytes/s
1 kbytes: 155.2 usec/IO = 6.3 Mbytes/s
2 kbytes: 164.3 usec/IO = 11.9 Mbytes/s
4 kbytes: 173.1 usec/IO = 22.6 Mbytes/s
8 kbytes: 229.4 usec/IO = 34.1 Mbytes/s
16 kbytes: 287.7 usec/IO = 54.3 Mbytes/s
32 kbytes: 417.6 usec/IO = 74.8 Mbytes/s
64 kbytes: 671.7 usec/IO = 93.1 Mbytes/s
128 kbytes: 1160.6 usec/IO = 107.7 Mbytes/s
256 kbytes: 2186.3 usec/IO = 114.3 Mbytes/s
512 kbytes: 4141.3 usec/IO = 120.7 Mbytes/s
1024 kbytes: 8038.8 usec/IO = 124.4 Mbytes/s
2048 kbytes: 15855.1 usec/IO = 126.1 Mbytes/s
4096 kbytes: 31412.3 usec/IO = 127.3 Mbytes/s
8192 kbytes: 62569.5 usec/IO = 127.9 Mbytes/s
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model: INTEL SSDSC2BA100G3T
Serial Number: BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id: DELL(tm)
Firmware Version: 5DV1DL06
User Capacity: 20,006,048,768 bytes [20.0 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Sun May 5 10:47:46 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
/dev/ada0
512 # sectorsize
20006048768 # mediasize in bytes (19G)
39074314 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
38764 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
INTEL SSDSC2BA100G3T # Disk descr.
BTTV428406CP100FGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Not_Zoned # Zone Mode
Synchronous random writes:
0.5 kbytes: 160.9 usec/IO = 3.0 Mbytes/s
1 kbytes: 163.1 usec/IO = 6.0 Mbytes/s
2 kbytes: 174.6 usec/IO = 11.2 Mbytes/s
4 kbytes: 176.9 usec/IO = 22.1 Mbytes/s
8 kbytes: 209.4 usec/IO = 37.3 Mbytes/s
16 kbytes: 285.0 usec/IO = 54.8 Mbytes/s
32 kbytes: 408.7 usec/IO = 76.5 Mbytes/s
64 kbytes: 675.9 usec/IO = 92.5 Mbytes/s
128 kbytes: 1183.8 usec/IO = 105.6 Mbytes/s
256 kbytes: 2186.7 usec/IO = 114.3 Mbytes/s
512 kbytes: 4133.4 usec/IO = 121.0 Mbytes/s
1024 kbytes: 8053.4 usec/IO = 124.2 Mbytes/s
2048 kbytes: 15835.8 usec/IO = 126.3 Mbytes/s
4096 kbytes: 31377.8 usec/IO = 127.5 Mbytes/s
8192 kbytes: 62694.8 usec/IO = 127.6 Mbytes/s
/dev/ada0
512 # sectorsize
20006048768 # mediasize in bytes (19G)
39074314 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
38764 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
INTEL SSDSC2BA100G3T # Disk descr.
BTTV428406CP100FGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Not_Zoned # Zone Mode
Synchronous random writes:
0.5 kbytes: 151.8 usec/IO = 3.2 Mbytes/s
1 kbytes: 158.4 usec/IO = 6.2 Mbytes/s
2 kbytes: 160.2 usec/IO = 12.2 Mbytes/s
4 kbytes: 176.8 usec/IO = 22.1 Mbytes/s
8 kbytes: 217.8 usec/IO = 35.9 Mbytes/s
16 kbytes: 279.4 usec/IO = 55.9 Mbytes/s
32 kbytes: 411.9 usec/IO = 75.9 Mbytes/s
64 kbytes: 661.2 usec/IO = 94.5 Mbytes/s
128 kbytes: 1169.2 usec/IO = 106.9 Mbytes/s
256 kbytes: 2152.0 usec/IO = 116.2 Mbytes/s
512 kbytes: 4137.1 usec/IO = 120.9 Mbytes/s
1024 kbytes: 8060.4 usec/IO = 124.1 Mbytes/s
2048 kbytes: 15820.2 usec/IO = 126.4 Mbytes/s
4096 kbytes: 31451.2 usec/IO = 127.2 Mbytes/s
8192 kbytes: 62290.7 usec/IO = 128.4 Mbytes/s
Again no discernible difference between 512 and 4096 bytes physical sector sizes.
The large records speed is obviously restricted by the 1.5Gbps interface speed. But that doesn't explain ~90us PCH latency I'm observing with small records. (155us total latency, 65us of which are from the drive itself).
It takes only 2.7us to transfer 512 bytes over 1.5Gbps interface, or 3.4us including 8b/10b overhead. For 1kb records it will be 6.8us, 2kb - 13.6us etc. We can actually see that increase in interface latency as the record size increases.
And I don't think latency-wise my PCH is particularly bad. Your
320 series ada1 was a bit faster for small sizes (115usec/IO), but there are some tests in this thread where a drive attached directly to the motherboard SATA has 200us latency or worse.