Resource icon

SLOG benchmarking and finding the best SLOG

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
NVDIMMs are (were?) a thing for a while, but they seem to have died off in popularity. Viking Technology was making a U.2 device that was DDR4 with flash backing (VT-PM8, VT-PM16) but they got discontinued with no fanfare.

See my post above - the NV1604 is definitely not a plug-and-play solution. Better scratch device than SLOG at this point.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
So far, the P4801x is whiling away its time inside my FreeNAS purgatory. I bet it was looking forward to a long productive career in a data center connected to every high-speed backbone on the planet. Instead, it's stuck in a no-name server in a no-name town with only 1 stop light. At this rate, it soon will only be able to talk via teletext to iXSystems.

Oh well! :p
 
Last edited:

knullzero

Cadet
Joined
Nov 26, 2015
Messages
3
I stood up my new server today and since I used this thread for SLOG device guidance I wanted to share some results. In my system I have six NVMe SSDs, a mixture of Intel Optane P4801X, Samsung SM961, and Intel 660p.

Intel Optane SSD DC P4801X (200G)

Code:
root@freenas[~]# diskinfo -wS /dev/nvd1
/dev/nvd1
    512             # sectorsize
    200049647616    # mediasize in bytes (186G)
    390721968       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPEL1K200GA    # Disk descr.
    PHKM8522004N200E    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:   1034.9 usec/IO =      0.5 Mbytes/s
       1 kbytes:   1041.9 usec/IO =      0.9 Mbytes/s
       2 kbytes:   1243.2 usec/IO =      1.6 Mbytes/s
       4 kbytes:    939.5 usec/IO =      4.2 Mbytes/s
       8 kbytes:   1027.0 usec/IO =      7.6 Mbytes/s
      16 kbytes:   1049.7 usec/IO =     14.9 Mbytes/s
      32 kbytes:   1031.8 usec/IO =     30.3 Mbytes/s
      64 kbytes:   1055.8 usec/IO =     59.2 Mbytes/s
     128 kbytes:   1184.7 usec/IO =    105.5 Mbytes/s
     256 kbytes:   1111.6 usec/IO =    224.9 Mbytes/s
     512 kbytes:   1147.5 usec/IO =    435.7 Mbytes/s
    1024 kbytes:   1288.8 usec/IO =    775.9 Mbytes/s
    2048 kbytes:   1967.1 usec/IO =   1016.7 Mbytes/s
    4096 kbytes:   3039.8 usec/IO =   1315.9 Mbytes/s
    8192 kbytes:   4918.8 usec/IO =   1626.4 Mbytes/s


Samsung SM961 Polaris (512GB)

Code:
root@freenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
    512             # sectorsize
    512110190592    # mediasize in bytes (477G)
    1000215216      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    SAMSUNG MZVKW512HMJP-000H7    # Disk descr.
    S34VNX0JC00872    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:   2338.5 usec/IO =      0.2 Mbytes/s
       1 kbytes:   2153.6 usec/IO =      0.5 Mbytes/s
       2 kbytes:   2260.0 usec/IO =      0.9 Mbytes/s
       4 kbytes:   2180.6 usec/IO =      1.8 Mbytes/s
       8 kbytes:   2358.1 usec/IO =      3.3 Mbytes/s
      16 kbytes:   2408.9 usec/IO =      6.5 Mbytes/s
      32 kbytes:   2504.1 usec/IO =     12.5 Mbytes/s
      64 kbytes:   2646.7 usec/IO =     23.6 Mbytes/s
     128 kbytes:   2765.5 usec/IO =     45.2 Mbytes/s
     256 kbytes:   3141.0 usec/IO =     79.6 Mbytes/s
     512 kbytes:   2996.6 usec/IO =    166.9 Mbytes/s
    1024 kbytes:   2356.5 usec/IO =    424.4 Mbytes/s
    2048 kbytes:   2809.2 usec/IO =    711.9 Mbytes/s
    4096 kbytes:   3890.4 usec/IO =   1028.2 Mbytes/s
    8192 kbytes:   6663.4 usec/IO =   1200.6 Mbytes/s


Intel 660p Series (2TB)

Code:
root@freenas[~]# diskinfo -wS /dev/nvd5
/dev/nvd5
    512             # sectorsize
    2048408248320    # mediasize in bytes (1.9T)
    4000797360      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPEKNW020T8    # Disk descr.
    PHNH8476008A2P0C    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:   2090.6 usec/IO =      0.2 Mbytes/s
       1 kbytes:   2129.8 usec/IO =      0.5 Mbytes/s
       2 kbytes:   2149.2 usec/IO =      0.9 Mbytes/s
       4 kbytes:   2148.7 usec/IO =      1.8 Mbytes/s
       8 kbytes:   2140.9 usec/IO =      3.6 Mbytes/s
      16 kbytes:   2051.9 usec/IO =      7.6 Mbytes/s
      32 kbytes:   2138.4 usec/IO =     14.6 Mbytes/s
      64 kbytes:   2178.1 usec/IO =     28.7 Mbytes/s
     128 kbytes:   2151.2 usec/IO =     58.1 Mbytes/s
     256 kbytes:   2219.9 usec/IO =    112.6 Mbytes/s
     512 kbytes:   2277.0 usec/IO =    219.6 Mbytes/s
    1024 kbytes:   2524.1 usec/IO =    396.2 Mbytes/s
    2048 kbytes:   3295.3 usec/IO =    606.9 Mbytes/s
    4096 kbytes:   4286.0 usec/IO =    933.3 Mbytes/s
    8192 kbytes:   6230.8 usec/IO =   1283.9 Mbytes/s


Honestly, I was expecting a little better for the Optane drives. I went with the 200GB model since the 100GB model had only half the write speed. I will change the sector size at some point and retest.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
My slowest speed with the P4801X was 17MB/s, or about 30x yours. The evidence suggests that something else is likely throttling all of those devices.

Is the Optane module sitting in a PCIe 3.0 x 4 slot? Is the processor not loaded with other stuff? Is the PCIe slot an its lanes dedicated or getting mux'ed with other stuff?
 
Last edited:

knullzero

Cadet
Joined
Nov 26, 2015
Messages
3
I just brought this online a few hours ago so the system is completely idle and updated to the latest stable version of 11.2. The motherboad is a Supermicro X11SDV-8C-TP8F upgraded to the latest BIOS version running on 128GB of PC4-21300 ECC registered RAM.

The Optane drive is running on a Supermicro AOC-SLG3-2M2, off a x8x8 riser, running in a x4x4x4x4 bifurcated PCIe 3.0 x16 slot (so no PLX or anything to worry about). All four NVMe drives are showing up fine (two P4801X and two SM961). BIOS looked good when I set things up but maybe I should double check things again.
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
You definitely have some serious PCIe issues in that system, the Optane and SM961 should be delivering much higher results.

So if I understand correctly you have an x8x8 riser in the x16 slot, and then the x4x4 card in each half of that? That should be x4 per slot, and with no PLX chips involved they shouldn't have excess latency that way. Check for any power-saving settings hiding in the BIOS?

The 660p is a QLC drive and isn't suitable for SLOG purposes so it's not unusual for it to have poor performance here. I assume those are your capacity drives; keep a close eye on their endurance levels as you use them since QLC is still a very young technology relative to even TLC NAND.
 

knullzero

Cadet
Joined
Nov 26, 2015
Messages
3
You definitely have some serious PCIe issues in that system, the Optane and SM961 should be delivering much higher results.

So if I understand correctly you have an x8x8 riser in the x16 slot, and then the x4x4 card in each half of that? That should be x4 per slot, and with no PLX chips involved they shouldn't have excess latency that way. Check for any power-saving settings hiding in the BIOS?

The 660p is a QLC drive and isn't suitable for SLOG purposes so it's not unusual for it to have poor performance here. I assume those are your capacity drives; keep a close eye on their endurance levels as you use them since QLC is still a very young technology relative to even TLC NAND.

Just as I saw your reply I was in BIOS and sure enough the CPU was set to energy efficient! I didn't expect this since all of my X10SDV motherboards for my VM hosts were not set that way by default (but I'll be double checking that now). So I set the system for maximum performance and modified some additional P state info and retested with much better results for the Optane and 660p.

Intel Optane SSD DC P4801X (200G)

Code:
root@freenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
    512             # sectorsize
    200049647616    # mediasize in bytes (186G)
    390721968       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPEL1K200GA    # Disk descr.
    PHKM8522004W200E    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:     24.8 usec/IO =     19.7 Mbytes/s
       1 kbytes:     25.0 usec/IO =     39.1 Mbytes/s
       2 kbytes:     25.3 usec/IO =     77.2 Mbytes/s
       4 kbytes:     22.9 usec/IO =    170.5 Mbytes/s
       8 kbytes:     25.1 usec/IO =    310.8 Mbytes/s
      16 kbytes:     29.8 usec/IO =    523.5 Mbytes/s
      32 kbytes:     41.0 usec/IO =    762.9 Mbytes/s
      64 kbytes:     60.3 usec/IO =   1037.3 Mbytes/s
     128 kbytes:     96.6 usec/IO =   1293.5 Mbytes/s
     256 kbytes:    162.0 usec/IO =   1543.1 Mbytes/s
     512 kbytes:    291.4 usec/IO =   1715.6 Mbytes/s
    1024 kbytes:    551.4 usec/IO =   1813.6 Mbytes/s
    2048 kbytes:   1073.1 usec/IO =   1863.8 Mbytes/s
    4096 kbytes:   2109.1 usec/IO =   1896.5 Mbytes/s
    8192 kbytes:   4183.8 usec/IO =   1912.1 Mbytes/s


Intel 660p Series (2TB)

Code:
root@freenas[~]# diskinfo -wS /dev/nvd5
/dev/nvd5
    512             # sectorsize
    2048408248320    # mediasize in bytes (1.9T)
    4000797360      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPEKNW020T8    # Disk descr.
    PHNH8476008A2P0C    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:    457.0 usec/IO =      1.1 Mbytes/s
       1 kbytes:    436.5 usec/IO =      2.2 Mbytes/s
       2 kbytes:    437.6 usec/IO =      4.5 Mbytes/s
       4 kbytes:    438.4 usec/IO =      8.9 Mbytes/s
       8 kbytes:    437.1 usec/IO =     17.9 Mbytes/s
      16 kbytes:    467.2 usec/IO =     33.4 Mbytes/s
      32 kbytes:    446.4 usec/IO =     70.0 Mbytes/s
      64 kbytes:    469.1 usec/IO =    133.2 Mbytes/s
     128 kbytes:    582.4 usec/IO =    214.6 Mbytes/s
     256 kbytes:    649.5 usec/IO =    384.9 Mbytes/s
     512 kbytes:    787.4 usec/IO =    635.0 Mbytes/s
    1024 kbytes:   1096.0 usec/IO =    912.4 Mbytes/s
    2048 kbytes:   1594.7 usec/IO =   1254.2 Mbytes/s
    4096 kbytes:   2671.1 usec/IO =   1497.5 Mbytes/s
    8192 kbytes:   4833.7 usec/IO =   1655.1 Mbytes/s


The results for the SM961 were not so impressive. I have three of these installed now and I tested each one with pretty similar results.

Samsung SM961 Polaris (512GB)

Code:
root@freenas[~]# diskinfo -wS /dev/nvd3
/dev/nvd3
    512             # sectorsize
    512110190592    # mediasize in bytes (477G)
    1000215216      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    SAMSUNG MZVKW512HMJP-000H7    # Disk descr.
    S34VNX0JC00873    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:   1219.4 usec/IO =      0.4 Mbytes/s
       1 kbytes:   1155.1 usec/IO =      0.8 Mbytes/s
       2 kbytes:   1191.7 usec/IO =      1.6 Mbytes/s
       4 kbytes:   1173.6 usec/IO =      3.3 Mbytes/s
       8 kbytes:   1170.6 usec/IO =      6.7 Mbytes/s
      16 kbytes:   1414.7 usec/IO =     11.0 Mbytes/s
      32 kbytes:   1573.3 usec/IO =     19.9 Mbytes/s
      64 kbytes:   1602.2 usec/IO =     39.0 Mbytes/s
     128 kbytes:   1725.9 usec/IO =     72.4 Mbytes/s
     256 kbytes:   2121.1 usec/IO =    117.9 Mbytes/s
     512 kbytes:   2017.7 usec/IO =    247.8 Mbytes/s
    1024 kbytes:   2052.1 usec/IO =    487.3 Mbytes/s
    2048 kbytes:   2649.1 usec/IO =    755.0 Mbytes/s
    4096 kbytes:   3578.8 usec/IO =   1117.7 Mbytes/s
    8192 kbytes:   6232.6 usec/IO =   1283.6 Mbytes/s


I am looking at some additional settings in the BIOS around PCIe to see if I can get slightly better results for everything.

The two Optane SSDs will serve as SLOGs. The 660p is just to install local VMs on since I am out of SATA ports (plus I was able to get it under $200 which was nice). I had the SM961 SSDs as spares so I thought I would test them to see how they compared to the others. I'll still modify some more and post the results later in the week.

Thanks Constantin and HoneyBadger for your inputs!
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
So happy you got that sorted. Who knew something as esoteric as energy-saving could effectively shut down your NVME module performance.

It's interesting how the majority of the benefit re: the 2000MB/s max. seq. write speed of the 200GB P4801x Optane Module vs. the 100GB version seems to accrue only for large record sizes. For small records, the IOPS and the MB/s transfer speed is only 10% different. That in turn suggests perhaps some limitation on the part of the processor / available lanes for doing all the work?

FWIW, the 100GB module in my use case seems to be wholly sufficient seeing given the single-vdev Z3 pool it's feeding. I imagine if I ever went Chris' route and added more Vdevs that a better case for the 200GB module would emerge. I also noted a huge benefit re: creating a metadata only L2ARC for the most time/data intensive tasks, i.e. external backups and scrubs thereof using rsync. However, I would caution reading too much into that experience unless you also have a similar use case (i.e. dedicated file server with mostly-static data).
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Intel 660p Series (2TB)
This thing appears to be quite slow for small record sizes. Is it normal for an NVME drive to exhibit 437us latency?

I'm getting about 80us with 100Gb Intel s3700 SATA drive connected to a SAS2208 controller in JBOD mode. The disk is overprovisioned to 20% in this test:

Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA100G3T
Serial Number:    BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:    20,006,048,768 bytes [20.0 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu May  2 18:39:07 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da13
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    33553920        # stripesize
    0               # stripeoffset
    2432            # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA INTEL SSDSC2BA10    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     89.3 usec/IO =      5.5 Mbytes/s
       1 kbytes:     90.3 usec/IO =     10.8 Mbytes/s
       2 kbytes:     88.5 usec/IO =     22.1 Mbytes/s
       4 kbytes:     84.0 usec/IO =     46.5 Mbytes/s
       8 kbytes:    110.0 usec/IO =     71.0 Mbytes/s
      16 kbytes:    124.6 usec/IO =    125.4 Mbytes/s
      32 kbytes:    162.0 usec/IO =    193.0 Mbytes/s
      64 kbytes:    313.2 usec/IO =    199.6 Mbytes/s
     128 kbytes:    629.8 usec/IO =    198.5 Mbytes/s
     256 kbytes:   1264.9 usec/IO =    197.6 Mbytes/s
     512 kbytes:   2494.1 usec/IO =    200.5 Mbytes/s
    1024 kbytes:   4995.6 usec/IO =    200.2 Mbytes/s
    2048 kbytes:  10045.5 usec/IO =    199.1 Mbytes/s
    4096 kbytes:  20140.1 usec/IO =    198.6 Mbytes/s
    8192 kbytes:  40346.7 usec/IO =    198.3 Mbytes/s
/dev/da13
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    33553920        # stripesize
    0               # stripeoffset
    2432            # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA INTEL SSDSC2BA10    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     85.7 usec/IO =      5.7 Mbytes/s
       1 kbytes:     82.1 usec/IO =     11.9 Mbytes/s
       2 kbytes:     84.2 usec/IO =     23.2 Mbytes/s
       4 kbytes:     84.3 usec/IO =     46.3 Mbytes/s
       8 kbytes:    106.2 usec/IO =     73.6 Mbytes/s
      16 kbytes:    123.1 usec/IO =    126.9 Mbytes/s
      32 kbytes:    168.1 usec/IO =    185.8 Mbytes/s
      64 kbytes:    312.0 usec/IO =    200.3 Mbytes/s
     128 kbytes:    625.5 usec/IO =    199.8 Mbytes/s
     256 kbytes:   1250.0 usec/IO =    200.0 Mbytes/s
     512 kbytes:   2518.4 usec/IO =    198.5 Mbytes/s
    1024 kbytes:   4995.6 usec/IO =    200.2 Mbytes/s
    2048 kbytes:  10017.4 usec/IO =    199.7 Mbytes/s
    4096 kbytes:  20114.5 usec/IO =    198.9 Mbytes/s
    8192 kbytes:  40448.9 usec/IO =    197.8 Mbytes/s
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA100G3T
Serial Number:    BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:    20,006,048,768 bytes [20.0 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu May  2 18:45:24 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da13
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    2432            # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA INTEL SSDSC2BA10    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     73.8 usec/IO =      6.6 Mbytes/s
       1 kbytes:     76.2 usec/IO =     12.8 Mbytes/s
       2 kbytes:     86.2 usec/IO =     22.7 Mbytes/s
       4 kbytes:     82.9 usec/IO =     47.1 Mbytes/s
       8 kbytes:     98.3 usec/IO =     79.5 Mbytes/s
      16 kbytes:    138.9 usec/IO =    112.5 Mbytes/s
      32 kbytes:    160.4 usec/IO =    194.8 Mbytes/s
      64 kbytes:    315.3 usec/IO =    198.2 Mbytes/s
     128 kbytes:    629.9 usec/IO =    198.4 Mbytes/s
     256 kbytes:   1245.2 usec/IO =    200.8 Mbytes/s
     512 kbytes:   2520.8 usec/IO =    198.4 Mbytes/s
    1024 kbytes:   5065.1 usec/IO =    197.4 Mbytes/s
    2048 kbytes:  10136.1 usec/IO =    197.3 Mbytes/s
    4096 kbytes:  20200.7 usec/IO =    198.0 Mbytes/s
    8192 kbytes:  40412.8 usec/IO =    198.0 Mbytes/s
/dev/da13
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    2432            # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA INTEL SSDSC2BA10    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    101.5 usec/IO =      4.8 Mbytes/s
       1 kbytes:     74.6 usec/IO =     13.1 Mbytes/s
       2 kbytes:     81.0 usec/IO =     24.1 Mbytes/s
       4 kbytes:     77.7 usec/IO =     50.3 Mbytes/s
       8 kbytes:     87.0 usec/IO =     89.8 Mbytes/s
      16 kbytes:    129.3 usec/IO =    120.9 Mbytes/s
      32 kbytes:    169.1 usec/IO =    184.8 Mbytes/s
      64 kbytes:    314.2 usec/IO =    198.9 Mbytes/s
     128 kbytes:    624.3 usec/IO =    200.2 Mbytes/s
     256 kbytes:   1251.8 usec/IO =    199.7 Mbytes/s
     512 kbytes:   2521.9 usec/IO =    198.3 Mbytes/s
    1024 kbytes:   5020.2 usec/IO =    199.2 Mbytes/s
    2048 kbytes:  10150.3 usec/IO =    197.0 Mbytes/s
    4096 kbytes:  20272.1 usec/IO =    197.3 Mbytes/s
    8192 kbytes:  40775.4 usec/IO =    196.2 Mbytes/s


At @HoneyBadger's request I run the test with two different physical sector sizes. First with 512b sectors, then with 4096b. I run both tests twice. It doesn't seem to make much difference.

compared to @Spearfoot's s3700, my results are slightly slower at large record sizes, but faster with small records. My small record results appear to be similar to @Elliot Dierksen's s3500.

Intel specifies 65us write latency for 100Gb s3700.
I assume the rest of the latency comes from the controller?
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I assume the rest of the latency comes from the controller?
Controller, driver, OS, etc. There's a lot in the chain, which is one of the things NVMe tries to minimize.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
This thing appears to be quite slow for small record sizes. Is it normal for an NVME drive to exhibit 437us latency?
A QLC NVMe drive with no power-loss-protection circuitry? Yes, absolutely. ;)

At @HoneyBadger's request I run the test with two different physical sector sizes. First with 512b sectors, then with 4096b. I run both tests twice. It doesn't seem to make much difference.
Is it possible to test this from a directly attached SATA port? I have a feeling the HBA is interfering somehow. It will also take the PCIe/SAS HBA latency out of the picture.
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Is it possible to test this from a directly attached SATA port? I have a feeling the HBA is interfering somehow. It will also take the PCIe/SAS HBA latency out of the picture.

OK, let's replace PCIe/SAS controller latency with QPI/PCH latency.
So I offlined one of my boot drives and put an s3700 in its place in the ODD slot. Like I did via the SAS controller, I ran two tests with each of the 512 and 4096 bytes physical sector sizes.
Bear in mind that my PCH is crippled with 1.5Gbps speed by IBM.
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA100G3T
Serial Number:    BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:    20,006,048,768 bytes [20.0 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Sun May  5 10:39:20 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/ada0
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    38764           # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    INTEL SSDSC2BA100G3T    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    160.2 usec/IO =      3.0 Mbytes/s
       1 kbytes:    164.1 usec/IO =      6.0 Mbytes/s
       2 kbytes:    170.5 usec/IO =     11.5 Mbytes/s
       4 kbytes:    181.0 usec/IO =     21.6 Mbytes/s
       8 kbytes:    212.6 usec/IO =     36.7 Mbytes/s
      16 kbytes:    287.8 usec/IO =     54.3 Mbytes/s
      32 kbytes:    417.0 usec/IO =     74.9 Mbytes/s
      64 kbytes:    670.4 usec/IO =     93.2 Mbytes/s
     128 kbytes:   1156.9 usec/IO =    108.0 Mbytes/s
     256 kbytes:   2182.7 usec/IO =    114.5 Mbytes/s
     512 kbytes:   4137.1 usec/IO =    120.9 Mbytes/s
    1024 kbytes:   8043.9 usec/IO =    124.3 Mbytes/s
    2048 kbytes:  15808.0 usec/IO =    126.5 Mbytes/s
    4096 kbytes:  31369.0 usec/IO =    127.5 Mbytes/s
    8192 kbytes:  62530.4 usec/IO =    127.9 Mbytes/s
/dev/ada0
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    38764           # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    INTEL SSDSC2BA100G3T    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    156.5 usec/IO =      3.1 Mbytes/s
       1 kbytes:    155.2 usec/IO =      6.3 Mbytes/s
       2 kbytes:    164.3 usec/IO =     11.9 Mbytes/s
       4 kbytes:    173.1 usec/IO =     22.6 Mbytes/s
       8 kbytes:    229.4 usec/IO =     34.1 Mbytes/s
      16 kbytes:    287.7 usec/IO =     54.3 Mbytes/s
      32 kbytes:    417.6 usec/IO =     74.8 Mbytes/s
      64 kbytes:    671.7 usec/IO =     93.1 Mbytes/s
     128 kbytes:   1160.6 usec/IO =    107.7 Mbytes/s
     256 kbytes:   2186.3 usec/IO =    114.3 Mbytes/s
     512 kbytes:   4141.3 usec/IO =    120.7 Mbytes/s
    1024 kbytes:   8038.8 usec/IO =    124.4 Mbytes/s
    2048 kbytes:  15855.1 usec/IO =    126.1 Mbytes/s
    4096 kbytes:  31412.3 usec/IO =    127.3 Mbytes/s
    8192 kbytes:  62569.5 usec/IO =    127.9 Mbytes/s
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA100G3T
Serial Number:    BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:    20,006,048,768 bytes [20.0 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Sun May  5 10:47:46 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/ada0
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    38764           # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    INTEL SSDSC2BA100G3T    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    160.9 usec/IO =      3.0 Mbytes/s
       1 kbytes:    163.1 usec/IO =      6.0 Mbytes/s
       2 kbytes:    174.6 usec/IO =     11.2 Mbytes/s
       4 kbytes:    176.9 usec/IO =     22.1 Mbytes/s
       8 kbytes:    209.4 usec/IO =     37.3 Mbytes/s
      16 kbytes:    285.0 usec/IO =     54.8 Mbytes/s
      32 kbytes:    408.7 usec/IO =     76.5 Mbytes/s
      64 kbytes:    675.9 usec/IO =     92.5 Mbytes/s
     128 kbytes:   1183.8 usec/IO =    105.6 Mbytes/s
     256 kbytes:   2186.7 usec/IO =    114.3 Mbytes/s
     512 kbytes:   4133.4 usec/IO =    121.0 Mbytes/s
    1024 kbytes:   8053.4 usec/IO =    124.2 Mbytes/s
    2048 kbytes:  15835.8 usec/IO =    126.3 Mbytes/s
    4096 kbytes:  31377.8 usec/IO =    127.5 Mbytes/s
    8192 kbytes:  62694.8 usec/IO =    127.6 Mbytes/s
/dev/ada0
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    38764           # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    INTEL SSDSC2BA100G3T    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    151.8 usec/IO =      3.2 Mbytes/s
       1 kbytes:    158.4 usec/IO =      6.2 Mbytes/s
       2 kbytes:    160.2 usec/IO =     12.2 Mbytes/s
       4 kbytes:    176.8 usec/IO =     22.1 Mbytes/s
       8 kbytes:    217.8 usec/IO =     35.9 Mbytes/s
      16 kbytes:    279.4 usec/IO =     55.9 Mbytes/s
      32 kbytes:    411.9 usec/IO =     75.9 Mbytes/s
      64 kbytes:    661.2 usec/IO =     94.5 Mbytes/s
     128 kbytes:   1169.2 usec/IO =    106.9 Mbytes/s
     256 kbytes:   2152.0 usec/IO =    116.2 Mbytes/s
     512 kbytes:   4137.1 usec/IO =    120.9 Mbytes/s
    1024 kbytes:   8060.4 usec/IO =    124.1 Mbytes/s
    2048 kbytes:  15820.2 usec/IO =    126.4 Mbytes/s
    4096 kbytes:  31451.2 usec/IO =    127.2 Mbytes/s
    8192 kbytes:  62290.7 usec/IO =    128.4 Mbytes/s

Again no discernible difference between 512 and 4096 bytes physical sector sizes.
The large records speed is obviously restricted by the 1.5Gbps interface speed. But that doesn't explain ~90us PCH latency I'm observing with small records. (155us total latency, 65us of which are from the drive itself).
It takes only 2.7us to transfer 512 bytes over 1.5Gbps interface, or 3.4us including 8b/10b overhead. For 1kb records it will be 6.8us, 2kb - 13.6us etc. We can actually see that increase in interface latency as the record size increases.

And I don't think latency-wise my PCH is particularly bad. Your 320 series ada1 was a bit faster for small sizes (115usec/IO), but there are some tests in this thread where a drive attached directly to the motherboard SATA has 200us latency or worse.
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110

Ctiger256

Dabbler
Joined
Nov 18, 2016
Messages
40
Here are results from my new server with a P4800X in a PCIe slot:

Code:
root@freenas[~]# smartctl -a /dev/nvme0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPED1K375GA
Serial Number:                      PHKS7481007J375AGN
Firmware Version:                   E2010435
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          375,083,606,016 [375 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 77ce000100
Local Time is:                      Tue May  7 07:59:04 2019 EDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    18.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -     512       8         2
 2 -     512      16         2
 3 -    4096       0         0
 4 -    4096       8         0
 5 -    4096      64         0
 6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


Code:
root@freenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        375083606016    # mediasize in bytes (349G)
        732585168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPED1K375GA     # Disk descr.
        PHKS7481007J375AGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     14.6 usec/IO =     33.4 Mbytes/s
           1 kbytes:     14.6 usec/IO =     66.8 Mbytes/s
           2 kbytes:     14.9 usec/IO =    130.8 Mbytes/s
           4 kbytes:     12.0 usec/IO =    326.1 Mbytes/s
           8 kbytes:     13.6 usec/IO =    573.8 Mbytes/s
          16 kbytes:     18.1 usec/IO =    864.2 Mbytes/s
          32 kbytes:     24.7 usec/IO =   1264.5 Mbytes/s
          64 kbytes:     40.7 usec/IO =   1535.0 Mbytes/s
         128 kbytes:     74.5 usec/IO =   1678.4 Mbytes/s
         256 kbytes:    132.2 usec/IO =   1891.3 Mbytes/s
         512 kbytes:    233.7 usec/IO =   2139.8 Mbytes/s
        1024 kbytes:    436.1 usec/IO =   2292.8 Mbytes/s
        2048 kbytes:    851.0 usec/IO =   2350.3 Mbytes/s
        4096 kbytes:   1673.7 usec/IO =   2389.9 Mbytes/s
        8192 kbytes:   3304.5 usec/IO =   2420.9 Mbytes/s
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
That's some serious speed. I hope you have a lot of spinning rust to take advantage of that! :)
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Are there potentially power-saving states or measures in effect here that could be introducing latency?
You were right. I went into BIOS and changed all power settings I could find to high performance ones. Shaved about 20us off.
Via the SAS controller:
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA100G3T
Serial Number:    BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:    20,006,048,768 bytes [20.0 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu May  9 18:28:55 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da13
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    2432            # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA INTEL SSDSC2BA10    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     64.0 usec/IO =      7.6 Mbytes/s
       1 kbytes:     61.6 usec/IO =     15.9 Mbytes/s
       2 kbytes:     64.1 usec/IO =     30.5 Mbytes/s
       4 kbytes:     65.8 usec/IO =     59.3 Mbytes/s
       8 kbytes:     77.1 usec/IO =    101.4 Mbytes/s
      16 kbytes:     95.7 usec/IO =    163.3 Mbytes/s
      32 kbytes:    158.4 usec/IO =    197.2 Mbytes/s
      64 kbytes:    317.6 usec/IO =    196.8 Mbytes/s
     128 kbytes:    644.0 usec/IO =    194.1 Mbytes/s
     256 kbytes:   1260.1 usec/IO =    198.4 Mbytes/s
     512 kbytes:   2490.0 usec/IO =    200.8 Mbytes/s
    1024 kbytes:   5078.6 usec/IO =    196.9 Mbytes/s
    2048 kbytes:   9989.5 usec/IO =    200.2 Mbytes/s
    4096 kbytes:  20038.0 usec/IO =    199.6 Mbytes/s
    8192 kbytes:  40191.6 usec/IO =    199.0 Mbytes/s
/dev/da13
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    2432            # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA INTEL SSDSC2BA10    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     62.6 usec/IO =      7.8 Mbytes/s
       1 kbytes:     55.7 usec/IO =     17.5 Mbytes/s
       2 kbytes:     65.3 usec/IO =     29.9 Mbytes/s
       4 kbytes:     65.7 usec/IO =     59.5 Mbytes/s
       8 kbytes:     77.1 usec/IO =    101.3 Mbytes/s
      16 kbytes:     96.1 usec/IO =    162.6 Mbytes/s
      32 kbytes:    157.6 usec/IO =    198.3 Mbytes/s
      64 kbytes:    313.3 usec/IO =    199.5 Mbytes/s
     128 kbytes:    626.3 usec/IO =    199.6 Mbytes/s
     256 kbytes:   1262.1 usec/IO =    198.1 Mbytes/s
     512 kbytes:   2520.8 usec/IO =    198.4 Mbytes/s
    1024 kbytes:   5045.5 usec/IO =    198.2 Mbytes/s
    2048 kbytes:  10097.7 usec/IO =    198.1 Mbytes/s
    4096 kbytes:  20075.7 usec/IO =    199.2 Mbytes/s
    8192 kbytes:  40137.9 usec/IO =    199.3 Mbytes/s


Via the PCH:
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA100G3T
Serial Number:    BTTV428406CP100FGN
LU WWN Device Id: 5 5cd2e4 04b68e2e8
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL06
User Capacity:    20,006,048,768 bytes [20.0 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Thu May  9 18:41:37 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/ada0
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    38764           # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    INTEL SSDSC2BA100G3T    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    139.3 usec/IO =      3.5 Mbytes/s
       1 kbytes:    143.7 usec/IO =      6.8 Mbytes/s
       2 kbytes:    150.0 usec/IO =     13.0 Mbytes/s
       4 kbytes:    159.4 usec/IO =     24.5 Mbytes/s
       8 kbytes:    196.9 usec/IO =     39.7 Mbytes/s
      16 kbytes:    259.1 usec/IO =     60.3 Mbytes/s
      32 kbytes:    382.6 usec/IO =     81.7 Mbytes/s
      64 kbytes:    639.8 usec/IO =     97.7 Mbytes/s
     128 kbytes:   1126.5 usec/IO =    111.0 Mbytes/s
     256 kbytes:   2122.4 usec/IO =    117.8 Mbytes/s
     512 kbytes:   4081.0 usec/IO =    122.5 Mbytes/s
    1024 kbytes:   7874.4 usec/IO =    127.0 Mbytes/s
    2048 kbytes:  15509.1 usec/IO =    129.0 Mbytes/s
    4096 kbytes:  30766.7 usec/IO =    130.0 Mbytes/s
    8192 kbytes:  61833.2 usec/IO =    129.4 Mbytes/s
/dev/ada0
    512             # sectorsize
    20006048768     # mediasize in bytes (19G)
    39074314        # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    38764           # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    INTEL SSDSC2BA100G3T    # Disk descr.
    BTTV428406CP100FGN    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    143.6 usec/IO =      3.4 Mbytes/s
       1 kbytes:    144.3 usec/IO =      6.8 Mbytes/s
       2 kbytes:    148.8 usec/IO =     13.1 Mbytes/s
       4 kbytes:    159.7 usec/IO =     24.5 Mbytes/s
       8 kbytes:    193.3 usec/IO =     40.4 Mbytes/s
      16 kbytes:    266.9 usec/IO =     58.5 Mbytes/s
      32 kbytes:    387.0 usec/IO =     80.8 Mbytes/s
      64 kbytes:    625.1 usec/IO =    100.0 Mbytes/s
     128 kbytes:   1131.8 usec/IO =    110.4 Mbytes/s
     256 kbytes:   2135.5 usec/IO =    117.1 Mbytes/s
     512 kbytes:   4072.9 usec/IO =    122.8 Mbytes/s
    1024 kbytes:   7972.1 usec/IO =    125.4 Mbytes/s
    2048 kbytes:  15699.7 usec/IO =    127.4 Mbytes/s
    4096 kbytes:  31069.2 usec/IO =    128.7 Mbytes/s
    8192 kbytes:  61854.3 usec/IO =    129.3 Mbytes/s

 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
A QLC NVMe drive with no power-loss-protection circuitry? Yes, absolutely. ;)
Here is Samsung 860 QVO 1Tb. A QLC SATA drive without a power loss protection. (It's not going to be used in the Freenas server. It was bought for other purposes but hasn't been used yet, so I have a chance to test it) 50-55us with small records. The transfer speed maxes out before 490MBps.
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 860 QVO 1TB
Serial Number:    S4CZNF0M265282F
LU WWN Device Id: 5 002538 e40cd344b
Firmware Version: RVQ01B6Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri May 10 12:02:35 2019 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da15
    512             # sectorsize
    1000204886016    # mediasize in bytes (932G)
    1953525168      # mediasize in sectors
    33553920        # stripesize
    0               # stripeoffset
    121601          # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA Samsung SSD 860    # Disk descr.
    S4CZNF0M265282F         # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     53.3 usec/IO =      9.2 Mbytes/s
       1 kbytes:     48.2 usec/IO =     20.3 Mbytes/s
       2 kbytes:     57.7 usec/IO =     33.9 Mbytes/s
       4 kbytes:     50.5 usec/IO =     77.4 Mbytes/s
       8 kbytes:     66.9 usec/IO =    116.8 Mbytes/s
      16 kbytes:     75.0 usec/IO =    208.4 Mbytes/s
      32 kbytes:    104.6 usec/IO =    298.8 Mbytes/s
      64 kbytes:    169.9 usec/IO =    368.0 Mbytes/s
     128 kbytes:    294.3 usec/IO =    424.7 Mbytes/s
     256 kbytes:    582.1 usec/IO =    429.5 Mbytes/s
     512 kbytes:   1068.1 usec/IO =    468.1 Mbytes/s
    1024 kbytes:   2118.4 usec/IO =    472.1 Mbytes/s
    2048 kbytes:   4154.5 usec/IO =    481.4 Mbytes/s
    4096 kbytes:   8200.7 usec/IO =    487.8 Mbytes/s
    8192 kbytes:  16398.7 usec/IO =    487.8 Mbytes/s
/dev/da15
    512             # sectorsize
    1000204886016    # mediasize in bytes (932G)
    1953525168      # mediasize in sectors
    33553920        # stripesize
    0               # stripeoffset
    121601          # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    ATA Samsung SSD 860    # Disk descr.
    S4CZNF0M265282F         # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     54.4 usec/IO =      9.0 Mbytes/s
       1 kbytes:     54.1 usec/IO =     18.1 Mbytes/s
       2 kbytes:     53.9 usec/IO =     36.2 Mbytes/s
       4 kbytes:     60.0 usec/IO =     65.1 Mbytes/s
       8 kbytes:     67.8 usec/IO =    115.3 Mbytes/s
      16 kbytes:     82.3 usec/IO =    189.9 Mbytes/s
      32 kbytes:    114.3 usec/IO =    273.4 Mbytes/s
      64 kbytes:    168.2 usec/IO =    371.6 Mbytes/s
     128 kbytes:    301.1 usec/IO =    415.2 Mbytes/s
     256 kbytes:    574.3 usec/IO =    435.3 Mbytes/s
     512 kbytes:   1070.6 usec/IO =    467.0 Mbytes/s
    1024 kbytes:   2093.0 usec/IO =    477.8 Mbytes/s
    2048 kbytes:   4205.1 usec/IO =    475.6 Mbytes/s
    4096 kbytes:   8245.5 usec/IO =    485.1 Mbytes/s
    8192 kbytes:  16344.4 usec/IO =    489.5 Mbytes/s
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
I bought a similar Samsung 860 EVO mSATA model as a L2ARC cache. Should be good for that task - fast writes are not required in my application (metadata only storage)
 
Top