Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

SLOG benchmarking and finding the best SLOG

js_level2

Super Moderator
Moderator
iXsystems
Joined
May 4, 2018
Messages
4
Thanks
17
#1
I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you need for a "great" SLOG:

1. Fast throughput
2. Long lifespan with regards to 100% write workloads
3. Fast cache flushes

SLOG workloads are a bit unique. So what goes into an SLOG?

Only sync writes will actually go to an SLOG. A sync write is a write that is being performed that cannot be acknowledged as complete until the write is performed to non-volatile storage. ESXi is particularly rough on this because its NFS implementation defaults to all writes being sync writes as it cannot differentiate from important file system writes inside a VM and some file update that may not be important. This conservatism is what will protect your VMs from corruption if something goes wrong. For this reason, it is typically not a good idea to try to make writes asynchronous.

Samba itself has no sync write function, so you would have to handle that on the ZFS layer. iSCSI supports it, but I have not seen any iSCSI clients that use the sync flag yet. I'm not sure about AFP as it is slowly dying and is less often used with each passing day.

I'll be focusing mostly on NFS sync writes because that's where many people's problems lie.

NFS prior to 11.1 let you use FHA (file handle association). This would have 2 effects:

1. Speed up reads from NFS shares.
2. Slow down writes to NFS shares.

The vice versa happens if you disable FHA.

As you could only turn on FHA for the entire NFS service, you had to pick the one you cared about, and the other was a situation where you "got what you got". The sysctl was vfs.nfsd.fha.enable.

Starting with 11.1, there are new sysctls (thanks to @mav@ at iX for the changes!):

vfs.nfsd.fha.write
vfs.nfsd.fha.read

These allow you to choose the best of both worlds if you desire and if your workloads use FHA properly.

Anyway, to get back on topic, SLOG writes to the SSD involve unique situations for the writes, so it requires very specific benchmarking. Here's how to do it:

1. Make sure the disk is not partitioned.
2. Make sure no data is on the disk. This test is DESTRUCTIVE.
3. Make sure the disk is not in use by anything else ( gstat can be your friend here).

This should only be used on SSDs, and this test is only useful for SLOG workloads. If you are not planning to use the disk for an slog, then this test is not going to give you results that are truly useful. It doesn't tell you how good or bad a disk is that is a spindle disk, nor how good of an l2arc it would be... only for SLOG.

The command to test:

# diskinfo -wS /dev/XXX

Here's a test on my system with one of the old (10+ year old) Sun Microsystem ZFS PCIe accelerator cards:
Code:
root@freenas:~ # diskinfo -wS /dev/da24
/dev/da24
		512			 # sectorsize
		24575868928	 # mediasize in bytes (23G)
		47999744		# mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		2987			# Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA MARVELL SD88SA02	# Disk descr.
		0946M01NAP			  # Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:   9990.9 usec/IO =	  0.0 Mbytes/s
		   1 kbytes:   9977.0 usec/IO =	  0.1 Mbytes/s
		   2 kbytes:   9986.2 usec/IO =	  0.2 Mbytes/s
		   4 kbytes:   9977.1 usec/IO =	  0.4 Mbytes/s
		   8 kbytes:   9991.1 usec/IO =	  0.8 Mbytes/s
		  16 kbytes:  10002.9 usec/IO =	  1.6 Mbytes/s
		  32 kbytes:   9854.5 usec/IO =	  3.2 Mbytes/s
		  64 kbytes:   9982.3 usec/IO =	  6.3 Mbytes/s
		 128 kbytes:   7868.0 usec/IO =	 15.9 Mbytes/s
		 256 kbytes:   9977.6 usec/IO =	 25.1 Mbytes/s
		 512 kbytes:   9982.8 usec/IO =	 50.1 Mbytes/s
		1024 kbytes:  10029.8 usec/IO =	 99.7 Mbytes/s
		2048 kbytes:  19998.0 usec/IO =	100.0 Mbytes/s
		4096 kbytes:  39990.5 usec/IO =	100.0 Mbytes/s
		8192 kbytes:  70000.2 usec/IO =	114.3 Mbytes/s


The important thing for sync writes is the microseconds of latency, lower is better. (Hint: Those numbers are relatively terrible!) The throughput does, however, matter as that is going to potentially be your bottleneck if you start getting large amounts of sync writes. dd is really just not a suitable alternative as the disk has to be given the flush commands and the time for the flush command to execute is important.

I'd like to compile a list of devices out there and the outputs. If people can test their devices along with providing the model number, disk size, and firmware version, that would be beneficial for people shopping for a disk to see what works and can look at current prices. That info can be obtained by looking at the output of smartctl -a /dev/XXX.

If we get enough data points, I'll try to put a table or something on here for easy access.

Thanks and happy NASing!
 
Last edited by a moderator:

sfcredfox

FreeNAS Experienced
Joined
Aug 26, 2014
Messages
316
Thanks
43
#2
Intel S3500
Code:
=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BB080G4
Serial Number:	PHWL446001B8080KGN
LU WWN Device Id: 5 5cd2e4 04c43e21d
Firmware Version: D2010370
User Capacity:	80,026,361,856 bytes [80.0 GB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat May 19 16:38:25 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:
root@Carmel-SANG2:~ # diskinfo -wS /dev/da48
/dev/da48
		512			 # sectorsize
		80026361856	 # mediasize in bytes (75G)
		156301488	   # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		9729			# Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA INTEL SSDSC2BB08	# Disk descr.
		PHWL446001B8080KGN	  # Disk ident.
		id1,enc@n500304800139d9fd/type@0/slot@3/elmdesc@Slot_03 # Physical path
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	243.1 usec/IO =	  2.0 Mbytes/s
		   1 kbytes:	226.8 usec/IO =	  4.3 Mbytes/s
		   2 kbytes:	201.3 usec/IO =	  9.7 Mbytes/s
		   4 kbytes:	124.6 usec/IO =	 31.3 Mbytes/s
		   8 kbytes:	147.5 usec/IO =	 53.0 Mbytes/s
		  16 kbytes:	164.4 usec/IO =	 95.1 Mbytes/s
		  32 kbytes:	289.9 usec/IO =	107.8 Mbytes/s
		  64 kbytes:	580.4 usec/IO =	107.7 Mbytes/s
		 128 kbytes:   1158.9 usec/IO =	107.9 Mbytes/s
		 256 kbytes:   2319.5 usec/IO =	107.8 Mbytes/s
		 512 kbytes:   4656.7 usec/IO =	107.4 Mbytes/s
		1024 kbytes:   9318.4 usec/IO =	107.3 Mbytes/s
		2048 kbytes:  18682.6 usec/IO =	107.1 Mbytes/s
		4096 kbytes:  37367.6 usec/IO =	107.0 Mbytes/s
		8192 kbytes:  74640.2 usec/IO =	107.2 Mbytes/s



Intel S3700 100GB
Code:
=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BA100G3T
Serial Number:	BTTV3452001M100FGN
LU WWN Device Id: 5 5cd2e4 04b52116d
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL05
User Capacity:	100,030,242,816 bytes [100 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:	Sat May 19 11:10:13 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:
root@Carmel-SANG2:~ # diskinfo -wS /dev/da7
/dev/da7
		512			 # sectorsize
		100030242816	# mediasize in bytes (93G)
		195371568	   # mediasize in sectors
		0			   # stripesize
		0			   # stripeoffset
		12161		   # Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA INTEL SSDSC2BA10	# Disk descr.
		BTTV3452001M100FGN	  # Disk ident.
		id1,enc@n500304800139d9fd/type@0/slot@1/elmdesc@Slot_01 # Physical path
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	301.8 usec/IO =	  1.6 Mbytes/s
		   1 kbytes:	294.7 usec/IO =	  3.3 Mbytes/s
		   2 kbytes:	265.3 usec/IO =	  7.4 Mbytes/s
		   4 kbytes:	176.9 usec/IO =	 22.1 Mbytes/s
		   8 kbytes:	191.4 usec/IO =	 40.8 Mbytes/s
		  16 kbytes:	235.7 usec/IO =	 66.3 Mbytes/s
		  32 kbytes:	300.8 usec/IO =	103.9 Mbytes/s
		  64 kbytes:	433.2 usec/IO =	144.3 Mbytes/s
		 128 kbytes:	686.3 usec/IO =	182.1 Mbytes/s
		 256 kbytes:   1265.0 usec/IO =	197.6 Mbytes/s
		 512 kbytes:   2489.4 usec/IO =	200.8 Mbytes/s
		1024 kbytes:   4998.6 usec/IO =	200.1 Mbytes/s
		2048 kbytes:  10029.3 usec/IO =	199.4 Mbytes/s
		4096 kbytes:  20055.8 usec/IO =	199.4 Mbytes/s
		8192 kbytes:  40179.3 usec/IO =	199.1 Mbytes/s



Mushkin 90GB **NOT enterprise class, no power loss protection, only comparison**
Code:
=== START OF INFORMATION SECTION ===
Model Family:	 SandForce Driven SSDs
Device Model:	 MKNSSDCR90GB
Serial Number:	MKN1251A0000147089
LU WWN Device Id: 0 000120 000000000
Firmware Version: 504ABBF0
User Capacity:	90,028,302,336 bytes [90.0 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS, ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Mon May 21 09:39:21 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:
root@Carmel-SANG2:~ # diskinfo -wS /dev/da36
/dev/da36
		512			 # sectorsize
		90028302336	 # mediasize in bytes (84G)
		175836528	   # mediasize in sectors
		0			   # stripesize
		0			   # stripeoffset
		10945		   # Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA MKNSSDCR90GB		# Disk descr.
		MKN1251A0000147089	  # Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	566.2 usec/IO =	  0.9 Mbytes/s
		   1 kbytes:	586.2 usec/IO =	  1.7 Mbytes/s
		   2 kbytes:	596.7 usec/IO =	  3.3 Mbytes/s
		   4 kbytes:	435.2 usec/IO =	  9.0 Mbytes/s
		   8 kbytes:	442.6 usec/IO =	 17.7 Mbytes/s
		  16 kbytes:	437.3 usec/IO =	 35.7 Mbytes/s
		  32 kbytes:	493.2 usec/IO =	 63.4 Mbytes/s
		  64 kbytes:	546.2 usec/IO =	114.4 Mbytes/s
		 128 kbytes:	704.3 usec/IO =	177.5 Mbytes/s
		 256 kbytes:   1009.6 usec/IO =	247.6 Mbytes/s
		 512 kbytes:   1627.8 usec/IO =	307.2 Mbytes/s
		1024 kbytes:   2838.0 usec/IO =	352.4 Mbytes/s
		2048 kbytes:   5247.6 usec/IO =	381.1 Mbytes/s
		4096 kbytes:  10337.1 usec/IO =	387.0 Mbytes/s
		8192 kbytes:  20317.7 usec/IO =	393.7 Mbytes/s
 
Last edited:

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,139
Thanks
3,867
#3
Place holder, so I can find this later.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
It's in the Resources section, under advanced. You don't have to search far. ;)
 

Stux

FreeNAS Wizard
Joined
Jun 2, 2016
Messages
4,180
Thanks
1,608
#4
Guess I can provide results for P3700

Does it matter if a machine is ESXi with pass through?
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,139
Thanks
3,867
#5
Since it's so high-performance, it's best to add a note to that effect.
 

js_level2

Super Moderator
Moderator
iXsystems
Joined
May 4, 2018
Messages
4
Thanks
17
#6
Don't forget to include the exact model and firmware. The best way is to use smartctl -a.

In the past, firmware versions have made a really big difference. ;)
 
Last edited by a moderator:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
1,943
Thanks
814
#7
Results for Intel S3500 (80G, 120G, 240G) and S3700 (100G over-provisioned to 8GB) confirm what I expected to find:
  • In general the S3700 is a better choice than the S3500 for a SLOG device
  • The speed of these SSDs increases in direct relationship to their capacity
You can see this by comparing just the 8192 kbyte results:
  • Intel S3500 (80G): 101.7 Mbytes/s
  • Intel S3500 (120G): 137.6 Mbytes/s
  • Intel S3500 (240G): 274.7 Mbytes/s
  • Intel S3700 (100/8G): 204.9 Mbytes/s
I suspect that an S3700 will always outperform an S3500 SSD of equivalent capacity.

Intel S3500 (80G):
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BB080G4
Serial Number:	...
LU WWN Device Id: 5 5cd2e4 04b9ea294
Firmware Version: D2012370
User Capacity:	80,026,361,856 bytes [80.0 GB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun May  6 07:04:48 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da16
	512			# sectorsize
	80026361856	 # mediasize in bytes (75G)
	156301488	  # mediasize in sectors
	4096			# stripesize
	0			  # stripeoffset
	9729			# Cylinders according to firmware.
	255			# Heads according to firmware.
	63			  # Sectors according to firmware.
	ATA INTEL SSDSC2BB08	# Disk descr.
	..................	  # Disk ident.
	Not_Zoned	  # Zone Mode

Synchronous random writes:
	 0.5 kbytes:	216.1 usec/IO =	  2.3 Mbytes/s
	   1 kbytes:	222.0 usec/IO =	  4.4 Mbytes/s
	   2 kbytes:	229.7 usec/IO =	  8.5 Mbytes/s
	   4 kbytes:	157.7 usec/IO =	 24.8 Mbytes/s
	   8 kbytes:	162.5 usec/IO =	 48.1 Mbytes/s
	  16 kbytes:	186.4 usec/IO =	 83.8 Mbytes/s
	  32 kbytes:	306.4 usec/IO =	102.0 Mbytes/s
	  64 kbytes:	613.7 usec/IO =	101.8 Mbytes/s
	 128 kbytes:   1225.5 usec/IO =	102.0 Mbytes/s
	 256 kbytes:   2447.5 usec/IO =	102.1 Mbytes/s
	 512 kbytes:   4891.0 usec/IO =	102.2 Mbytes/s
	1024 kbytes:   9839.9 usec/IO =	101.6 Mbytes/s
	2048 kbytes:  19644.1 usec/IO =	101.8 Mbytes/s
	4096 kbytes:  39283.1 usec/IO =	101.8 Mbytes/s
	8192 kbytes:  78637.4 usec/IO =	101.7 Mbytes/s

Intel S3500 (120G):
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BB120G4
Serial Number:	...
LU WWN Device Id: 5 5cd2e4 04b50bb20
Firmware Version: D2010355
User Capacity:	120,034,123,776 bytes [120 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun May  6 07:13:38 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da16
	512			# sectorsize
	120034123776	# mediasize in bytes (112G)
	234441648	  # mediasize in sectors
	0			  # stripesize
	0			  # stripeoffset
	14593		  # Cylinders according to firmware.
	255			# Heads according to firmware.
	63			  # Sectors according to firmware.
	ATA INTEL SSDSC2BB12	# Disk descr.
	..................	  # Disk ident.
	Not_Zoned	  # Zone Mode

Synchronous random writes:
	 0.5 kbytes:	228.2 usec/IO =	  2.1 Mbytes/s
	   1 kbytes:	211.1 usec/IO =	  4.6 Mbytes/s
	   2 kbytes:	197.1 usec/IO =	  9.9 Mbytes/s
	   4 kbytes:	183.5 usec/IO =	 21.3 Mbytes/s
	   8 kbytes:	200.7 usec/IO =	 38.9 Mbytes/s
	  16 kbytes:	217.8 usec/IO =	 71.8 Mbytes/s
	  32 kbytes:	258.1 usec/IO =	121.1 Mbytes/s
	  64 kbytes:	454.5 usec/IO =	137.5 Mbytes/s
	 128 kbytes:	897.4 usec/IO =	139.3 Mbytes/s
	 256 kbytes:   1803.6 usec/IO =	138.6 Mbytes/s
	 512 kbytes:   3625.5 usec/IO =	137.9 Mbytes/s
	1024 kbytes:   7205.6 usec/IO =	138.8 Mbytes/s
	2048 kbytes:  14477.0 usec/IO =	138.2 Mbytes/s
	4096 kbytes:  29208.6 usec/IO =	136.9 Mbytes/s
	8192 kbytes:  58143.3 usec/IO =	137.6 Mbytes/s

Intel S3500 (240G):
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BB240G4
Serial Number:	...
LU WWN Device Id: 5 5cd2e4 04b410446
Firmware Version: D2010370
User Capacity:	240,057,409,536 bytes [240 GB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun May  6 07:04:57 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da17
	512			# sectorsize
	240057409536	# mediasize in bytes (224G)
	468862128	  # mediasize in sectors
	4096			# stripesize
	0			  # stripeoffset
	29185		  # Cylinders according to firmware.
	255			# Heads according to firmware.
	63			  # Sectors according to firmware.
	ATA INTEL SSDSC2BB24	# Disk descr.
	..................	  # Disk ident.
	Not_Zoned	  # Zone Mode

Synchronous random writes:
	 0.5 kbytes:	141.1 usec/IO =	  3.5 Mbytes/s
	   1 kbytes:	138.6 usec/IO =	  7.0 Mbytes/s
	   2 kbytes:	150.5 usec/IO =	 13.0 Mbytes/s
	   4 kbytes:	164.5 usec/IO =	 23.8 Mbytes/s
	   8 kbytes:	143.2 usec/IO =	 54.5 Mbytes/s
	  16 kbytes:	235.9 usec/IO =	 66.2 Mbytes/s
	  32 kbytes:	238.8 usec/IO =	130.8 Mbytes/s
	  64 kbytes:	349.1 usec/IO =	179.0 Mbytes/s
	 128 kbytes:	504.8 usec/IO =	247.6 Mbytes/s
	 256 kbytes:	916.5 usec/IO =	272.8 Mbytes/s
	 512 kbytes:   1827.9 usec/IO =	273.5 Mbytes/s
	1024 kbytes:   3653.6 usec/IO =	273.7 Mbytes/s
	2048 kbytes:   7279.8 usec/IO =	274.7 Mbytes/s
	4096 kbytes:  14575.2 usec/IO =	274.4 Mbytes/s
	8192 kbytes:  29119.6 usec/IO =	274.7 Mbytes/s

Intel S3700 (100G ==> 8G):
Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:	 INTEL SSDSC2BA100G3L
Serial Number:	...
LU WWN Device Id: 5 5cd2e4 04b5acb83
Firmware Version: 5DV10265
User Capacity:	8,589,934,592 bytes [8.58 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun May  6 07:27:57 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

/dev/da6
	512			# sectorsize
	8589934592	  # mediasize in bytes (8.0G)
	16777216		# mediasize in sectors
	0			  # stripesize
	0			  # stripeoffset
	1044			# Cylinders according to firmware.
	255			# Heads according to firmware.
	63			  # Sectors according to firmware.
	ATA INTEL SSDSC2BA10	# Disk descr.
	..................	  # Disk ident.
	Not_Zoned	  # Zone Mode

Synchronous random writes:
	 0.5 kbytes:	377.4 usec/IO =	  1.3 Mbytes/s
	   1 kbytes:	361.3 usec/IO =	  2.7 Mbytes/s
	   2 kbytes:	301.2 usec/IO =	  6.5 Mbytes/s
	   4 kbytes:	212.6 usec/IO =	 18.4 Mbytes/s
	   8 kbytes:	223.7 usec/IO =	 34.9 Mbytes/s
	  16 kbytes:	236.5 usec/IO =	 66.1 Mbytes/s
	  32 kbytes:	298.8 usec/IO =	104.6 Mbytes/s
	  64 kbytes:	373.7 usec/IO =	167.3 Mbytes/s
	 128 kbytes:	610.6 usec/IO =	204.7 Mbytes/s
	 256 kbytes:   1213.1 usec/IO =	206.1 Mbytes/s
	 512 kbytes:   2438.8 usec/IO =	205.0 Mbytes/s
	1024 kbytes:   4868.4 usec/IO =	205.4 Mbytes/s
	2048 kbytes:   9756.7 usec/IO =	205.0 Mbytes/s
	4096 kbytes:  20534.1 usec/IO =	194.8 Mbytes/s
	8192 kbytes:  39052.6 usec/IO =	204.9 Mbytes/s
 

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,350
Thanks
2,984
#8
I also happen to have access to a 400GB, Intel SSD DC P3700, that has been formatted with 4k sectors and over provisioned to 144GB.
As I understand it, this is a fairly high performance drive.
I couldn't easily get information about this drive from smartctl -a, so I used the Intel SSD Data Center Tool 3.0.11 for Windows.
Here is the output of that:
Code:
C:\isdct>isdct show -intelssd

- Intel SSD DC P3700 Series CVFT7383003H400BGN -

Bootloader : 8B1B0133
DevicePath : \\\\.\\PHYSICALDRIVE1
DeviceStatus : Healthy
Firmware : 8DV101H0
FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.
Index : 0
ModelNumber : INTEL SSDPEDMD400G4
ProductFamily : Intel SSD DC P3700 Series
SerialNumber : CVFT7383003H400BGN
I only wish it were mine.

Here are the numbers from the diskinfo -wS test:
Code:
root@Emily-NAS:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
		4096			# sectorsize
		144000000000	# mediasize in bytes (134G)
		35156250		# mediasize in sectors
		131072		  # stripesize
		0			   # stripeoffset
		INTEL SSDPEDMD400G4	 # Disk descr.
		CVFT7383003H400BGN	  # Disk ident.

Synchronous random writes:
		   4 kbytes:	 15.0 usec/IO =	260.2 Mbytes/s
		   8 kbytes:	 17.8 usec/IO =	437.8 Mbytes/s
		  16 kbytes:	 26.0 usec/IO =	601.2 Mbytes/s
		  32 kbytes:	 34.9 usec/IO =	896.2 Mbytes/s
		  64 kbytes:	 63.3 usec/IO =	988.0 Mbytes/s
		 128 kbytes:	134.6 usec/IO =	928.3 Mbytes/s
		 256 kbytes:	248.1 usec/IO =   1007.7 Mbytes/s
		 512 kbytes:	488.0 usec/IO =   1024.6 Mbytes/s
		1024 kbytes:	945.7 usec/IO =   1057.4 Mbytes/s
		2048 kbytes:   1879.6 usec/IO =   1064.1 Mbytes/s
		4096 kbytes:   3706.1 usec/IO =   1079.3 Mbytes/s
		8192 kbytes:   7465.5 usec/IO =   1071.6 Mbytes/s
root@Emily-NAS:~ #
I hope this is useful to someone.

PS. When I use the smartctl -a /dev/nvd0 to try and get SMART info, I get this instead:
Code:
root@Emily-NAS:~ # smartctl -a /dev/nvd0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/nvd0: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary
Is there some other command for NVMe drives?
 
Last edited:

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,350
Thanks
2,984
#9
Guess I can provide results for P3700

Does it matter if a machine is ESXi with pass through?
The one I did was bare metal, if you do a test with your drive it might show if there is a difference because of the pass through?
 

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,350
Thanks
2,984
#10
These Hitachi SAS SSDs give about half the performance of the Intel DC P3700 but when I bought mine, they were only $50 and that isn't bad.
Code:
root@Emily-NAS:~/scripts # smartctl -a /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   HITACHI
Product:			  HUSSL4010BSS600
Revision:			 A110
Compliance:		   SPC-4
User Capacity:		100,030,242,816 bytes [100 GB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		Solid State Device
Form Factor:		  2.5 inches
Logical Unit id:	  0x5000cca0132cdfa8
Serial number:		XSVTP1GA
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sat May  5 20:43:10 2018 CDT
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Do you think that, if you were to stripe a couple of these together, they might be as fast as the P3700, with a lot less cost?
Code:
root@Emily-NAS:~/scripts # diskinfo -wS /dev/da0
/dev/da0
		512			 # sectorsize
		100030242816	# mediasize in bytes (93G)
		195371568	   # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		12161		   # Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		HITACHI HUSSL4010BSS600 # Disk descr.
		XSVTP1GA		# Disk ident.
		id1,enc@n5001c4500095d8bd/type@0/slot@5/elmdesc@Slot_004		# Physical path
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	242.1 usec/IO =	  2.0 Mbytes/s
		   1 kbytes:	234.3 usec/IO =	  4.2 Mbytes/s
		   2 kbytes:	219.0 usec/IO =	  8.9 Mbytes/s
		   4 kbytes:	187.3 usec/IO =	 20.9 Mbytes/s
		   8 kbytes:	195.3 usec/IO =	 40.0 Mbytes/s
		  16 kbytes:	215.1 usec/IO =	 72.6 Mbytes/s
		  32 kbytes:	252.8 usec/IO =	123.6 Mbytes/s
		  64 kbytes:	307.7 usec/IO =	203.1 Mbytes/s
		 128 kbytes:	440.9 usec/IO =	283.5 Mbytes/s
		 256 kbytes:	673.6 usec/IO =	371.1 Mbytes/s
		 512 kbytes:   1158.3 usec/IO =	431.7 Mbytes/s
		1024 kbytes:   2156.5 usec/IO =	463.7 Mbytes/s
		2048 kbytes:   4067.4 usec/IO =	491.7 Mbytes/s
		4096 kbytes:   7891.8 usec/IO =	506.9 Mbytes/s
		8192 kbytes:  15591.2 usec/IO =	513.1 Mbytes/s
Something to consider.
 
Last edited:

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,139
Thanks
3,867
#11
PS. When I use the smartctl -a /dev/nvd0 to try and get SMART info, I get this instead:
Another victim of the nvme/nvd dichotomy.

nvd is the NVMe namespace device driver. An NVMe disk nvme0 can provide multiple namespaces nvd0, nvd1, ..., nvdn.

The namespaces are kinda like fancy partitions. In the same way you don't run SMART on partitions, you need to run SMART on the device itself: smartctl -x /dev/nvme0.

Now, obligatory question: How the hell do you figure out the mapping if you have more than one NVMe device, other than by brute force checking every namespace? I have no idea. I went through both the nvme and the nvd drivers and could not figure out if and how this mapping is tracked by them.

(sidenote: I didn't spend hours in the drivers, just looked around for about an hour since I don't actually have any NVMe devices)
 
Last edited:

sfcredfox

FreeNAS Experienced
Joined
Aug 26, 2014
Messages
316
Thanks
43
#12
So, can the results be summarized? Or is more data needed?

When looking at the standard SSDs S3500/S3700, it appears they have similar latency (although the S3500s are faster than the S3700s at 4K), and the throughput differs at the larger record size end?

When looking at the NVE series, the latency is extremely low compared to the SSDs, and the throughput is doubled on the larger record size end? (I think that's fairly well known and expected)

Can a 4K write workload 'feel' the difference between ~160u-180u of latency (S3500s) versus ~212u of latency (S3700)? Can your workload 'feel' the difference between the S3500/S3700?

I imagine some workloads could 'feel' the difference between the S and P series for SLOG though?

Also, what is the more common expected record size for the SLOG? Probably workload dependent? Windows is usually 4k? SQL 16K/64K? Can't remember if VMware changes the write size with VMFS (1M)?

Thoughts? (This post was a pretty cool idea, good quantified data is awesome)
 

sfcredfox

FreeNAS Experienced
Joined
Aug 26, 2014
Messages
316
Thanks
43
#13
Wouldn't it be cool if you could run 'diskinfo -wS /dev/XXX' against a striped SLOG?

I wonder if the latency between two devices is as efficient as half of one device. One would assume, but maybe not?
 

Stux

FreeNAS Wizard
Joined
Jun 2, 2016
Messages
4,180
Thanks
1,608
#14
Joined
Dec 29, 2014
Messages
681
Thanks
158
#15
@Elliot Dierksen Any chance of some 900p benchmarks?
Sure. I assume I would need to remove this as an SLOG from my pool prior to running tests? Are you looking for the diskinfo stuff?

nvme0: <Generic NVMe Device> mem 0xdf010000-0xdf013fff irq 40 at device 0.0 numa-domain 0 on pci5

nvd0: <INTEL SSDPED1D280GA> NVMe namespace
nvd0: 267090MB (547002288 512 byte sectors)

I assume the ncd0 is what I should test, right?
 
Joined
Dec 29, 2014
Messages
681
Thanks
158
#16
@Elliot Dierksen Any chance of some 900p benchmarks?
How about this?

Code:
root@freenas2:/nonexistent # smartctl -x /dev/nvme0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:					   INTEL SSDPED1D280GA
Serial Number:					  PHMB742401A6280CGN
Firmware Version:				   E2010325
PCI Vendor/Subsystem ID:			0x8086
IEEE OUI Identifier:				0x5cd2e4
Controller ID:					  0
Number of Namespaces:			   1
Namespace 1 Size/Capacity:		  280,065,171,456 [280 GB]
Namespace 1 Formatted LBA Size:	 512
Local Time is:					  Thu May 31 10:04:14 2018 EDT
Firmware Updates (0x02):			1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):	 Wr_Unc DS_Mngmt
Maximum Data Transfer Size:		 32 Pages

Supported Power States
St Op	 Max   Active	 Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +	18.00W	   -		-	0  0  0  0		0	   0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +	 512	   0		 2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:				   0x00
Temperature:						41 Celsius
Available Spare:					100%
Available Spare Threshold:		  0%
Percentage Used:					0%
Data Units Read:					61 [31.2 MB]
Data Units Written:				 6,944,078 [3.55 TB]
Host Read Commands:				 1,321
Host Write Commands:				49,753,025
Controller Busy Time:			   28
Power Cycles:					   8
Power On Hours:					 202
Unsafe Shutdowns:				   0
Media and Data Integrity Errors:	0
Error Information Log Entries:	  0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged


Code:
root@freenas2:/nonexistent # diskinfo -wS /dev/nvd0
/dev/nvd0
		512			 # sectorsize
		280065171456	# mediasize in bytes (261G)
		547002288	   # mediasize in sectors
		0			   # stripesize
		0			   # stripeoffset
		INTEL SSDPED1D280GA	 # Disk descr.
		PHMB742401A6280CGN	  # Disk ident.

Synchronous random writes:
		 0.5 kbytes:	 22.8 usec/IO =	 21.4 Mbytes/s
		   1 kbytes:	 20.6 usec/IO =	 47.3 Mbytes/s
		   2 kbytes:	 21.4 usec/IO =	 91.5 Mbytes/s
		   4 kbytes:	 15.8 usec/IO =	247.0 Mbytes/s
		   8 kbytes:	 19.6 usec/IO =	397.9 Mbytes/s
		  16 kbytes:	 22.8 usec/IO =	684.8 Mbytes/s
		  32 kbytes:	 31.8 usec/IO =	981.9 Mbytes/s
		  64 kbytes:	 51.2 usec/IO =   1221.4 Mbytes/s
		 128 kbytes:	102.8 usec/IO =   1216.5 Mbytes/s
		 256 kbytes:	166.0 usec/IO =   1506.2 Mbytes/s
		 512 kbytes:	291.7 usec/IO =   1714.0 Mbytes/s
		1024 kbytes:	555.4 usec/IO =   1800.6 Mbytes/s
		2048 kbytes:   1060.2 usec/IO =   1886.4 Mbytes/s
		4096 kbytes:   2099.1 usec/IO =   1905.5 Mbytes/s
		8192 kbytes:   4179.4 usec/IO =   1914.2 Mbytes/s
 
Last edited by a moderator:

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,350
Thanks
2,984
#17
Last edited:

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,139
Thanks
3,867
#18
Fixed the lack of CODE tags.
 
Joined
Dec 29, 2014
Messages
681
Thanks
158
#19
If you could put that output inside code blocks, it would be much easier to read.
Really hoping this isn't a stupid question, but how do you do that? Despite being a FreeNAS user for some time, I am just now getting a little more active in the forums. *** Update *** Duh. started a dummy reply so I can see how it is formatted in the post. I'll do that in the future, thanks.

With small data, it isn't any faster than the DC P3700, but when the data gets bigger, it is almost double the speed. Nice.
I have been very happy with it so far. I have some used S3500's showing up at some point, and I am going to give those a whirl. I also nabbed another 9207-8e which I will put in the primary system so I can see how much of a jolt the 900P gives to the older drives in the D2700 enclosure.
 
Joined
Dec 29, 2014
Messages
681
Thanks
158
#20
I have some used S3500's showing up at some point, and I am going to give those a whirl.
One of them arrived this morning. Here are the same tests from that.

Code:
root@freenas:/nonexistent # smartctl -a /dev/da53
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BB120G4
Serial Number:    PHWL505302DL120LGN
LU WWN Device Id: 5 5cd2e4 04c4d49a8
Firmware Version: D2010370
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jun  2 10:33:51 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (    0) seconds.
Offline data collection
capabilities:                    (0x79) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (   2) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       74
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       43
170 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
174 Unsafe_Shutdown_Count   0x0032   100   100   000    Old_age   Always       -       41
175 Power_Loss_Cap_Test     0x0033   100   100   010    Pre-fail  Always       -       660 (1 12)
183 SATA_Downshift_Count    0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Temperature_Case        0x0022   077   077   000    Old_age   Always       -       23 (Min/Max 21/24)
192 Unsafe_Shutdown_Count   0x0032   100   100   000    Old_age   Always       -       41
194 Temperature_Internal    0x0022   100   100   000    Old_age   Always       -       30
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
225 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       3617
226 Workld_Media_Wear_Indic 0x0032   100   100   000    Old_age   Always       -       0
227 Workld_Host_Reads_Perc  0x0032   100   100   000    Old_age   Always       -       1
228 Workload_Minutes        0x0032   100   100   000    Old_age   Always       -       4259
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       0
234 Thermal_Throttle        0x0032   100   100   000    Old_age   Always       -       0/0
241 Host_Writes_32MiB       0x0032   100   100   000    Old_age   Always       -       3617
242 Host_Reads_32MiB        0x0032   100   100   000    Old_age   Always       -       57

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Code:
root@freenas:/nonexistent # diskinfo -wS /dev/da53
/dev/da53
        512             # sectorsize
        120034123776    # mediasize in bytes (112G)
        234441648       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        14593           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA INTEL SSDSC2BB12    # Disk descr.
        PHWL505302DL120LGN      # Disk ident.
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     82.0 usec/IO =      6.0 Mbytes/s
           1 kbytes:     78.8 usec/IO =     12.4 Mbytes/s
           2 kbytes:     82.3 usec/IO =     23.7 Mbytes/s
           4 kbytes:     83.4 usec/IO =     46.8 Mbytes/s
           8 kbytes:    105.0 usec/IO =     74.4 Mbytes/s
          16 kbytes:    139.6 usec/IO =    111.9 Mbytes/s
          32 kbytes:    220.3 usec/IO =    141.8 Mbytes/s
          64 kbytes:    441.7 usec/IO =    141.5 Mbytes/s
         128 kbytes:    881.8 usec/IO =    141.8 Mbytes/s
         256 kbytes:   1761.2 usec/IO =    141.9 Mbytes/s
         512 kbytes:   3535.8 usec/IO =    141.4 Mbytes/s
        1024 kbytes:   7306.8 usec/IO =    136.9 Mbytes/s
        2048 kbytes:  14152.1 usec/IO =    141.3 Mbytes/s
        4096 kbytes:  28319.4 usec/IO =    141.2 Mbytes/s
        8192 kbytes:  56847.8 usec/IO =    140.7 Mbytes/s


I have another one on the way, but I am not quite sure how I want to configure these. I could use one of them as a single JBOD drive for an SLOG. I could use the RAID controller to make a RAID1 mirror of the two of them, or I could use them as JBOD and let FreeNAS mirror them. Obviously the SLOG seems important, so I am leaning towards mirroring them. That said, I am not mirroring the new 900P because I only have one. Granted the 900P is brand new, so I would hope not to have issues with it. Any recommendations there?
 
Top