Resource icon

SLOG benchmarking and finding the best SLOG

Turgin

Dabbler
Joined
Feb 20, 2016
Messages
43
Intel DC S3500 300GB attached via X9SRL-F onboard SATA. ESXi 6.5 Passthrough

Code:
=== START OF INFORMATION SECTION ===
Model Family:	 Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:	 INTEL SSDSC2BB300G4T
Serial Number:	BTWL414500AG300PGN
LU WWN Device Id: 5 5cd2e4 04b611880
Add. Product Id:  DELL(tm)
Firmware Version: D201DL13
User Capacity:	300,069,052,416 bytes [300 GB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	Solid State Device
Form Factor:	  2.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Tue Jul  3 21:40:22 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


Code:
root@nas01:~ # diskinfo -wS /dev/ada0
/dev/ada0
		512			 # sectorsize
		300069052416	# mediasize in bytes (279G)
		586072368	   # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		581421		  # Cylinders according to firmware.
		16			  # Heads according to firmware.
		63			  # Sectors according to firmware.
		INTEL SSDSC2BB300G4T	# Disk descr.
		BTWL414500AG300PGN	  # Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	190.2 usec/IO =	  2.6 Mbytes/s
		   1 kbytes:	179.2 usec/IO =	  5.4 Mbytes/s
		   2 kbytes:	151.3 usec/IO =	 12.9 Mbytes/s
		   4 kbytes:	 92.0 usec/IO =	 42.5 Mbytes/s
		   8 kbytes:	106.9 usec/IO =	 73.1 Mbytes/s
		  16 kbytes:	133.6 usec/IO =	116.9 Mbytes/s
		  32 kbytes:	173.2 usec/IO =	180.4 Mbytes/s
		  64 kbytes:	303.3 usec/IO =	206.1 Mbytes/s
		 128 kbytes:	476.6 usec/IO =	262.3 Mbytes/s
		 256 kbytes:	809.6 usec/IO =	308.8 Mbytes/s
		 512 kbytes:   1530.9 usec/IO =	326.6 Mbytes/s
		1024 kbytes:   2982.0 usec/IO =	335.3 Mbytes/s
		2048 kbytes:   5911.5 usec/IO =	338.3 Mbytes/s
		4096 kbytes:  11753.5 usec/IO =	340.3 Mbytes/s
		8192 kbytes:  23607.6 usec/IO =	338.9 Mbytes/s
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
  • INTL SSD DC P3700 - 400GB [ESXi 6.5U2 | Passthrough to 11.1-U5]
  • INTL Optane 900p - 280GB [ESXi 6.5U2 | 20GB vDisk to 11.1-U5]
Notes:
  1. intel-nvme driver 1.2.1.15-1OEM in use for Optane 900p (local NVMe Disk in ESXi 6.5 U2)
  2. for your reference = Low I/O performance using intel-nvme drivers 1.3.2.8-1OEM and 1.3.2.4-1OEM with block sizes larger than 4K (55693)

SLOG_Graph.png

diskinfo -wS /dev/nvdX [above]
smartctl -a /dev/nvmeX [below]

INTL Optane 900p - 280GB [ESXi 6.5U2 | 20GB vDisk to 11.1-U5]

Code:
diskinfo -wS /dev/nvd2
/dev/nvd2
		  512			 # sectorsize
		  21474836480	 # mediasize in bytes (20G)
		  41943040		# mediasize in sectors
		  0			   # stripesize
		  0			   # stripeoffset
		  Virtual disk	# Disk descr.
		  VMWare NVME-0001		# Disk ident.

Synchronous random writes:
		   0.5 kbytes:	 45.2 usec/IO =	 10.8 Mbytes/s
			 1 kbytes:	 47.6 usec/IO =	 20.5 Mbytes/s
			 2 kbytes:	 46.5 usec/IO =	 42.0 Mbytes/s
			 4 kbytes:	 44.6 usec/IO =	 87.5 Mbytes/s
			 8 kbytes:	 47.0 usec/IO =	166.2 Mbytes/s
			16 kbytes:	 49.8 usec/IO =	314.0 Mbytes/s
			32 kbytes:	 58.6 usec/IO =	533.5 Mbytes/s
			64 kbytes:	 76.4 usec/IO =	818.3 Mbytes/s
		   128 kbytes:	114.6 usec/IO =   1091.1 Mbytes/s
		   256 kbytes:	178.9 usec/IO =   1397.5 Mbytes/s
		   512 kbytes:	294.3 usec/IO =   1699.2 Mbytes/s
		  1024 kbytes:	527.2 usec/IO =   1896.7 Mbytes/s
		  2048 kbytes:	981.7 usec/IO =   2037.3 Mbytes/s
		  4096 kbytes:   1896.4 usec/IO =   2109.2 Mbytes/s
		  8192 kbytes:   3739.1 usec/IO =   2139.6 Mbytes/s

INTL SSD DC P3700 - 400GB [ESXi 6.5U2 | Passthrough to 11.1-U5]
Code:
diskinfo -wS /dev/nvd1
/dev/nvd1
		  4096			# sectorsize
		  400088457216	# mediasize in bytes (373G)
		  97677846		# mediasize in sectors
		  131072		  # stripesize
		  0			   # stripeoffset
		  INTEL SSDPE2MD400G4	 # Disk descr.
		  000000000000000000 	 # Disk ident.

Synchronous random writes:
			 4 kbytes:	 29.4 usec/IO =	132.9 Mbytes/s
			 8 kbytes:	 36.2 usec/IO =	215.6 Mbytes/s
			16 kbytes:	 37.1 usec/IO =	421.1 Mbytes/s
			32 kbytes:	 43.3 usec/IO =	722.4 Mbytes/s
			64 kbytes:	 70.0 usec/IO =	893.4 Mbytes/s
		   128 kbytes:	134.0 usec/IO =	932.7 Mbytes/s
		   256 kbytes:	245.5 usec/IO =   1018.2 Mbytes/s
		   512 kbytes:	477.5 usec/IO =   1047.1 Mbytes/s
		  1024 kbytes:	974.8 usec/IO =   1025.8 Mbytes/s
		  2048 kbytes:   1936.2 usec/IO =   1032.9 Mbytes/s
		  4096 kbytes:   3872.2 usec/IO =   1033.0 Mbytes/s
		  8192 kbytes:   7646.6 usec/IO =   1046.2 Mbytes/s

INTL Optane 900p - 280GB [ESXi 6.5U2 | 20GB vDisk to 11.1-U5]
Code:
smartctl -a /dev/nvme2
=== START OF INFORMATION SECTION ===
Model Number:					   Virtual disk
Serial Number:					  VMWare NVME-0001
Firmware Version:				   1.0
PCI Vendor/Subsystem ID:			0x15ad
IEEE OUI Identifier:				0x565000
Controller ID:					  0
Number of Namespaces:			   1
Namespace 1 Size/Capacity:		  21,474,836,480 [21.4 GB]
Namespace 1 Formatted LBA Size:	 512
Local Time is:					  Wed Jul  4 14:50:11 2018 EDT
Firmware Updates (0x03):			1 Slot, Slot 1 R/O
Maximum Data Transfer Size:		 256 Pages

Supported Power States
St Op	 Max   Active	 Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +	 0.00W	   -		-	0  0  0  0		0	   0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +	 512	   0		 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:				   0x00
Temperature:						11759 Celsius
Available Spare:					1%
Available Spare Threshold:		  100%
Percentage Used:					0%
Data Units Read:					~110680464442257309696
Data Units Written:				 ~92233720368547758080
Host Read Commands:				 ~110680464442257309696000
Host Write Commands:				~92233720368547758080000
Controller Busy Time:			   ~92233720368547758080
Power Cycles:					   ~184467440737095516160
Power On Hours:					 ~1106804644422573096960
Unsafe Shutdowns:				   0
Media and Data Integrity Errors:	0
Error Information Log Entries:	  0

Error Information (NVMe Log 0x01, max 4 entries)
No Errors Logged

INTL SSD DC P3700 - 400GB [ESXi 6.5U2 | Passthrough to 11.1-U5]
Code:
smartctl -a /dev/nvme1

=== START OF INFORMATION SECTION ===
Model Number:					   INTEL SSDPE2MD400G4
Serial Number:					  000000000000000000
Firmware Version:				   8DV101H0
PCI Vendor/Subsystem ID:			0x8086
IEEE OUI Identifier:				0x5cd2e4
Controller ID:					  0
Number of Namespaces:			   1
Namespace 1 Size/Capacity:		  400,088,457,216 [400 GB]
Namespace 1 Formatted LBA Size:	 4096
Local Time is:					  Wed Jul  4 14:34:40 2018 EDT
Firmware Updates (0x02):			1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):	 Wr_Unc DS_Mngmt
Maximum Data Transfer Size:		 32 Pages

Supported Power States
St Op	 Max   Active	 Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +	25.00W	   -		-	0  0  0  0		0	   0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -	 512	   0		 2
 1 -	 512	   8		 2
 2 -	 512	  16		 2
 3 +	4096	   0		 0
 4 -	4096	   8		 0
 5 -	4096	  64		 0
 6 -	4096	 128		 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:				   0x00
Temperature:						37 Celsius
Available Spare:					100%
Available Spare Threshold:		  10%
Percentage Used:					0%
Data Units Read:					5,408,890 [2.76 TB]
Data Units Written:				 7,249,543 [3.71 TB]
Host Read Commands:				 44,863,224
Host Write Commands:				176,103,658
Controller Busy Time:			   2
Power Cycles:					   27
Power On Hours:					 2,206
Unsafe Shutdowns:				   0
Media and Data Integrity Errors:	0
Error Information Log Entries:	  0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
Wow.

No update on being able to pass through the 900p then?

The question is, does uses ESXi local disk mode for the P3700 affect its speed? Ie to help ensure that the faster 900p transfers are by a side-effect if not being able to pass through
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Pretty graph, hunh??? Oh wait, you must mean the Optane's speed. ;) [bad humour per usual]
No update on being able to pass through the 900p then?
I could not get pass through to work. AFAIK - it remains an open issue: https://redmine.ixsystems.com/issues/26508
The question is, does uses ESXi local disk mode for the P3700 affect its speed? Ie to help ensure that the faster 900p transfers are by a side-effect if not being able to pass through
  • Just to confirm, Optane = local datastore w/ vDisk for FreeNAS (not P3700, but I think you meant Optane). P3700 is pass through to FreeNAS.
  • If I follow correctly, you are asking if this affects speed? If so, I would imagine: Unequivocally, yes ...
    • I suppose the question becomes, how much? And does it have it to spare? No answer on former, "probably" on the later, but who likes to give up performance?
    • I have never been able to get close to bare metal performance out of an NVMe drive as a local datastore in ESXi (and don't think I'm alone there).
    • Pre-Optane, I was so dumbfounded by the lack of P3700 performance as an ESXi datastore, I pulled it to bench it in a workstation, to see if it was the driver, U2 to PCIe adapter, etc.
    • There, bare metal performance = INTL's specifications; however, that lack of performance in ESXi may have been because I was using one of the driver's addressed in VMware's kb (see link in earlier message).
    • I'd be happy to lend a hand however you like (benchmarking Optane's speed), just let me know.
  • Right or wrong, you can tell me (and please!), but this is what my configuration looks like at the moment and without being able to pass through, what made most sense to me (however, I don't proclaim to be an expert w/ ESXi):
    • I'm more than happy to play around with the config given guidance and revert feedback for the benefit of others.
    • Optane = datastore = reserved for FreeNAS
    • vDisks:
      • freebsd-boot = Boot
      • freebsd-log = SLOG
      • freebsd-cache = L2ARC (toying around with)
    • Each vdisk has its own NVMe Controller, so 3 total. Possible only 1 is needed, or an NVMe Controller is not needed at all?
On a separate note, I have to compliment you on your X10SDV-TLN4F [ESXi/FreeNAS AIO] build thread. Amazing stuff. I tried to use a vDisk for swap, but could not accomplish that. I'll reply to your thread on the topic, regarding that.
Code:
t10.NVMe____INTEL_SSDPED1D280GA_blahblah__00000001
   Device Display Name: Local NVMe Disk (t10.NVMe____INTEL_SSDPED1D280GA_blahblah__00000001)
   Storage Array Type: VMW_SATP_LOCAL
   Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
   Path Selection Policy: VMW_PSP_FIXED
   Path Selection Policy Device Config: {preferred=vmhba3:C0:T0:L0;current=vmhba3:C0:T0:L0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba3:C0:T0:L0
   Is USB: false
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
I suppose the question becomes, how much? And does it have it to spare? No answer on former, "probably" on the later, but who likes to give up performance?

Maybe I'm wrong ...

Look at the below results, where I've lined up @Elliot Dierksen 's results (900p earlier in thread) against my own. His are presumably from a bare metal or pass through use of the 900p (based upon this SMART data), whereas my results are using a vDisk.

So, my results should be worse as that disk is already in use and its local to ESXi, right?

However, my results appear slightly more performant, as @ 8192 kbytes my latency is 10.5% lower (3739.1 µs v. 4179.4 µs) and speed is 11.8% faster (2139.6 Mbytes/s v. 1914.2 Mbytes/s). I would expect precisely the opposite due to that disk already being in use (measured a 20GB bit) and overhead from ESXi. If you look closely, you can see he outperforms from 0.5 - 512 kbytes, but at 1024 kbytes and above my results appear better. Possibly due to VMFS6 1MB blocksize? You can rule out any difference of firmware driving that as firmware rev E2010325 hasn't been updated since the 900p shipped. In trying to rationalize this, I simply cannot as I can't think of any externalities that would have a favourable impact, i.e. no virtual flash present or even datastore cache is defined (not that that would have an impact).

Also, interestingly, my top figure exceeds ark.intel.com 's rated max performance of 2000 MB/s @ 107% whereas the P3700 only achieves 52% of the same rated speed. I understand this benchmark is for looking at latency, nonetheless, find that interesting.

Not sure what to make of that ... as no optimizations are applied, etc. Sully = scratches head.

Code:
[Elliot Dierksen Bench]											   [Sully Bench]
(presumably bare metal or pass through from SMART)					(vDisk)

		   0.5 kbytes:	 22.8 usec/IO =	 21.4 Mbytes/s		   0.5 kbytes:	 45.2 usec/IO =	 10.8 Mbytes/s
			 1 kbytes:	 20.6 usec/IO =	 47.3 Mbytes/s			 1 kbytes:	 47.6 usec/IO =	 20.5 Mbytes/s
			 2 kbytes:	 21.4 usec/IO =	 91.5 Mbytes/s			 2 kbytes:	 46.5 usec/IO =	 42.0 Mbytes/s
			 4 kbytes:	 15.8 usec/IO =	247.0 Mbytes/s			 4 kbytes:	 44.6 usec/IO =	 87.5 Mbytes/s
			 8 kbytes:	 19.6 usec/IO =	397.9 Mbytes/s			 8 kbytes:	 47.0 usec/IO =	166.2 Mbytes/s
			16 kbytes:	 22.8 usec/IO =	684.8 Mbytes/s			16 kbytes:	 49.8 usec/IO =	314.0 Mbytes/s
			32 kbytes:	 31.8 usec/IO =	981.9 Mbytes/s			32 kbytes:	 58.6 usec/IO =	533.5 Mbytes/s
			64 kbytes:	 51.2 usec/IO =   1221.4 Mbytes/s			64 kbytes:	 76.4 usec/IO =	818.3 Mbytes/s
		   128 kbytes:	102.8 usec/IO =   1216.5 Mbytes/s		   128 kbytes:	114.6 usec/IO =   1091.1 Mbytes/s
		   256 kbytes:	166.0 usec/IO =   1506.2 Mbytes/s		   256 kbytes:	178.9 usec/IO =   1397.5 Mbytes/s
		   512 kbytes:	291.7 usec/IO =   1714.0 Mbytes/s		   512 kbytes:	294.3 usec/IO =   1699.2 Mbytes/s
		  1024 kbytes:	555.4 usec/IO =   1800.6 Mbytes/s		  1024 kbytes:	527.2 usec/IO =   1896.7 Mbytes/s
		  2048 kbytes:   1060.2 usec/IO =   1886.4 Mbytes/s		  2048 kbytes:	981.7 usec/IO =   2037.3 Mbytes/s
		  4096 kbytes:   2099.1 usec/IO =   1905.5 Mbytes/s		  4096 kbytes:   1896.4 usec/IO =   2109.2 Mbytes/s
		  8192 kbytes:   4179.4 usec/IO =   1914.2 Mbytes/s		  8192 kbytes:   3739.1 usec/IO =   2139.6 Mbytes/s
 
Joined
Dec 29, 2014
Messages
1,135
Look at the below results, where I've lined up @Elliot Dierksen 's results (900p earlier in thread) against my own. His are presumably from a bare metal or pass through use of the 900p (based upon this SMART data), whereas my results are using a vDisk.

Yes, my FreeNAS is bare metal. That is kind of surprising, but perhaps there is something else in the hardware of the server. My server is not super new, but not super old either. Dual E5-2660 v2 @ 2.20GHz with 10600R DRAM.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Yes, my FreeNAS is bare metal. That is kind of surprising, but perhaps there is something else in the hardware of the server. My server is not super new, but not super old either. Dual E5-2660 v2 @ 2.20GHz with 10600R DRAM.
  • Thanks for confirming. Similar specs here ... 2x E5-2680V2 + 192GB (128GB to FN) 10600 (1333 MHz) RDIMMs..
  • It is really surprising ... I'm not sure what to make of it and frankly I'm not knowledgeable enough with either ESXi or FreeNAS to offer further insight (but I'm happy to run additional benchmarks, etc.).
  • I'm not sure if a single Optane is enough to create a bottleneck at the CPU with 20 cores / 40 threads? And if so, what is more important, more cores or higher clock (with SMB it would be the latter). Note I have less than 20% of cores allocated to FN.
  • But I will say that when I tossed an 800GB P3700 in a workstation and ran it through IOmeter hell (queue depth/workers), my OCed i7-4770k (4c/8t) @ 4.5 GHz continuously produced BSODs so I had to back off the overclock. Possible that may be it here (600 MHz delta @ base clock and turbo), but I think it is probably something else due to that flip at 1024 kbytes. I doubt it, but at those latencies / speeds, it could be that the trace between CPU and PCIe slot is longer in your case (a stretch I'm sure).
  • I think it has to do with ESXi defaulting to 1MB block size in VMFS6. I wonder if that could be normalized for on your end by way of changing the zfs recordsize/volblocksize (don't know enough to answer that though).
  • You may have noted earlier in the thread, and I don't have the time to review at the moment, but how did you end up deploying and do you have any real-world benchmarks? I think that test is great at examining theoretical maximum and further, better at comparing potential slog devices, but the real-world is probably more important.
  • Also, did you change any zfs attributes or add any tunables (I'm not that far along yet)?
Enjoy the balance of your day.
 
Joined
Dec 29, 2014
Messages
1,135
Thanks for confirming. Similar specs here ... 2x E5-2680V2 + 192GB (128GB to FN) 10600 (1333 MHz) RDIMMs..

128GB here, so no big difference.

I'm not sure if a single Optane is enough to create a bottleneck at the CPU with 20 cores / 40 threads? And if so, what is more important, more cores or higher clock (with SMB it would be the latter).

I wouldn't think so. I don't if this is still true, but I know Samba/SMB was single threaded for some time. I still operate on that assumption, but my high volume traffic is NFS.

I think it has to do with ESXi defaulting to 1MB block size in VMFS6.

I haven't done any serious manual tuning on these kind of parameters in either ESXi or FreeNAS. I did a bunch hardware thrashing prior to getting the SLOG memo, so I reset everything (disable autotune and deleted all tunables, reboot, enable autotune, reboot). The only tuning I have done on the ESXi side has to do with disabling LRO because it messed with of the Cisco virtual machine I run in my lab. I changed some of the Vcenter parameters to allow storage Vmotion to use more resources, but that is about it.

how did you end up deploying and do you have any real-world benchmarks?

I mainly used storage Vmotion to measure the performance. An inexact tool to be sure, but it was pretty easy. Subsequent migrations to FreeNAS always went faster if I did them before FreeNAS had released all the RAM it had for the ARC.

My setup is mostly a lab. Truth be told, my e-mail server is really the only production server I have and it is a small FreeBSD VM. The other two machines normally running are Vcenter and PC Network Shutdown (APC UPS monitoring). I Vmotion things around when I do updates because I can :). I could easily shut stuff down for that, but it fun to not have to do so. I have 2 6.5 ESXi hosts, 1 6.0 ESXi, and 1 5.5 ESXi, and they all mount FreeNAS datastore via NFS. I do some light file server stuff/backups via CIFS/SMB, but nothing of any great volume.
 
Joined
Dec 29, 2014
Messages
1,135
I forgot to mention that I did increase the number NFS server processes to 20.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
For giggles I ran the benchmark against one of my old Intel 320 80GB drives, as they were popular "budget SLOG devices" back in the day.

Code:
=== START OF INFORMATION SECTION ===
Model Family:	 Intel 320 Series SSDs
Device Model:	 INTEL SSDSA2CW080G3
Serial Number:	CVPR136200EX080BGN
LU WWN Device Id: 5 001517 95968e06c
Firmware Version: 4PC10362
User Capacity:	8,589,934,592 bytes [8.58 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:	Thu Jul  5 16:45:57 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


Code:
/dev/ada1
		512			 # sectorsize
		8589934592	  # mediasize in bytes (8.0G)
		16777216		# mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		16644		   # Cylinders according to firmware.
		16			  # Heads according to firmware.
		63			  # Sectors according to firmware.
		INTEL SSDSA2CW080G3	 # Disk descr.
		CVPR136200EX080BGN	  # Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	115.2 usec/IO =	  4.2 Mbytes/s
		   1 kbytes:	133.0 usec/IO =	  7.3 Mbytes/s
		   2 kbytes:	138.6 usec/IO =	 14.1 Mbytes/s
		   4 kbytes:	147.6 usec/IO =	 26.5 Mbytes/s
		   8 kbytes:	161.0 usec/IO =	 48.5 Mbytes/s
		  16 kbytes:	201.4 usec/IO =	 77.6 Mbytes/s
		  32 kbytes:	364.7 usec/IO =	 85.7 Mbytes/s
		  64 kbytes:	694.6 usec/IO =	 90.0 Mbytes/s
		 128 kbytes:   1354.2 usec/IO =	 92.3 Mbytes/s
		 256 kbytes:   2685.9 usec/IO =	 93.1 Mbytes/s
		 512 kbytes:   5385.6 usec/IO =	 92.8 Mbytes/s
		1024 kbytes:  10893.6 usec/IO =	 91.8 Mbytes/s
		2048 kbytes:  21654.7 usec/IO =	 92.4 Mbytes/s
		4096 kbytes:  43468.9 usec/IO =	 92.0 Mbytes/s
		8192 kbytes:  86637.2 usec/IO =	 92.3 Mbytes/s


Impressive enough results from an old, small drive.
 
Last edited:

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Impressive enough results from an old, small drive.
Honey badger don't care how small his drive is (had to watch the video, I ROFL laughing, it had been a while). Here is your drive's pint size brother (Intel 320 Series - 40GB): Half as large, twice as much latency, and half as fast. He was one of two faithful boot drives until i went to satadom.

A 2 x 1 mirror manages an astounding 66 MB/s with a SLOG, sync=always, no compression vs. 56 MB/s without a SLOG, sync=disabled, no compression with
Code:
dd if=/dev/zero of=/mnt/Tank0/testfile1 bs=4M count=1000

Not a match made in heaven for a good SLOG, but agreed a good drive for its era.
Code:
smartctl -a /dev/da16

=== START OF INFORMATION SECTION ===
Model Family:	 Intel 320 Series SSDs
Device Model:	 INTEL SSDSA2CT040G3
Serial Number:	BTPR224202J6040AGN
LU WWN Device Id: 5 001517 972e74553
Firmware Version: 4PC10362
User Capacity:	40,020,664,320 bytes [40.0 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:	Thu Jul  5 17:08:32 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:
diskinfo -wS /dev/da16
/dev/da16
		512			 # sectorsize
		40020664320	 # mediasize in bytes (37G)
		78165360		# mediasize in sectors
		0			   # stripesize
		0			   # stripeoffset
		4865			# Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA INTEL SSDSA2CT04	# Disk descr.
		BTPR224202J6040AGN	  # Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	163.9 usec/IO =	  3.0 Mbytes/s
		   1 kbytes:	167.8 usec/IO =	  5.8 Mbytes/s
		   2 kbytes:	176.3 usec/IO =	 11.1 Mbytes/s
		   4 kbytes:	177.2 usec/IO =	 22.0 Mbytes/s
		   8 kbytes:	189.2 usec/IO =	 41.3 Mbytes/s
		  16 kbytes:	338.5 usec/IO =	 46.2 Mbytes/s
		  32 kbytes:	649.5 usec/IO =	 48.1 Mbytes/s
		  64 kbytes:   1290.6 usec/IO =	 48.4 Mbytes/s
		 128 kbytes:   2578.7 usec/IO =	 48.5 Mbytes/s
		 256 kbytes:   5147.4 usec/IO =	 48.6 Mbytes/s
		 512 kbytes:  10328.4 usec/IO =	 48.4 Mbytes/s
		1024 kbytes:  20705.3 usec/IO =	 48.3 Mbytes/s
		2048 kbytes:  41406.9 usec/IO =	 48.3 Mbytes/s
		4096 kbytes:  82803.0 usec/IO =	 48.3 Mbytes/s
		8192 kbytes: 165685.9 usec/IO =	 48.3 Mbytes/s
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
The 160GB Intel 320s are more performant, but they're still very old tech. If you can get them very inexpensively, they'll do in a pinch - but they're better as boot drives.

In the meantime, I benchmarked a couple SSDs. First up is an OCZ Vertex 2 - this one is notable because the performance jumps up at 8KB - I believe this matches the NAND page size. This isn't a viable SLOG SSD due to lack of power-loss prevention, but it's worth noting that this drive would perform much better under ashift=13 than ashift=12.

OCZ Vertex 2 - 60GB
Code:
=== START OF INFORMATION SECTION ===
Model Family:	 SandForce Driven SSDs
Device Model:	 OCZ-VERTEX2
Serial Number:	OCZ-1235L9GXLSEHI3CY
LU WWN Device Id: 5 e83a97 f359b3db8
Firmware Version: 1.37
User Capacity:	60,022,480,896 bytes [60.0 GB]
Sector Size:	  512 bytes logical/physical
Rotation Rate:	Solid State Device
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:	Fri Jul  6 00:55:01 2018 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


Code:
/dev/ada5
		512			 # sectorsize
		60022480896	 # mediasize in bytes (56G)
		117231408	   # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		116301		  # Cylinders according to firmware.
		16			  # Heads according to firmware.
		63			  # Sectors according to firmware.
		OCZ-VERTEX2	 # Disk descr.
		OCZ-1235L9GXLSEHI3CY	# Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	235.7 usec/IO =	  2.1 Mbytes/s
		   1 kbytes:	234.5 usec/IO =	  4.2 Mbytes/s
		   2 kbytes:	242.8 usec/IO =	  8.0 Mbytes/s
		   4 kbytes:	 58.0 usec/IO =	 67.3 Mbytes/s
		   8 kbytes:	 73.9 usec/IO =	105.7 Mbytes/s
		  16 kbytes:	183.7 usec/IO =	 85.1 Mbytes/s
		  32 kbytes:	324.9 usec/IO =	 96.2 Mbytes/s
		  64 kbytes:	543.1 usec/IO =	115.1 Mbytes/s
		 128 kbytes:	872.5 usec/IO =	143.3 Mbytes/s
		 256 kbytes:   1483.0 usec/IO =	168.6 Mbytes/s
		 512 kbytes:   2761.3 usec/IO =	181.1 Mbytes/s
		1024 kbytes:   4914.1 usec/IO =	203.5 Mbytes/s
		2048 kbytes:   9094.3 usec/IO =	219.9 Mbytes/s
		4096 kbytes:  17274.7 usec/IO =	231.6 Mbytes/s
		8192 kbytes:  33385.6 usec/IO =	239.6 Mbytes/s
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Now let's move on to a more interesting device. 4GB of DDR memory, backed by a rechargeable battery, and presented via SATA1, I give you:

Gigabyte i-RAM 4GB
Code:
=== START OF INFORMATION SECTION ===
Device Model:	 GIGABYTE i-RAM
Serial Number:	4B56363A04B4B3AA93D5
Firmware Version: v0.98
User Capacity:	4,293,918,208 bytes [4.29 GB]
Sector Size:	  512 bytes logical/physical
Device is:		Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA/ATAPI-7 (minor revision not indicated)
Local Time is:	Fri Jul  6 01:04:41 2018 EDT
SMART support is: Unavailable - device lacks SMART capability.


Code:
/dev/ada4
		512			 # sectorsize
		4293918208	  # mediasize in bytes (4.0G)
		8386559		 # mediasize in sectors
		0			   # stripesize
		0			   # stripeoffset
		8319			# Cylinders according to firmware.
		16			  # Heads according to firmware.
		63			  # Sectors according to firmware.
		GIGABYTE i-RAM  # Disk descr.
		4B56363A04B4B3AA93D5	# Disk ident.
		Not_Zoned	   # Zone Mode

Synchronous random writes:
		 0.5 kbytes:	 39.8 usec/IO =	 12.3 Mbytes/s
		   1 kbytes:	 43.3 usec/IO =	 22.6 Mbytes/s
		   2 kbytes:	 50.2 usec/IO =	 38.9 Mbytes/s
		   4 kbytes:	 64.9 usec/IO =	 60.2 Mbytes/s
		   8 kbytes:	100.4 usec/IO =	 77.8 Mbytes/s
		  16 kbytes:	160.6 usec/IO =	 97.3 Mbytes/s
		  32 kbytes:	281.8 usec/IO =	110.9 Mbytes/s
		  64 kbytes:	528.5 usec/IO =	118.3 Mbytes/s
		 128 kbytes:   1016.4 usec/IO =	123.0 Mbytes/s
		 256 kbytes:   2002.6 usec/IO =	124.8 Mbytes/s
		 512 kbytes:   3971.1 usec/IO =	125.9 Mbytes/s
		1024 kbytes:   7907.9 usec/IO =	126.5 Mbytes/s
		2048 kbytes:  15780.5 usec/IO =	126.7 Mbytes/s
		4096 kbytes:  31526.5 usec/IO =	126.9 Mbytes/s
		8192 kbytes:  63016.1 usec/IO =	127.0 Mbytes/s


While it bottlenecks badly at larger record sizes from the SATA1 connection, the smaller sizes (eg: 4K/8K writes) are quite impressive for a device from 2005 - and it'll never run out of P/E cycles.
 
Joined
Dec 29, 2014
Messages
1,135
Now let's move on to a more interesting device. 4GB of DDR memory, backed by a rechargeable battery, and presented via SATA1, I give you:

And I thought i needed to get out more often.... :smile:
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
And I thought i needed to get out more often.... :)

Hey, it was a good card. I'm not certain, but I swear a commercial SAN/NAS vendor (NetApp, maybe?) used this card in their filers for a write-cache.
 

C2Opps

Dabbler
Joined
Jun 11, 2018
Messages
25
Hi All,
For the benchmarks listed here for example the Intel S3500 where it shows " 512 kbytes: 4656.7 usec/IO = 107.4 Mbytes/s " if my workload is almost entirely 512kb synchronous writes will I be able to get 107.4MB/sec worth of synchronous writes to my SLOG and therefore my pool (assuming i use an Intel S3500 as SLOG and the spinning disks can keep up)

Also I was looking at using the Samsung SM863a 240GB as a SLOG (includes Power Loss Protection) The Spec sheet (here) which shows a Max sequential write using 128KB blocks is 450MB/sec - can I therefore safely assume that when using it as an SLOG device and writing 128KB blocks or above that I would get fairly close to that quoted 450MB/sec number - as nowhere does it say they tested using synchronous writes, i say this as the Intel S3500 80GB 2.5" spec here lists 100MB/sec as max sequential writes which is close to the diskinfo benchmark so wondering if I can make that assumption ?

If so and my sync write speed of my pool under the above circumstances with the SM863a would be limited to around 450MB/sec then I'd be very happy to use it as an SLOG as this is more than enough and a good price point - I can't budget for the more expensive PCIe/NVME drives at the moment.

N.B I've emailed Samsung to ask about whether the 450MB/sec is for synchronous writes and also if they could perform the diskinfo -vS test on the drive using a bootable FreeBSD - if they respond I will add the info here :)

cheers,
C2
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Make sure you are not confusing the units - 512KB is not the same as 512 bytes (0.5 KB) that something like "ashift=9" implies. If your workload truly is 512KB synchronous writes, then yes, you should be able to sustain roughly the listed throughput. But if you're going to be putting that kind of sustain write load on it, you should be looking at something with higher write endurance like the S3700 (if you have to stick with SATA.)

For the Samsung stated rate of 450MB/s I doubt that it will be able to handle that at a sustained workload level, but the large block size and PLP on the drive will help it honor flushes quickly.

Ultimately if you can't budget for a faster NVMe device, you'll get what you get. Perhaps a separate thread in Hardware or a build-check with your workload might be of use.
 

C2Opps

Dabbler
Joined
Jun 11, 2018
Messages
25
Thanks HoneyBadger - I'm sure of the 512KB - this Veeam article shows it will be 512KB on current backup Job settings - may be 128KB or 256KB if we tweak the Job settings (this is a FreeNAS server that will act as a Veeam repository)

Also this wouldn't be a constant stream of 450MB/sec just a total of around 1TB per day of backups mostly overnight - the drive spec shows 3.6 DWPD over 5 years for endurance so 1TB being around 4 full writes per day suggests around a 4.5 year lifepsan - by which time I'm sure we will be happy it has lived a good long life

Do you know if it's the case that for drives tested in this thread whether the block size results mean they can sustain that as a SLOG device?
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Do you know if it's the case that for drives tested in this thread whether the block size results mean they can sustain that as a SLOG device?

Yes, that's the entire raison d'etre of this thread - the numbers shared in this thread from the diskinfo -sW results should translate directly into "performance as an SLOG device" for the given block size.

With your workload there's also a strong argument to set a recordsize larger than the default 128KB - 256KB or 512KB may deliver some benefits. Might also want to look into whether or not you get better performance and space consumption based on letting Veeam do the compression vs. sending the data raw and having ZFS do it.

Here's a thread on the Veeam boards discussing compression (and deduplication, which I'm of the mind that you shouldn't use) but we're starting to disgress quite a bit from the original thread intent, so let's get some more results in here!
 
Last edited:

C2Opps

Dabbler
Joined
Jun 11, 2018
Messages
25
Perfect thanks HB! - just wanted to 100% confirm as going to be spending a lot on the hardware and budgets are tight.
I will read up on a larger record size and checkout the thread.
Also once setup I will definitely post my SM863a diskinfo benchmark for you guys!
 
Top