SSD storage freenas Hardware

chetan003

Dabbler
Joined
May 3, 2019
Messages
13
Hello All,

We would like to setup SSD storage with below hardware. please suggest best setup for IOPS performance. we are going to use this storage as iscsi/NFS storage to customers hosted on same DC.


server
Dell 730 XD
cpu: dual Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz
RAM:32 Gb x 16 DDR4 total 512GB
local 16 bay with 2TB crucial SSD
raid card:dell 730 Mini non raid mode for local drive
DELL H830 expander card
NIc: 2 10G x2 in LACP and 3 1G Nic in LACP

we also connecting two ex-pander dell MD 1220 with 24 bay size each.
Both MD will have 24 x 2TB SSD

Both MD will be connected to DELL H830 expander card in Non raid mode.

Please advice, correct hardware setup for SSD storage.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Get rid of the LACP. Get faster CPUs and use striped mirrors.

Is this new hardware or reused? Generally people define performance targets, configuration to meet the target and then buy hardware.
 

chetan003

Dabbler
Joined
May 3, 2019
Messages
13
Dear kdragon75,
Get rid of the LACP. Get faster CPUs and use striped mirrors.

Is this new hardware or reused? Generally people define performance targets, configuration to meet the target and then buy hardware.

Please explain why lacp is unused here ? yes it's new hardware.Basically we would like to use this storage as vmware cluster shared storage connected over iscsi as well we going to sell this storage as external storage connected over private vlan.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Dear kdragon75,


Please explain why lacp is unused here ? yes it's new hardware.Basically we would like to use this storage as vmware cluster shared storage connected over iscsi as well we going to sell this storage as external storage connected over private vlan.

iSCSI uses MPIO (Multi-Path I/O) which has its own redundancy and load-balancing; trying to layer LACP with it will result in poorer performance.

If you plan to offer both protocols I suggest using a quad 10Gb card, with 2x10Gb dedicated to iSCSI (no LACP, use MPIO here) and 2x10Gb dedicated to NFS (do use LACP here)

Other suggested changes to the hardware in addition to @kdragon75 's suggest of "faster CPUs and use mirrors":

Get rid of the RAID controllers entirely. Internal card for that model should be the HBA330, ensure your external card is also an HBA for those MD1220 shelves.

Order your R730XD with the 4x NVMe U.2 hotswap bays up front. Add a minimum of two Optane P4800X/P4801X U.2 devices for SLOG. Preferably four, so that you can support multiple pools or improve performance/fault tolerance.

If this is intended for resale purposes, strongly consider a professional support contract with the vendor (with better SLAs than you are offering your customers, so that you can guarantee the ability to deliver on your end!)

Edit: Please also post the model of the "2TB Crucial SSD" you intend to use. Consider the workload you intend to put on them as well as their rated TBW.
 

chetan003

Dabbler
Joined
May 3, 2019
Messages
13
iSCSI uses MPIO (Multi-Path I/O) which has its own redundancy and load-balancing; trying to layer LACP with it will result in poorer performance.

If you plan to offer both protocols I suggest using a quad 10Gb card, with 2x10Gb dedicated to iSCSI (no LACP, use MPIO here) and 2x10Gb dedicated to NFS (do use LACP here)

Other suggested changes to the hardware in addition to @kdragon75 's suggest of "faster CPUs and use mirrors":

Get rid of the RAID controllers entirely. Internal card for that model should be the HBA330, ensure your external card is also an HBA for those MD1220 shelves.

Order your R730XD with the 4x NVMe U.2 hotswap bays up front. Add a minimum of two Optane P4800X/P4801X U.2 devices for SLOG. Preferably four, so that you can support multiple pools or improve performance/fault tolerance.

If this is intended for resale purposes, strongly consider a professional support contract with the vendor (with better SLAs than you are offering your customers, so that you can guarantee the ability to deliver on your end!)

Edit: Please also post the model of the "2TB Crucial SSD" you intend to use. Consider the workload you intend to put on them as well as their rated TBW.

For Mpio connection does client server require 2 nic with same subnet as target ? since current vmware setup does not have additional Nic for MPIO setup on esxi side. Does MPIO works on with client side single nic ?

Insted of NVMeand slog what about if we upgrade RAM let say DDR4 512GB ? for Cpu we have upgraded to E5 2620 v3 (2.4 GHZ)

We are using HP SSD please find attached SSD config
 

Attachments

  • ssd.JPG
    ssd.JPG
    69.1 KB · Views: 281

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
For Mpio connection does client server require 2 nic with same subnet as target ? since current vmware setup does not have additional Nic for MPIO setup on esxi side. Does MPIO works on with client side single nic ?

Insted of NVMeand slog what about if we upgrade RAM let say DDR4 512GB ? for Cpu we have upgraded to E5 2620 v3 (2.4 GHZ)

We are using HP SSD please find attached SSD config

MPIO will still give you two paths from a single NIC, spreading the workload across both targets; you'll get better load-balancing across the two FreeNAS NICs this way. I would strongly suggest that your ESXi hosts have redundant storage connectivity. This could be via two distinct vmkernel interfaces that have different vmnic failover order (eg: NFS vmkernel has failover vmnic2:vmnic3, iSCSI vmkernel has failover vmnic3:vmnic2) if you have a limited number of ports.

Best practices state that you shouldn't mix NAS (SMB/NFS) and SAN (iSCSI) traffic on the same interfaces, this is where the suggestion to have multiple and redundant uplinks from FreeNAS comes in.

SLOG solves a completely different problem than increasing your ARC size - it is used to deliver your sync writes at a high rate of speed, and if you will be providing iSCSI LUNs or NFS exports to hypervisors, fast sync writes are a must and you should ensure to set sync=always on your iSCSI ZVOLs. With that said, more RAM will never hurt. CPU is also significantly better now at 2.4GHz base clock.

The HP SSD appears to be an SK/Hynix "read intensive" mode with 0.8 DWPD (drive wipes per day) - it is a good device in general but the relatively lower endurance vs a "mixed workload" at ~3DPWD further reinforces the need for SLOG to mitigate write amplification and prolong device life.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
@HoneyBadger is on point as usual. You really need redundant paths to watch host you serve if this is remotely important.
 

MikeyG

Patron
Joined
Dec 8, 2017
Messages
442
If you are thinking of ordering Crucial MX500s, you may want to search for the pending sector errors that people are getting. I got a bunch to use, and while they work fine, it fills up my log with irrelevant errors. Would not buy them again. If there is some enterprise crucial SSD you are buying, then of course disregard as they are probably totally different.
 

chetan003

Dabbler
Joined
May 3, 2019
Messages
13
Thanks mgittelman and HoneyBadger,

We are going to test two Optane P4800X/P4801X devices for write cache. Our total usable space would be 120TB which will be distributed NFS/iscsi z volumes.
Please suggest correc size for Optane P4800X/P4801X devices.


for network part our dell 730xd aprt from external 4 10G Nics we also have onboard 4 1G Nic shall we use those for 2 x 1G for iscsi MPIO and 1x 1G for NFS and last 1G for Frennas Gui access.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Please suggest correc size for Optane P4800X/P4801X devices.

"The biggest you can get, within budget and reason." Larger devices tend to have higher performance and endurance levels, but it doesn't need to be a 2TB device. The link in my signature about SLOG benchmarking has quite a few results, and the cooling system in the R730 should mean any device you use stays cool and doesn't suffer from thermal throttling.

for network part our dell 730xd aprt from external 4 10G Nics we also have onboard 4 1G Nic shall we use those for 2 x 1G for iscsi MPIO and 1x 1G for NFS and last 1G for Frennas Gui access.

Don't mix and match interface speeds - performance will be inconsistent. Just use a 2x10G pair for each of iSCSI and NFS, and make a redundant team of 2x1G for the management interface.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
it is a good device in general but the relatively lower endurance vs a "mixed workload" at ~3DPWD further reinforces the need for SLOG to mitigate write amplification and prolong device life.

That sentence is probably deceptive to a newcomer.

Having a separate SLOG device is better than committing writes to the in-pool ZIL, which would cause write amplification/shorten device life.

But you could also just turn off sync writes and get the same benefit, so this isn't really a "need for SLOG". The justification for SLOG is independent.

We are going to test two Optane P4800X/P4801X devices for write cache.

They won't work for "write cache." SLOG doesn't work that way - it is no kind of cache.

https://www.ixsystems.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Our total usable space would be 120TB which will be distributed NFS/iscsi z volumes.

If "z volumes" means RAIDZ, that's also not recommended.

https://www.ixsystems.com/community...d-why-we-use-mirrors-for-block-storage.44068/
 

chetan003

Dabbler
Joined
May 3, 2019
Messages
13
That sentence is probably deceptive to a newcomer.

Having a separate SLOG device is better than committing writes to the in-pool ZIL, which would cause write amplification/shorten device life.

But you could also just turn off sync writes and get the same benefit, so this isn't really a "need for SLOG". The justification for SLOG is independent.



They won't work for "write cache." SLOG doesn't work that way - it is no kind of cache.

https://www.ixsystems.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/



If "z volumes" means RAIDZ, that's also not recommended.

https://www.ixsystems.com/community...d-why-we-use-mirrors-for-block-storage.44068/


We have order two Nvme devices 1.6 TB in size each
1 Intel p3700
1 Hitachi IMP 16113

Please find diskinfo output for both he devices.
Intel
diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
1600321314816 # mediasize in bytes (1.5T)
3125627568 # mediasize in sectors
131072 # stripesize
0 # stripeoffset
INTEL SSDPEDMD016T4K # Disk descr.
CVFT543300881P6DGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 23.0 usec/IO = 21.2 Mbytes/s
1 kbytes: 22.5 usec/IO = 43.5 Mbytes/s
2 kbytes: 22.4 usec/IO = 87.2 Mbytes/s
4 kbytes: 21.6 usec/IO = 180.9 Mbytes/s
8 kbytes: 24.9 usec/IO = 314.1 Mbytes/s
16 kbytes: 31.3 usec/IO = 499.4 Mbytes/s
32 kbytes: 43.1 usec/IO = 724.5 Mbytes/s
64 kbytes: 68.4 usec/IO = 914.1 Mbytes/s
128 kbytes: 142.3 usec/IO = 878.7 Mbytes/s
256 kbytes: 221.8 usec/IO = 1127.1 Mbytes/s
512 kbytes: 361.1 usec/IO = 1384.7 Mbytes/s
1024 kbytes: 656.9 usec/IO = 1522.4 Mbytes/s
2048 kbytes: 1253.6 usec/IO = 1595.3 Mbytes/s
4096 kbytes: 2457.9 usec/IO = 1627.4 Mbytes/s
8192 kbytes: 4850.0 usec/IO = 1649.5 Mbytes/s
----------------------------
diskinfo -wS /dev/nvd1
/dev/nvd1
512 # sectorsize
1600321314816 # mediasize in bytes (1.5T)
3125627568 # mediasize in sectors
0 # stripesize
0 # stripeoffset
UCSC-F-H16003 # Disk descr.
SDM00000DE7A # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 25.8 usec/IO = 18.9 Mbytes/s
1 kbytes: 25.9 usec/IO = 37.7 Mbytes/s
2 kbytes: 26.5 usec/IO = 73.8 Mbytes/s
4 kbytes: 26.5 usec/IO = 147.4 Mbytes/s
8 kbytes: 30.6 usec/IO = 255.5 Mbytes/s
16 kbytes: 38.5 usec/IO = 406.3 Mbytes/s
32 kbytes: 51.9 usec/IO = 602.6 Mbytes/s
64 kbytes: 104.4 usec/IO = 598.5 Mbytes/s
128 kbytes: 187.2 usec/IO = 667.6 Mbytes/s
256 kbytes: 222.4 usec/IO = 1124.3 Mbytes/s
512 kbytes: 460.6 usec/IO = 1085.5 Mbytes/s
1024 kbytes: 488.2 usec/IO = 2048.5 Mbytes/s
2048 kbytes: 952.8 usec/IO = 2099.0 Mbytes/s
4096 kbytes: 1878.6 usec/IO = 2129.3 Mbytes/s
8192 kbytes: 3751.3 usec/IO = 2132.6 Mbytes/s

I have created 2 Zvol 5TB in size each . One ZVol with sync Off and another one with Sync always. The iscsi drive with sync always performance very pool compare to sync off iscsi Please find attached report.


root@freenas[~]# zpool status
pool: dell730Mini
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
dell730Mini ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b5ecfc17-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/b6aa0a9a-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/b74eaae8-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/b7faa461-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/b8ca3219-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/b9861c04-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/ba6278ca-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/bb2d2353-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/bc1ac869-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
gptid/bd09b311-c193-11e9-ac52-1418772dfffb ONLINE 0 0 0
cache
gptid/4d6258d1-c1a6-11e9-ac52-1418772dfffb ONLINE 0 0 0

root@freenas[~]# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
dell730Mini 2.47G 17.4T 0 163 0 2.83M
raidz2 2.47G 17.4T 0 163 0 2.83M
gptid/b5ecfc17-c193-11e9-ac52-1418772dfffb - - 0 23 29 520K
gptid/b6aa0a9a-c193-11e9-ac52-1418772dfffb - - 0 23 29 521K
gptid/b74eaae8-c193-11e9-ac52-1418772dfffb - - 0 23 29 520K
gptid/b7faa461-c193-11e9-ac52-1418772dfffb - - 0 24 29 520K
gptid/b8ca3219-c193-11e9-ac52-1418772dfffb - - 0 23 29 520K
gptid/b9861c04-c193-11e9-ac52-1418772dfffb - - 0 24 29 521K
gptid/ba6278ca-c193-11e9-ac52-1418772dfffb - - 0 23 29 520K
gptid/bb2d2353-c193-11e9-ac52-1418772dfffb - - 0 24 29 520K
gptid/bc1ac869-c193-11e9-ac52-1418772dfffb - - 0 23 29 520K
gptid/bd09b311-c193-11e9-ac52-1418772dfffb - - 0 23 29 521K
cache - - - - - -
gptid/4d6258d1-c1a6-11e9-ac52-1418772dfffb 366M 1.46T 0 5 283 953K
-------------------------------------- ----- ----- ----- ----- ----- -----
 

Attachments

  • read-write.png
    read-write.png
    443.3 KB · Views: 279

chetan003

Dabbler
Joined
May 3, 2019
Messages
13
E5 2620 v3 (2.4 GHZ)
250GB DDR4 RAM
server :dell 730
DELL H830 expander card
freen nas os install in 480SSD x 2 raid1
NIc: 2 10G x2 and 1G x 4

Ultrastar Data60 WD encloser 14TB x 60 connected to DELL H830 expander card in Non raid mode. Please find smart output for 14TB Wd drive.

root@freenas[~]# smartctl -a /dev/da5
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: WDC
Product: WUH721414AL4204
Revision: C07G
Compliance: SPC-4
User Capacity: 14,000,519,643,136 bytes [14.0 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca25827dbdc
Serial number: 9JGPXK1T
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 18 01:17:12 2019 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
 
Top