Advise on storage configuration to share to clients that will use space as ZPOOLs

[Brett Bergquist]

I have a FreeNAS 11 setup with 16 2TB drives. The FreeNAS server has 64G of ECC memory as well. The complete specs of the hardware are listed in this post:

https://forums.freenas.org/index.ph...-showing-corruption-after-zpool-status.55178/

I need to share this space with multiple system that will use the shared space for their ZFS pools. I will be sharing the space using ISCSI.

The clients will primarily be running some form of Unix (Solaris/OpenIndian/Linux) that supports ZFS and will be creating one or more ZPOOLs using ISCSI targets.

I would like some thoughts and advice on how best to organize the FreeNAS storage for this type of sharing.

One thought is to create a RAID-10 storage pool on the FreeNAS and share a ZVOL. The ZVOL itself is protected by the RAID-10 mirroring/striping. The problem is the client system will only have a non-mirrored ZPOOL in this case and actual ZFS corruption will not be recoverable.

A second though is to share two ZVOL's to the client systems and then have the client system create a mirrored ZPOOL from these. The problem here is that pretty much twice the space is consumed: the RAID-10 is mirroring and then the client's ZPOOL is mirrored.

Another though it to just share whole disks to the clients and have the clients ZPOOL mirrored there.

Basically I would like the best space utilization, with good performance (that is why I was thinking RAID-10) along with client ZPOOL redundancy.

The client's are pretty much running an J2EE enterprise application with moderate database updates being performed (in the order of 100/second).

This is also a LAB environment that is mimicking customer setups. So while it is not required to have no loss of data and the setup can be rebuilt, it is time consuming, so a setup that can deliver with near no loss of data would be useful.

 

[SweetAndLow]

Do not share using iscsi just to mirror/stripe 2 zvols with zfs again. This is just dumb and makes zero sense. Do not layer zfs on top of zfs unless you like dataloss and problems. Just give them a nfs export to use and be happy. They have no need for iscsi if they are just going to use zfs.

 

[Brett Bergquist]

Do not share using iscsi just to mirror/stripe 2 zvols with zfs again. This is just dumb and makes zero sense. Do not layer zfs on top of zfs unless you like dataloss and problems. Just give them a nfs export to use and be happy. They have no need for iscsi if they are just going to use zfs.

I guess I am not explaining it correctly. My client systems MUST use ZFS as their filesystem. There are operations that rely on ZFS snapshot capability for example.

This is from the "zpool" manual page from Sun/Oracle:

Code:

	 file

		 A regular file. The use of files as a backing  store  is
		 strongly  discouraged.  It  is  designed  primarily  for
		 experimental purposes, as the fault tolerance of a  file
		 is  only  as  good  as  the file system of which it is a
		 part. A file must be specified by a full path.

I don't think a file shared through NFS is appropriate to build the client ZFS ZPOOLs on.

ISCSI will be used for its block level storage capability. The clients will use use the ISCSI targets to create their ZFS ZPOOLs.

So I could access the whole disks on the FreeNAS as ISCSI targets but that also means that the size of the ISCSI targets are the whole disk. Of course on the client side, I can mirror and stripe if I want.

This seems to limit the flexibility of carving out chunks of storage.

 

[SweetAndLow]

I guess I am not explaining it correctly. My client systems MUST use ZFS as their filesystem. There are operations that rely on ZFS snapshot capability for example.

This is from the "zpool" manual page from Sun/Oracle:

Code:

	 file

		 A regular file. The use of files as a backing  store  is
		 strongly  discouraged.  It  is  designed  primarily  for
		 experimental purposes, as the fault tolerance of a  file
		 is  only  as  good  as  the file system of which it is a
		 part. A file must be specified by a full path.

I don't think a file shared through NFS is appropriate to build the client ZFS ZPOOLs on.

ISCSI will be used for its block level storage capability. The clients will use use the ISCSI targets to create their ZFS ZPOOLs.

So I could access the whole disks on the FreeNAS as ISCSI targets but that also means that the size of the ISCSI targets are the whole disk. Of course on the client side, I can mirror and stripe if I want.

This seems to limit the flexibility of carving out chunks of storage.

I'll repeat myself since you didn't get it the first time. This is a terrible idea and should not be done. You want to create a zvol and then use iscsi to expose it to clients that will then create another zfs filesystem on that zvol. You will lose data, it will be slow and it will not work correctly.

 

[Brett Bergquist]

I'll repeat myself since you didn't get it the first time. This is a terrible idea and should not be done. You want to create a zvol and then use iscsi to expose it to clients that will then create another zfs filesystem on that zvol. You will lose data, it will be slow and it will not work correctly.

Can you then explain your suggestion? Use NFS and create a ZVOL on an NFS share file?

The requirement is to use the disk storage on the FreeNAS as ZFS ZPOOL storage on client systems.

I am just trying to understand your suggestion.

 

[SweetAndLow]

Can you then explain your suggestion? Use NFS and create a ZVOL on an NFS share file?

The requirement is to use the disk storage on the FreeNAS as ZFS ZPOOL storage on client systems.

I am just trying to understand your suggestion.

I'm not really suggesting anything because I have no clue what you are trying to do. I just saying you should never group a bunch of zvols together with zfs.

What I think you want is zfs running on your linux systems. If that is the case then you need to put the disks in them and run zfs there not on a nas. If you need a shared storage on the network then you use freenas with smb or nfs.

 

[Spearfoot]

I have a FreeNAS 11 setup with 16 2TB drives. The FreeNAS server has 64G of ECC memory as well. The complete specs of the hardware are listed in this post:

https://forums.freenas.org/index.ph...-showing-corruption-after-zpool-status.55178/

I need to share this space with multiple system that will use the shared space for their ZFS pools. I will be sharing the space using ISCSI.

The clients will primarily be running some form of Unix (Solaris/OpenIndian/Linux) that supports ZFS and will be creating one or more ZPOOLs using ISCSI targets.

I would like some thoughts and advice on how best to organize the FreeNAS storage for this type of sharing.

One thought is to create a RAID-10 storage pool on the FreeNAS and share a ZVOL. The ZVOL itself is protected by the RAID-10 mirroring/striping. The problem is the client system will only have a non-mirrored ZPOOL in this case and actual ZFS corruption will not be recoverable.

A second though is to share two ZVOL's to the client systems and then have the client system create a mirrored ZPOOL from these. The problem here is that pretty much twice the space is consumed: the RAID-10 is mirroring and then the client's ZPOOL is mirrored.

Another though it to just share whole disks to the clients and have the clients ZPOOL mirrored there.

Basically I would like the best space utilization, with good performance (that is why I was thinking RAID-10) along with client ZPOOL redundancy.

The client's are pretty much running an J2EE enterprise application with moderate database updates being performed (in the order of 100/second).

This is also a LAB environment that is mimicking customer setups. So while it is not required to have no loss of data and the setup can be rebuilt, it is time consuming, so a setup that can deliver with near no loss of data would be useful.

iSCSI provides block storage across the network, but it doesn't give the kind of direct access to the disks that you would want for creating ZFS-based mirrors or RAIDZ'n' arrays or for performing scrubs. Probably the best you can do on your client systems is to create ZFS pools made up of "Single Drive" vdevs based on the iSCSI storage provided by your FreeNAS server, with redundancy and scrubs and so forth handled strictly by the FreeNAS server.

For the FreeNAS server configuration: mirrors will give the most IOPS and you will probably want to install a ZIL SLOG device and force synchronous writes on the iSCSI zvols.

But I agree with @SweetAndLow that you might be better off installing disks on the client systems that require ZFS, instead of trying to provide their storage via iSCSI.

 

[Brett Bergquist]

So this is basically what I am planning to do

http://www.unixarena.com/2014/09/sun-zfs-storage-appliance-administration.html

Substitute the FreeNAS server for the Sun ZFS Storage Appliance and my clients (in one case a Sun/Oracle M3000 server).

I actually have this up and running. For a simple test, I have a 3TB ZVOL carved out of a 7TB RAID-10 volume on the FreeNAS. I set this up as an ISCSI target/extent on the FreeNAS.

On my M3000 I setup the ISCSI initiator and created a ZFS pool on this LUN.

I can copy a 62G data file to the ZFS pool in 6 minutes 4 seconds.

I can copy the 62G data file from this ZFS pool to a local ZFS pool backed by 2 Seagate Savvio 10K.3 drives (mirrored) that make up the ZFS pool in 7 minutes 8 seconds.

I can copy from one local ZFS pool on the M3000 backed by 2 Seagate Savvio 10K.3 drives to another local ZFS pool on the M3000 again backed by 2 Seagate Savvio 10K.3 drives, in 11 minutes 21 seconds.

All simple tests done by

Code:

bash# time cp 2017-05-22_07-00.tar.Z /sanspace/data
bash# time cp /sanspace/data/2017-05-22_07-00.tar.Z /space
bash# time cp /space/2017-05-22_07-00.tar.Z /rpool

where the various ZFS pools look like:

Code:

bash-3.2# zpool list
NAME	   SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
rpool	  279G  92.7G   186G  33%  ONLINE  -
sanspace  2.98T   647G  2.35T  21%  ONLINE  -
space	  832G   148G   684G  17%  ONLINE  -
bash-3.2#

bash-3.2# zpool status
  pool: rpool
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
		still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
		pool will no longer be accessible on older software versions.
 scan: resilvered 85.8G in 0h29m with 0 errors on Wed Aug 19 14:18:02 2015
config:

		NAME		  STATE	 READ WRITE CKSUM
		rpool		 ONLINE	   0	 0	 0
		  mirror-0	ONLINE	   0	 0	 0
			c0t1d0s0  ONLINE	   0	 0	 0
			c0t0d0s0  ONLINE	   0	 0	 0

errors: No known data errors

  pool: sanspace
 state: ONLINE
 scan: scrub repaired 0 in 0h32m with 0 errors on Thu Jun 22 15:13:42 2017
config:

		NAME									 STATE	 READ WRITE CKSUM
		sanspace								 ONLINE	   0	 0	 0
		  c1t6589CFC00000029AADCFEEAA310AE53Bd0  ONLINE	   0	 0	 0

errors: No known data errors

  pool: space
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
		still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
		pool will no longer be accessible on older software versions.
 scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		space	   ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			c0t2d0  ONLINE	   0	 0	 0
			c0t3d0  ONLINE	   0	 0	 0

errors: No known data errors
bash-3.2#

Note between my FreeNAS and the M3000 there is a dedicated 10G link.

My FreeNAS ZFS pool looks like:

Code:

root@freenas:~ # zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
M0			7.25T   363G  6.90T		 -	 2%	 4%  1.00x  ONLINE  /mnt
freenas-boot  14.6G   743M  13.9G		 -	  -	 4%  1.00x  ONLINE  -
root@freenas:~ #

root@freenas:~ # zpool status
  pool: M0
 state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		M0											  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/49ae084c-54fe-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/4d4043bb-54fe-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/0fd0a31c-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/13c808b1-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/531bb992-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/56deec82-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/85ec7848-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/89821c42-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		cache
		  gptid/c311550f-5503-11e7-95e1-00074305936a	ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da19p2	ONLINE	   0	 0	 0

errors: No known data errors
root@freenas:~ #

and the ZVOL looks like:

Code:

root@freenas:~ # zfs list
NAME												  USED  AVAIL  REFER  MOUNTPOINT
M0												   3.05T  3.98T	88K  /mnt/M0
M0/.system										   34.7M  3.98T	96K  legacy
M0/.system/configs-66311c036e824820af44b2dbf4c55f10   392K  3.98T   392K  legacy
M0/.system/cores									   88K  3.98T	88K  legacy
M0/.system/rrd-66311c036e824820af44b2dbf4c55f10	  30.9M  3.98T  30.9M  legacy
M0/.system/samba4									 192K  3.98T   192K  legacy
M0/.system/syslog-66311c036e824820af44b2dbf4c55f10   2.98M  3.98T  2.98M  legacy
M0/jails											   88K  3.98T	88K  /mnt/M0/jails
M0/m3000											 3.05T  6.67T   363G  -
freenas-boot										  743M  13.4G	64K  none
freenas-boot/ROOT									 736M  13.4G	29K  none
freenas-boot/ROOT/Initial-Install					   1K  13.4G   734M  legacy
freenas-boot/ROOT/default							 736M  13.4G   734M  legacy
freenas-boot/grub									6.28M  13.4G  6.28M  legacy
root@freenas:~ #

the "M0/m3000" is the ZVOL

So it does not seem to be working bad for me. From a simple test, it appears that copying to/from the ZFS pool over ISCSI is actually a little faster than copying from one local ZFS pool to another ZFS pool on the same server.

I appreciate all of the input!

 

[SweetAndLow]

So this is basically what I am planning to do

http://www.unixarena.com/2014/09/sun-zfs-storage-appliance-administration.html

Substitute the FreeNAS server for the Sun ZFS Storage Appliance and my clients (in one case a Sun/Oracle M3000 server).

I actually have this up and running. For a simple test, I have a 3TB ZVOL carved out of a 7TB RAID-10 volume on the FreeNAS. I set this up as an ISCSI target/extent on the FreeNAS.

On my M3000 I setup the ISCSI initiator and created a ZFS pool on this LUN.

I can copy a 62G data file to the ZFS pool in 6 minutes 4 seconds.

I can copy the 62G data file from this ZFS pool to a local ZFS pool backed by 2 Seagate Savvio 10K.3 drives (mirrored) that make up the ZFS pool in 7 minutes 8 seconds.

I can copy from one local ZFS pool on the M3000 backed by 2 Seagate Savvio 10K.3 drives to another local ZFS pool on the M3000 again backed by 2 Seagate Savvio 10K.3 drives, in 11 minutes 21 seconds.

All simple tests done by

Code:

bash# time cp 2017-05-22_07-00.tar.Z /sanspace/data
bash# time cp /sanspace/data/2017-05-22_07-00.tar.Z /space
bash# time cp /space/2017-05-22_07-00.tar.Z /rpool

where the various ZFS pools look like:

Code:

bash-3.2# zpool list
NAME	   SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
rpool	  279G  92.7G   186G  33%  ONLINE  -
sanspace  2.98T   647G  2.35T  21%  ONLINE  -
space	  832G   148G   684G  17%  ONLINE  -
bash-3.2#

bash-3.2# zpool status
  pool: rpool
state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
		still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
		pool will no longer be accessible on older software versions.
scan: resilvered 85.8G in 0h29m with 0 errors on Wed Aug 19 14:18:02 2015
config:

		NAME		  STATE	 READ WRITE CKSUM
		rpool		 ONLINE	   0	 0	 0
		  mirror-0	ONLINE	   0	 0	 0
			c0t1d0s0  ONLINE	   0	 0	 0
			c0t0d0s0  ONLINE	   0	 0	 0

errors: No known data errors

  pool: sanspace
state: ONLINE
scan: scrub repaired 0 in 0h32m with 0 errors on Thu Jun 22 15:13:42 2017
config:

		NAME									 STATE	 READ WRITE CKSUM
		sanspace								 ONLINE	   0	 0	 0
		  c1t6589CFC00000029AADCFEEAA310AE53Bd0  ONLINE	   0	 0	 0

errors: No known data errors

  pool: space
state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
		still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
		pool will no longer be accessible on older software versions.
scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		space	   ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			c0t2d0  ONLINE	   0	 0	 0
			c0t3d0  ONLINE	   0	 0	 0

errors: No known data errors
bash-3.2#

Note between my FreeNAS and the M3000 there is a dedicated 10G link.

My FreeNAS ZFS pool looks like:

Code:

root@freenas:~ # zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
M0			7.25T   363G  6.90T		 -	 2%	 4%  1.00x  ONLINE  /mnt
freenas-boot  14.6G   743M  13.9G		 -	  -	 4%  1.00x  ONLINE  -
root@freenas:~ #

root@freenas:~ # zpool status
  pool: M0
state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		M0											  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/49ae084c-54fe-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/4d4043bb-54fe-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/0fd0a31c-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/13c808b1-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/531bb992-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/56deec82-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/85ec7848-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
			gptid/89821c42-54ff-11e7-bdd3-00074305936a  ONLINE	   0	 0	 0
		cache
		  gptid/c311550f-5503-11e7-95e1-00074305936a	ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da19p2	ONLINE	   0	 0	 0

errors: No known data errors
root@freenas:~ #

and the ZVOL looks like:

Code:

root@freenas:~ # zfs list
NAME												  USED  AVAIL  REFER  MOUNTPOINT
M0												   3.05T  3.98T	88K  /mnt/M0
M0/.system										   34.7M  3.98T	96K  legacy
M0/.system/configs-66311c036e824820af44b2dbf4c55f10   392K  3.98T   392K  legacy
M0/.system/cores									   88K  3.98T	88K  legacy
M0/.system/rrd-66311c036e824820af44b2dbf4c55f10	  30.9M  3.98T  30.9M  legacy
M0/.system/samba4									 192K  3.98T   192K  legacy
M0/.system/syslog-66311c036e824820af44b2dbf4c55f10   2.98M  3.98T  2.98M  legacy
M0/jails											   88K  3.98T	88K  /mnt/M0/jails
M0/m3000											 3.05T  6.67T   363G  -
freenas-boot										  743M  13.4G	64K  none
freenas-boot/ROOT									 736M  13.4G	29K  none
freenas-boot/ROOT/Initial-Install					   1K  13.4G   734M  legacy
freenas-boot/ROOT/default							 736M  13.4G   734M  legacy
freenas-boot/grub									6.28M  13.4G  6.28M  legacy
root@freenas:~ #

the "M0/m3000" is the ZVOL

So it does not seem to be working bad for me. From a simple test, it appears that copying to/from the ZFS pool over ISCSI is actually a little faster than copying from one local ZFS pool to another ZFS pool on the same server.

I appreciate all of the input!

Do what your want but I would fire someone for doing this as a storage admin.

Sent from my Nexus 5X using Tapatalk