Help with ZFS performance. Another one of those ;)

Sparx · Apr 22, 2017

So Im new to FreeNAS and all the features and stability looks great. Hope the docker stuff comes soon as I started with Corral just at the same time it was dropped! =)
Im coming from openmediavault and switching mainly because of the instability of that platform.

HW:
Xeon-D 1528
64GB 2133 RDIMM
8x6TB SAS over LSI 3008 HBA conf as RAIDZ2 one big pool with ~10ish volumes under main pool. No snapshots or L2ARC or ZIL.
Intended as file server to be able to write big files over 10GB quickly. Read speed not super high importance.

The pool was created in openmediavault. Pool upgrade was made in corral. Seen no issues neither in 9.10.2.U3 (currently running) nor openmediavault 3.0.70 something with the pool.
So with my omv I never really drop below 800MB/s when writing.

Code:

 openmediavault dd
root@openmediavault:/pool/users/asdf# dd if=/dev/zero of=testfile bs=1M count=1k

1073741824 bytes (1.1 GB) copied, 0.404014 s, 2.7 GB/s
root@openmediavault:/pool/users/asdf# dd if=/dev/zero of=testfile bs=1M count=10k

10737418240 bytes (11 GB) copied, 11.6819 s, 919 MB/s
root@openmediavault:/pool/users/asdf# dd if=/dev/zero of=testfile bs=1M count=100k

107374182400 bytes (107 GB) copied, 135.652 s, 792 MB/s

If I do the same in freenas i get a drop after 4k in same dd command.

Code:

 freenas dd

[root@freenas] /mnt/pool/users/asdf# dd if=/dev/zero of=testfile1 bs=1M count=1k
1073741824 bytes transferred in 0.550282 secs (1951257394 bytes/sec)

[root@freenas] /mnt/pool/users/asdf# dd if=/dev/zero of=testfile1 bs=1M count=4k
4294967296 bytes transferred in 2.256826 secs (1903100833 bytes/sec)

[root@freenas] /mnt/pool/users/asdf# dd if=/dev/zero of=testfile1 bs=1M count=10k
10737418240 bytes transferred in 28.753772 secs (373426423 bytes/sec)

[root@freenas] /mnt/pool/users/asdf# dd if=/dev/zero of=testfile1 bs=1M count=100k
107374182400 bytes transferred in 410.500419 secs (261568996 bytes/sec)

I was trying to compare the zfs tunables but didnt really find the golden ticket. And yes. I have been playing around with some tunables to see if i could fix it.
While Im asking around. Is it possible to have the drives spin down to save some power without zfs going haywire?

Code:

 arc_summary.py

[root@freenas] /mnt/pool/users/asdf# arc_summary.py
System Memory:

		0.51%   322.41  MiB Active,	 0.34%   215.81  MiB Inact
		51.59%  32.08   GiB Wired,	  0.01%   3.88	MiB Cache
		47.56%  29.58   GiB Free,	   0.00%   0	   Bytes Gap

		Real Installed:						 64.00   GiB
		Real Available:				 99.80%  63.87   GiB
		Real Managed:				   97.37%  62.19   GiB

		Logical Total:						  64.00   GiB
		Logical Used:				   53.45%  34.21   GiB
		Logical Free:				   46.55%  29.79   GiB

Kernel Memory:								  380.51  MiB
		Data:						   92.73%  352.83  MiB
		Text:						   7.27%   27.68   MiB

Kernel Memory Map:							  79.84   GiB
		Size:						   39.10%  31.22   GiB
		Free:						   60.90%  48.62   GiB
																Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
		Storage pool Version:				   5000
		Filesystem Version:					 5
		Memory Throttle Count:				  0

ARC Misc:
		Deleted:								36
		Mutex Misses:						   0
		Evict Skips:							0

ARC Size:							   7.51%   4.32	GiB
		Target Size: (Adaptive)		 100.00% 57.48   GiB
		Min Size (Hard Limit):		  13.31%  7.65	GiB
		Max Size (High Water):		  7:1	 57.48   GiB

ARC Size Breakdown:
		Recently Used Cache Size:	   53.36%  30.67   GiB
		Frequently Used Cache Size:	 46.64%  26.81   GiB

ARC Hash Breakdown:
		Elements Max:						   253.17k
		Elements Current:			   16.03%  40.59k
		Collisions:							 4.00k
		Chain Max:							  2
		Chains:								 27
																Page:  2
------------------------------------------------------------------------

ARC Total accesses:									 26.29k
		Cache Hit Ratio:				58.84%  15.47k
		Cache Miss Ratio:			   41.16%  10.82k
		Actual Hit Ratio:			   41.61%  10.94k

		Data Demand Efficiency:		 46.82%  9.11k

		CACHE HITS BY CACHE LIST:
		  Anonymously Used:			 29.29%  4.53k
		  Most Recently Used:		   52.74%  8.16k
		  Most Frequently Used:		 17.97%  2.78k
		  Most Recently Used Ghost:	 0.00%   0
		  Most Frequently Used Ghost:   0.00%   0

		CACHE HITS BY DATA TYPE:
		  Demand Data:				  27.58%  4.27k
		  Prefetch Data:				0.00%   0
		  Demand Metadata:			  43.12%  6.67k
		  Prefetch Metadata:			29.29%  4.53k

		CACHE MISSES BY DATA TYPE:
		  Demand Data:				  44.79%  4.85k
		  Prefetch Data:				0.00%   0
		  Demand Metadata:			  46.34%  5.01k
		  Prefetch Metadata:			8.86%   959
																Page:  3
------------------------------------------------------------------------

																Page:  4
------------------------------------------------------------------------

																Page:  5
------------------------------------------------------------------------

VDEV Cache Summary:							 10.13k
		Hit Ratio:					  39.75%  4.03k
		Miss Ratio:					 22.42%  2.27k
		Delegations:					37.83%  3.83k
																Page:  6
------------------------------------------------------------------------

ZFS Tunable (sysctl):
		kern.maxusers						   4423
		vm.kmem_size							85726392320
		vm.kmem_size_scale					  1
		vm.kmem_size_min						0
		vm.kmem_size_max						1319413950874
		vfs.zfs.vol.unmap_enabled			   1
		vfs.zfs.vol.mode						2
		vfs.zfs.sync_pass_rewrite			   2
		vfs.zfs.sync_pass_dont_compress		 5
		vfs.zfs.sync_pass_deferred_free		 2
		vfs.zfs.zio.dva_throttle_enabled		1
		vfs.zfs.zio.exclude_metadata			0
		vfs.zfs.zio.use_uma					 1
		vfs.zfs.zil_slog_limit				  786432
		vfs.zfs.cache_flush_disable			 0
		vfs.zfs.zil_replay_disable			  0
		vfs.zfs.version.zpl					 5
		vfs.zfs.version.spa					 5000
		vfs.zfs.version.acl					 1
		vfs.zfs.version.ioctl				   7
		vfs.zfs.debug						   0
		vfs.zfs.super_owner					 0
		vfs.zfs.min_auto_ashift				 12
		vfs.zfs.max_auto_ashift				 13
		vfs.zfs.vdev.queue_depth_pct			1000
		vfs.zfs.vdev.write_gap_limit			4096
		vfs.zfs.vdev.read_gap_limit			 32768
		vfs.zfs.vdev.aggregation_limit		  131072
		vfs.zfs.vdev.trim_max_active			64
		vfs.zfs.vdev.trim_min_active			1
		vfs.zfs.vdev.scrub_max_active		   8
		vfs.zfs.vdev.scrub_min_active		   4
		vfs.zfs.vdev.async_write_max_active	 10
		vfs.zfs.vdev.async_write_min_active	 1
		vfs.zfs.vdev.async_read_max_active	  10
		vfs.zfs.vdev.async_read_min_active	  1
		vfs.zfs.vdev.sync_write_max_active	  10
		vfs.zfs.vdev.sync_write_min_active	  10
		vfs.zfs.vdev.sync_read_max_active	   10
		vfs.zfs.vdev.sync_read_min_active	   10
		vfs.zfs.vdev.max_active				 1000
		vfs.zfs.vdev.async_write_active_max_dirty_percent60
		vfs.zfs.vdev.async_write_active_min_dirty_percent30
		vfs.zfs.vdev.mirror.non_rotating_seek_inc1
		vfs.zfs.vdev.mirror.non_rotating_inc	0
		vfs.zfs.vdev.mirror.rotating_seek_offset1048576
		vfs.zfs.vdev.mirror.rotating_seek_inc   5
		vfs.zfs.vdev.mirror.rotating_inc		0
		vfs.zfs.vdev.trim_on_init			   1
		vfs.zfs.vdev.larger_ashift_minimal	  0
		vfs.zfs.vdev.bio_delete_disable		 0
		vfs.zfs.vdev.bio_flush_disable		  0
		vfs.zfs.vdev.cache.bshift			   16
		vfs.zfs.vdev.cache.size				 134217728
		vfs.zfs.vdev.cache.max				  134217728
		vfs.zfs.vdev.metaslabs_per_vdev		 200
		vfs.zfs.vdev.trim_max_pending		   10000
		vfs.zfs.txg.timeout					 5
		vfs.zfs.trim.enabled					1
		vfs.zfs.trim.max_interval			   1
		vfs.zfs.trim.timeout					30
		vfs.zfs.trim.txg_delay				  32
		vfs.zfs.space_map_blksz				 4096
		vfs.zfs.spa_min_slop					134217728
		vfs.zfs.spa_slop_shift				  5
		vfs.zfs.spa_asize_inflation			 24
		vfs.zfs.deadman_enabled				 1
		vfs.zfs.deadman_checktime_ms			5000
		vfs.zfs.deadman_synctime_ms			 1000000
		vfs.zfs.debug_flags					 0
		vfs.zfs.recover						 0
		vfs.zfs.spa_load_verify_data			1
		vfs.zfs.spa_load_verify_metadata		1
		vfs.zfs.spa_load_verify_maxinflight	 10000
		vfs.zfs.ccw_retry_interval			  300
		vfs.zfs.check_hostid					1
		vfs.zfs.mg_fragmentation_threshold	  85
		vfs.zfs.mg_noalloc_threshold			0
		vfs.zfs.condense_pct					200
		vfs.zfs.metaslab.bias_enabled		   1
		vfs.zfs.metaslab.lba_weighting_enabled  1
		vfs.zfs.metaslab.fragmentation_factor_enabled1
		vfs.zfs.metaslab.preload_enabled		1
		vfs.zfs.metaslab.preload_limit		  3
		vfs.zfs.metaslab.unload_delay		   8
		vfs.zfs.metaslab.load_pct			   50
		vfs.zfs.metaslab.min_alloc_size		 33554432
		vfs.zfs.metaslab.df_free_pct			4
		vfs.zfs.metaslab.df_alloc_threshold	 131072
		vfs.zfs.metaslab.debug_unload		   0
		vfs.zfs.metaslab.debug_load			 0
		vfs.zfs.metaslab.fragmentation_threshold70
		vfs.zfs.metaslab.gang_bang			  16777217
		vfs.zfs.free_bpobj_enabled			  1
		vfs.zfs.free_max_blocks				 18446744073709551615
		vfs.zfs.no_scrub_prefetch			   0
		vfs.zfs.no_scrub_io					 0
		vfs.zfs.resilver_min_time_ms			3000
		vfs.zfs.free_min_time_ms				1000
		vfs.zfs.scan_min_time_ms				1000
		vfs.zfs.scan_idle					   50
		vfs.zfs.scrub_delay					 0
		vfs.zfs.resilver_delay				  2
		vfs.zfs.top_maxinflight				 512
		vfs.zfs.delay_scale					 500000
		vfs.zfs.delay_min_dirty_percent		 60
		vfs.zfs.dirty_data_sync				 67108864
		vfs.zfs.dirty_data_max_percent		  10
		vfs.zfs.dirty_data_max_max			  16864249856
		vfs.zfs.dirty_data_max				  6745699942
		vfs.zfs.max_recordsize				  1048576
		vfs.zfs.zfetch.array_rd_sz			  1048576
		vfs.zfs.zfetch.max_distance			 33554432
		vfs.zfs.zfetch.min_sec_reap			 2
		vfs.zfs.zfetch.max_streams			  8
		vfs.zfs.prefetch_disable				1
		vfs.zfs.send_holes_without_birth_time   1
		vfs.zfs.mdcomp_disable				  0
		vfs.zfs.nopwrite_enabled				1
		vfs.zfs.dedup.prefetch				  0
		vfs.zfs.l2c_only_size				   0
		vfs.zfs.mfu_ghost_data_esize			0
		vfs.zfs.mfu_ghost_metadata_esize		0
		vfs.zfs.mfu_ghost_size				  0
		vfs.zfs.mfu_data_esize				  360448
		vfs.zfs.mfu_metadata_esize			  295936
		vfs.zfs.mfu_size						6443520
		vfs.zfs.mru_ghost_data_esize			0
		vfs.zfs.mru_ghost_metadata_esize		0
		vfs.zfs.mru_ghost_size				  0
		vfs.zfs.mru_data_esize				  4424614912
		vfs.zfs.mru_metadata_esize			  3023360
		vfs.zfs.mru_size						4608163840
		vfs.zfs.anon_data_esize				 0
		vfs.zfs.anon_metadata_esize			 0
		vfs.zfs.anon_size					   32768
		vfs.zfs.l2arc_norw					  0
		vfs.zfs.l2arc_feed_again				1
		vfs.zfs.l2arc_noprefetch				0
		vfs.zfs.l2arc_feed_min_ms			   200
		vfs.zfs.l2arc_feed_secs				 1
		vfs.zfs.l2arc_headroom				  2
		vfs.zfs.l2arc_write_boost			   8388608
		vfs.zfs.l2arc_write_max				 8388608
		vfs.zfs.arc_meta_limit				  15430750617
		vfs.zfs.arc_free_target				 113069
		vfs.zfs.compressed_arc_enabled		  1
		vfs.zfs.arc_shrink_shift				7
		vfs.zfs.arc_average_blocksize		   8192
		vfs.zfs.arc_min						 8212864512
		vfs.zfs.arc_max						 61723002470
																Page:  7
------------------------------------------------------------------------

There is something strange with the mapping of disks as far as i can see.

Code:

 zpool status
[root@freenas] /mnt/pool/users/asdf# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  ada0p2	ONLINE	   0	 0	 0

errors: No known data errors

  pool: pool
 state: ONLINE
  scan: scrub canceled on Sat Apr 22 00:38:43 2017
config:

		NAME						  STATE	 READ WRITE CKSUM
		pool						  ONLINE	   0	 0	 0
		  raidz2-0					ONLINE	   0	 0	 0
			da0p1					 ONLINE	   0	 0	 0
			da4p1					 ONLINE	   0	 0	 0
			da2p1					 ONLINE	   0	 0	 0
			da3p1					 ONLINE	   0	 0	 0
			da6p1					 ONLINE	   0	 0	 0
			gpt/zfs-1758ced06bdfef91  ONLINE	   0	 0	 0
			da1p1					 ONLINE	   0	 0	 0
			da5p1					 ONLINE	   0	 0	 0

But the GUI volumes manager shows all drives from da0 to da7 without the gpt partition.

BigDave · Apr 22, 2017

Sparx said:
There is something strange with the mapping of disks as far as i can see.

gpt/zfs-1758ced06bdfef91 ONLINE 0 0 0

But the GUI volumes manager shows all drives from da0 to da7 without the gpt partition.

Are all your drives the same make and model?

A bad drive will have a decided effect on I/O pereformance

I recommend you run # smartctl -t long /dev/da7 and see if the drive shows any errors.

Sparx · Apr 22, 2017

All drives are same yes.
Ill start the long test now. But since the performance is still there in the other os. I kind of believed it would be a freenas issue and not a hw issue.

Sparx · Apr 22, 2017

Itll be a while before the results are in. But there isnt much indication to support a fault so far. Ill post the results for all drives previously.
I see now that I have setup the short test for 8 hour intervals. That seems a bit too much. =)

Code:

da0
[root@freenas] ~# smartctl -a /dev/da0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca656d
Serial number:		95D0A01ZFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:22:13 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  62
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  1006
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		5		 0		 0		  0	 217567.532		   0
write:		 0		0		 0		 0		  0	   8004.347		   0

Non-medium error count:	 3223

Self-test execution status:			 100% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	3656				 - [-   -	-]
# 3  Background short  Completed				   -	3650				 - [-   -	-]
# 4  Background short  Completed				   -	3641				 - [-   -	-]
# 5  Background short  Completed				   -	3632				 - [-   -	-]
# 6  Background short  Completed				   -	3626				 - [-   -	-]
# 7  Background short  Completed				   -	3617				 - [-   -	-]
# 8  Background short  Completed				   -	3608				 - [-   -	-]
# 9  Background short  Completed				   -	3603				 - [-   -	-]
#10  Background short  Completed				   -	3600				 - [-   -	-]
#11  Background long   Completed				   -	3149				 - [-   -	-]
#12  Background long   Completed				   -	1220				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da1
[root@freenas] ~# smartctl -a /dev/da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca0231
Serial number:		95B0A0QVFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:24:56 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  81
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  84
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		2		 0		 0		  0	 156785.230		   0
write:		 0		0		 0		 0		  0	   5951.666		   0

Non-medium error count:	 2090

Self-test execution status:			 100% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	2495				 - [-   -	-]
# 3  Background short  Completed				   -	2489				 - [-   -	-]
# 4  Background short  Completed				   -	2480				 - [-   -	-]
# 5  Background short  Completed				   -	2471				 - [-   -	-]
# 6  Background short  Completed				   -	2465				 - [-   -	-]
# 7  Background short  Completed				   -	2456				 - [-   -	-]
# 8  Background short  Completed				   -	2447				 - [-   -	-]
# 9  Background short  Completed				   -	2442				 - [-   -	-]
#10  Background short  Completed				   -	2439				 - [-   -	-]
#11  Background long   Completed				   -	1986				 - [-   -	-]
#12  Background long   Completed				   -	  58				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da2
[root@freenas] ~# smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca193d
Serial number:		95C0A011FWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:27:14 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  37
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  39
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		2		 0		 0		  0	 150409.209		   0
write:		 0		0		 0		 0		  0	   5806.671		   0

Non-medium error count:	 1781

Self-test execution status:			 100% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	2455				 - [-   -	-]
# 3  Background short  Completed				   -	2449				 - [-   -	-]
# 4  Background short  Completed				   -	2440				 - [-   -	-]
# 5  Background short  Completed				   -	2431				 - [-   -	-]
# 6  Background short  Completed				   -	2425				 - [-   -	-]
# 7  Background short  Completed				   -	2416				 - [-   -	-]
# 8  Background short  Completed				   -	2407				 - [-   -	-]
# 9  Background short  Completed				   -	2401				 - [-   -	-]
#10  Background short  Completed				   -	2398				 - [-   -	-]
#11  Background short  Completed				   -	2355				 - [-   -	-]
#12  Background long   Completed				   -	1947				 - [-   -	-]
#13  Background long   Completed				   -	  17				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da3
[root@freenas] ~# smartctl -a /dev/da3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca02d9
Serial number:		95B0A0RWFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:29:18 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 33 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  51
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  4129
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		3		 0		 0		  0	 408268.724		   0
write:		 0		1		 0		 0		  0	   5998.380		   0

Non-medium error count:	 1841

Self-test execution status:			 100% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	8138				 - [-   -	-]
# 3  Background short  Completed				   -	8132				 - [-   -	-]
# 4  Background short  Completed				   -	8122				 - [-   -	-]
# 5  Background short  Completed				   -	8113				 - [-   -	-]
# 6  Background short  Completed				   -	8107				 - [-   -	-]
# 7  Background short  Completed				   -	8099				 - [-   -	-]
# 8  Background short  Completed				   -	8090				 - [-   -	-]
# 9  Background short  Completed				   -	8084				 - [-   -	-]
#10  Background short  Completed				   -	8081				 - [-   -	-]
#11  Background short  Completed				   -	8022				 - [-   -	-]
#12  Background long   Completed				   -	7631				 - [-   -	-]
#13  Background long   Completed				   -	5701				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

[root@freenas] ~#

Code:

da4
[root@freenas] ~# smartctl -a /dev/da4
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688c9fe35
Serial number:		95B0A0JUFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:32:14 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  60
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  999
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		0		 0		 0		  0	 223803.617		   0
write:		 0		0		 0		 0		  0	   7904.584		   0

Non-medium error count:	 2146

Self-test execution status:			 99% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	3656				 - [-   -	-]
# 3  Background short  Completed				   -	3650				 - [-   -	-]
# 4  Background short  Completed				   -	3641				 - [-   -	-]
# 5  Background short  Completed				   -	3632				 - [-   -	-]
# 6  Background short  Completed				   -	3626				 - [-   -	-]
# 7  Background short  Completed				   -	3617				 - [-   -	-]
# 8  Background short  Completed				   -	3608				 - [-   -	-]
# 9  Background short  Completed				   -	3602				 - [-   -	-]
#10  Background short  Completed				   -	3600				 - [-   -	-]
#11  Background short  Completed				   -	3540				 - [-   -	-]
#12  Background long   Completed				   -	3148				 - [-   -	-]
#13  Background long   Completed				   -	1220				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da5
[root@freenas] ~# smartctl -a /dev/da5
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca0071
Serial number:		95B0A0MMFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:33:26 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 31 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  92
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  1036
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		0		 0		 0		  0	 230241.334		   0
write:		 0		0		 0		 0		  0	   7937.827		   0

Non-medium error count:	 2204

Self-test execution status:			 99% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	3697				 - [-   -	-]
# 3  Background short  Completed				   -	3691				 - [-   -	-]
# 4  Background short  Completed				   -	3682				 - [-   -	-]
# 5  Background short  Completed				   -	3673				 - [-   -	-]
# 6  Background short  Completed				   -	3667				 - [-   -	-]
# 7  Background short  Completed				   -	3658				 - [-   -	-]
# 8  Background short  Completed				   -	3649				 - [-   -	-]
# 9  Background short  Completed				   -	3644				 - [-   -	-]
#10  Background short  Completed				   -	3641				 - [-   -	-]
#11  Background long   Completed				   -	3190				 - [-   -	-]
#12  Background long   Completed				   -	1261				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da6
[root@freenas] ~# smartctl -a /dev/da6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688cab821
Serial number:		95E0A09SFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:33:57 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 38 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  72
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  75
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		2		 0		 0		  0	 156838.952		   0
write:		 0		0		 0		 0		  0	   5915.619		   0

Non-medium error count:	 1994

Self-test execution status:			 99% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	2493				 - [-   -	-]
# 3  Background short  Completed				   -	2487				 - [-   -	-]
# 4  Background short  Completed				   -	2478				 - [-   -	-]
# 5  Background short  Completed				   -	2469				 - [-   -	-]
# 6  Background short  Completed				   -	2463				 - [-   -	-]
# 7  Background short  Completed				   -	2454				 - [-   -	-]
# 8  Background short  Completed				   -	2445				 - [-   -	-]
# 9  Background short  Completed				   -	2440				 - [-   -	-]
#10  Background short  Completed				   -	2437				 - [-   -	-]
#11  Background long   Completed				   -	1985				 - [-   -	-]
#12  Background long   Completed				   -	  56				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da7
[root@freenas] ~# smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca023d
Serial number:		95B0A0QXFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:34:40 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  79
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  82
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		4		 0		 0		  0	 156846.246		   0
write:		 0		0		 0		 0		  0	   5903.981		   0

Non-medium error count:	 1117

Self-test execution status:			 99% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	2494				 - [-   -	-]
# 3  Background short  Completed				   -	2488				 - [-   -	-]
# 4  Background short  Completed				   -	2479				 - [-   -	-]
# 5  Background short  Completed				   -	2470				 - [-   -	-]
# 6  Background short  Completed				   -	2464				 - [-   -	-]
# 7  Background short  Completed				   -	2455				 - [-   -	-]
# 8  Background short  Completed				   -	2446				 - [-   -	-]
# 9  Background short  Completed				   -	2441				 - [-   -	-]
#10  Background short  Completed				   -	2438				 - [-   -	-]
#11  Background long   Completed				   -	1987				 - [-   -	-]
#12  Background long   Completed				   -	  58				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Code:

da7 smartctl -x
[root@freenas] ~# smartctl -x /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca023d
Serial number:		95B0A0QXFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 07:36:24 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled
Read Cache is:		Enabled
Writeback Cache is:   Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 32 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  79
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  82
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		4		 0		 0		  0	 156863.227		   0
write:		 0		0		 0		 0		  0	   5904.006		   0

Non-medium error count:	 1117

Self-test execution status:			 99% of test remaining
SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Self test in progress ...   -	 NOW				 - [-   -	-]
# 2  Background short  Completed				   -	2494				 - [-   -	-]
# 3  Background short  Completed				   -	2488				 - [-   -	-]
# 4  Background short  Completed				   -	2479				 - [-   -	-]
# 5  Background short  Completed				   -	2470				 - [-   -	-]
# 6  Background short  Completed				   -	2464				 - [-   -	-]
# 7  Background short  Completed				   -	2455				 - [-   -	-]
# 8  Background short  Completed				   -	2446				 - [-   -	-]
# 9  Background short  Completed				   -	2441				 - [-   -	-]
#10  Background short  Completed				   -	2438				 - [-   -	-]
#11  Background long   Completed				   -	1987				 - [-   -	-]
#12  Background long   Completed				   -	  58				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Background scan results log
  Status: waiting until BMS interval timer expires
	Accumulated power on time, hours:minutes 2498:45 [149925 minutes]
	Number of background scans performed: 20,  scan progress: 0.00%
	Number of background medium scans performed: 0

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 16
  number of phys = 1
  phy identifier = 0
	attached device type: no device attached
	attached reason: unknown
	reason: unknown
	negotiated logical link rate: phy enabled; unknown
	attached initiator port: ssp=0 stp=0 smp=0
	attached target port: ssp=0 stp=0 smp=0
	SAS address = 0x5000039688ca023e
	attached SAS address = 0x0
	attached phy identifier = 0
	Invalid DWORD count = 942
	Running disparity error count = 929
	Loss of DWORD synchronization = 203
	Phy reset problem = 0
	Phy event descriptors:
	 Invalid word count: 942
	 Running disparity error count: 929
	 Loss of dword synchronization count: 203
	 Phy reset problem count: 0
	 Elasticity buffer overflow count: 0
	 Received abandon-class OPEN_REJECT count: 0
	 Transmitted BREAK count: 0
	 Received BREAK count: 0
	 Transmitted SSP frame error count: 0
	 Received SSP frame error count: 0
relative target port id = 2
  generation code = 16
  number of phys = 1
  phy identifier = 1
	attached device type: SAS or SATA device
	attached reason: loss of dword synchronization
	reason: loss of dword synchronization
	negotiated logical link rate: phy enabled; 12 Gbps
	attached initiator port: ssp=1 stp=1 smp=1
	attached target port: ssp=0 stp=0 smp=0
	SAS address = 0x5000039688ca023f
	attached SAS address = 0x500605b00a753104
	attached phy identifier = 4
	Invalid DWORD count = 2357895
	Running disparity error count = 2289650
	Loss of DWORD synchronization = 65
	Phy reset problem = 0
	Phy event descriptors:
	 Invalid word count: 2357895
	 Running disparity error count: 2289650
	 Loss of dword synchronization count: 65
	 Phy reset problem count: 0
	 Elasticity buffer overflow count: 0
	 Received abandon-class OPEN_REJECT count: 0
	 Transmitted BREAK count: 0
	 Received BREAK count: 0
	 Transmitted SSP frame error count: 0
	 Received SSP frame error count: 0

Sparx · Apr 23, 2017

So here is the result for da7. As far as I can see there are no real faults. All drives are more or less new so there is no surprise there.
Only strangeness is the non medium error count. But that could be that I lived on the dangerous side and upgraded the LSI 3008 firmware/bios when the pool was live? In any case it didnt do anything for the read/write. Dont really know what it indicates. Same before and after...

Code:

[root@freenas] ~# smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   TOSHIBA
Product:			  MG04SCA60EE
Revision:			 0101
Compliance:		   SPC-4
User Capacity:		6,001,175,126,016 bytes [6.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:		7200 rpm
Form Factor:		  3.5 inches
Logical Unit id:	  0x5000039688ca023d
Serial number:		95B0A0QXFWWB
Device type:		  disk
Transport protocol:   SAS (SPL-3)
Local Time is:		Sun Apr 23 23:19:39 2017 CEST
SMART support is:	 Available - device has SMART capability.
SMART support is:	 Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:	 31 C
Drive Trip Temperature:		65 C

Manufactured in week 37 of year 2015
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  79
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  82
Elements in grown defect list: 0

Error counter log:
		   Errors Corrected by		   Total   Correction	 Gigabytes	Total
			   ECC		  rereads/	errors   algorithm	  processed	uncorrected
		   fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:		  0		4		 0		 0		  0	 162765.973		   0
write:		 0		0		 0		 0		  0	   5938.466		   0

Non-medium error count:	 1117

SMART Self-test log
Num  Test			  Status				 segment  LifeTime  LBA_first_err [SK ASC ASQ]
	 Description							  number   (hours)
# 1  Background long   Completed				   -	2512				 - [-   -	-]
# 2  Background short  Completed				   -	2494				 - [-   -	-]
# 3  Background short  Completed				   -	2488				 - [-   -	-]
# 4  Background short  Completed				   -	2479				 - [-   -	-]
# 5  Background short  Completed				   -	2470				 - [-   -	-]
# 6  Background short  Completed				   -	2464				 - [-   -	-]
# 7  Background short  Completed				   -	2455				 - [-   -	-]
# 8  Background short  Completed				   -	2446				 - [-   -	-]
# 9  Background short  Completed				   -	2441				 - [-   -	-]
#10  Background short  Completed				   -	2438				 - [-   -	-]
#11  Background long   Completed				   -	1987				 - [-   -	-]
#12  Background long   Completed				   -	  58				 - [-   -	-]

Long (extended) Self Test duration: 45470 seconds [757.8 minutes]

Sparx · Apr 28, 2017

So. I guess everyone is very busy with release of 11. But any ways. I bit the bullet and trashed the old pool and made a new one from the gui of freenas. Not any big diff. Getting about the same speeds. One thing i did was to make the pool smaller so its now 6x6TB big. Leaving 1 for hot spare and 1 backup drive.

Code:

 New pool results.
[root@freenas] /mnt/pool/users/asdf# dd if=/dev/zero of=testfile bs=1M count=100k
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 386.518291 secs (277798451 bytes/sec)

Code:

 Result from single backup drive
[root@freenas] /mnt/fake# dd if=/dev/zero of=testfile bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 34.950667 secs (307216403 bytes/sec)

Code:

 checked the write cache
[root@freenas] /mnt/fake# camcontrol modepage da0 -m 0x08 -e
IC:  0
WCE:  1
RCD:  0

So basically the ZFS-raid2 of 6 drives is slower than a single drive. That really shouldnt be the case right?
Plz halp

Dice · Apr 29, 2017

Here you'll find soem clues to additional investigation if a single drive causes problem:
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

Sparx · Apr 29, 2017

I dont have any single drive that causes problems. Not as far as i can tell.

Sparx · Apr 29, 2017

The server that the script points to seems to be down at the moment. Does anyone have this script?

Edit: Pebcak. Downloading it now.

Dice · Apr 29, 2017

Sparx said:
Downloading it now.

Can you tell how? I did not realize the url in the resource appeared dead.

Sparx · Apr 29, 2017

The link is just faulty. A ":" is missing.

Code:

http://ftp//ftp.sol.net/incoming/solnet-array-test-v2.sh
ftp://ftp.sol.net/incoming/solnet-array-test-v2.sh

Sparx · Apr 29, 2017

Gnnnnnnnnn.. So it seems I am wrong about da7. For some reason it seems faulty from the script also. Cant really understand why.

Code:


Selected disks: da0 da1 da2 da3 da4 da5 da6 da7 ada0
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 0 lun 0 (pass0,da0)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 1 lun 0 (pass1,da1)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 2 lun 0 (pass2,da2)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 3 lun 0 (pass3,da3)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 4 lun 0 (pass4,da4)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 5 lun 0 (pass5,da5)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 6 lun 0 (pass6,da6)
<TOSHIBA MG04SCA60EE 0101>		 at scbus0 target 7 lun 0 (pass7,da7)
<SAMSUNG MZ7LM480HCHP-00003 GXT3003Q>  at scbus3 target 0 lun 0 (pass8,ada0)
Is this correct? (y/N): y
Performing initial serial array read (baseline speeds)
Sat Apr 29 14:32:59 CEST 2017
Sat Apr 29 14:53:18 CEST 2017
Completed: initial serial array read (baseline speeds)
!!ERROR!! dd: /dev/da7: Input/output error

Array's average speed is 188.699 MB/sec per disk

Disk	Disk Size  MB/sec %ofAvg
------- ---------- ------ ------
da0	  5723166MB	199	105
da1	  5723166MB	201	107
da2	  5723166MB	198	105
da3	  5723166MB	198	105
da4	  5723166MB	196	104
da5	  5723166MB	200	106
da6	  5723166MB	202	107
da7	  5723166MB	  0	  0 --SLOW--
ada0	  457862MB	304	161 ++FAST++

Will try and replace/resilver without da7. Is it possible to do a swap for the hot spare from GUI?

Dice · Apr 29, 2017

Good find.

*cough*

Sparx said:
I dont have any single drive that causes problems. Not as far as i can tell.

Sparx · Apr 29, 2017

Oh man I have been fighting so much with this bloody array. I was sure there wasnt any problem with the drives since they are new and since i had good performance on the other os. Ill come back later with the result of replacement. *mumbles a bit more*

SweetAndLow · Apr 29, 2017

Run bad blocks on all disks to burn things in. Also run your tests on a dataset with compression of and ignore this small runs you need to use a 25GB+ file to get meaningful results.

Sent from my Nexus 5X using Tapatalk

Sparx · Apr 29, 2017

Didnt run badblocks. But that could be a later thing. Compression was off for /dev/zero tests. And dd if=blabla of=blabla bs=1M count=100k should be 100GB right?

SweetAndLow · Apr 29, 2017

Sparx said:
Didnt run badblocks. But that could be a later thing. Compression was off for /dev/zero tests. And dd if=blabla of=blabla bs=1M count=100k should be 100GB right?

Sounds good!

Sent from my Nexus 5X using Tapatalk

Sparx · Apr 29, 2017

So its not the fastest ever resilvering I guess. But I suppose its because of the broken disk. Might have been better to just rip the slow one out and resilver after. There are some checksum faults in the spare. Looks strange. But shouldnt be any problem since its a raidz2(?).
Hopefully a scrub will fix any eventual problems.

Code:

  pool: pool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
		continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Apr 29 15:09:16 2017
		3.43T scanned out of 12.3T at 57.5M/s, 45h10m to go
		585G resilvered, 27.79% done
config:

		NAME											  STATE	 READ WRITE CKSUM
		pool											  ONLINE	   0	 0	 0
		  raidz2-0										ONLINE	   0	 0	 0
			spare-0									   ONLINE	   0	 0	21
			  gptid/1b2489af-29f8-11e7-a752-0cc47ac2d7a2  ONLINE	   0	 0	 0  (resilvering)
			  gptid/25868a4b-29f8-11e7-a752-0cc47ac2d7a2  ONLINE	   0	 0	 0  (resilvering)
			gptid/1cd5d458-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/1e8f9c8b-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/20491adb-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/2208be5c-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/23c4b81c-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
		spares
		  13913525625398401322							INUSE	 was /dev/gptid/25868a4b-29f8-11e7-a752-0cc47ac2d7a2

SweetAndLow · Apr 30, 2017

Sparx said:

So its not the fastest ever resilvering I guess. But I suppose its because of the broken disk. Might have been better to just rip the slow one out and resilver after. There are some checksum faults in the spare. Looks strange. But shouldnt be any problem since its a raidz2(?).
Hopefully a scrub will fix any eventual problems.

Code:

  pool: pool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
		continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Apr 29 15:09:16 2017
		3.43T scanned out of 12.3T at 57.5M/s, 45h10m to go
		585G resilvered, 27.79% done
config:

		NAME											  STATE	 READ WRITE CKSUM
		pool											  ONLINE	   0	 0	 0
		  raidz2-0										ONLINE	   0	 0	 0
			spare-0									   ONLINE	   0	 0	21
			  gptid/1b2489af-29f8-11e7-a752-0cc47ac2d7a2  ONLINE	   0	 0	 0  (resilvering)
			  gptid/25868a4b-29f8-11e7-a752-0cc47ac2d7a2  ONLINE	   0	 0	 0  (resilvering)
			gptid/1cd5d458-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/1e8f9c8b-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/20491adb-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/2208be5c-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
			gptid/23c4b81c-29f8-11e7-a752-0cc47ac2d7a2	ONLINE	   0	 0	 0
		spares
		  13913525625398401322							INUSE	 was /dev/gptid/25868a4b-29f8-11e7-a752-0cc47ac2d7a2

Whoa this is not looking good. Stuff like this doesn't normally happen. I would get that lucky rabbits foot out if I was you.

So you have a backup?

Sent from my Nexus 5X using Tapatalk

Sparx · Apr 30, 2017

SweetAndLow said:
So you have a backup?

Yeah I do. I am running two NAS in parallel so the data isnt my biggest worry actually. I was thinking about trashing the pool again and copying the data once more.
Do you know why I get checksum faults in the spare but not in the drives? Could it be due to the replacement of the faulty drive? That it tries to read from a super slow disk but fails?

Important Announcement for the TrueNAS Community.

Help with ZFS performance. Another one of those ;)

Contributor

FreeNAS Enthusiast

Contributor

Contributor

Contributor

Contributor

Wizard

Contributor

Contributor

Wizard

Contributor

Contributor

Wizard

Contributor

Sweet'NASty

Contributor

Sweet'NASty

Contributor

Sweet'NASty

Contributor

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Help with ZFS performance. Another one of those ;)"

Similar threads