Deleting files is extremely slow

[Zombie Twiglet]

Deleting files is extremely slow on my box, it's be bugging me for a while but I've only just had a chance to look into it. It is slow deleting files over the network (smaba, windows file manager gets stuck "calculating") which at first I blamed on smaba (and windows) after the last batch of it playing up I decided to try locally (root, using rm -rf) and it was just as slow. I'm talking 5ish minutes for a 2gb file.

As far as I can see I'm not even touching the ram (58gb free during delete both network and locally).

All other operations are working as expected (read, copy etc...)

Any advice on where to look or where the issue may lie?


Setup details:

11.0-U4 (has happened since 9 when I installed)

Intel Xeon E5-1620V4 2011-V3
64gb ecc ddr
12 x 8tb WD Reds (6x mirrors)
2 x 400gb Intel 750 pice ssd (mirrored, logs)

No compression
No dedupe

Top during delete:

Code:

last pid:  7193;  load averages:  0.22,  0.24,  0.53																																					 up 0+00:36:14  20:11:22
59 processes:  1 running, 58 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 478M Active, 163M Inact, 3419M Wired, 58G Free
ARC: 2168M Total, 555M MFU, 1488M MRU, 166K Anon, 86M Header, 38M Other
Swap: 28G Total, 28G Free

tank:

Code:

root@freenas:~ # zpool status tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 0 in 16h50m with 0 errors on Mon Dec  4 19:50:06 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		tank											ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/36d6fd5d-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/375729b4-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/37e3618c-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/3864333f-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/38e66a6a-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/396cb9aa-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/39ef6448-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/3a778eaf-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/3afce52e-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/3b9151f2-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/3c262404-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/3cae0ed3-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
		logs
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/3ccf675d-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0
			gptid/3ceab89f-3e49-11e7-8ee2-0cc47adad972  ONLINE	   0	 0	 0

errors: No known data errors

 

[wblock]

With compression off, deletes will take longer than normal. But probably not that long. One question is whether you have modified the SMB service settings or SMB share settings. Many guides exist on the web with obsolete advice that should not be followed.

 

[Zombie Twiglet]

As I said, it takes just as long when deleting directly on the FreeNAS box (ssh in and use rm). SMB settings would not affect that. With that said the SMB settings are default.

It also happens with compression turned on (lz4), it was turned off for testing purposes.

 

[wblock]

From the hardware, you are set up for running VMs. Is the file that is being deleted a clone?

 

[Zombie Twiglet]

Nope no VMs or jails, just a plain fileserver. One volume, two datasets (one 25T, one 1.5T). The issue happens on both datasets.

 

[wblock]

So, just to be clear, the 2G file being deleted is just a simple data file, created and written by an application?

 

[Zombie Twiglet]

To be clear this affects all files, not just specific ones. I used a 2gb/5 min as a rough example of the speed.

But yes, these are normal files, either written over SMB from Windows or Linux, or created locally using dd to create some test files.

 

[rs225]

And you're sure you didn't create this dataset from a snapshot?(a clone) or it hasn't had de-dupe on?

 

[Zombie Twiglet]

It was created fresh. Dedupe was enabled when created but turned off shortly after (months ago).

 

[Zombie Twiglet]

Ok, so I was just trying to get some test data so I copied some files from one folder to another test folder and they deleted almost instantly using rm -rf on the freshly created folder, same from windows via smb.

So I made another copy of the files and tried deleting the original files, slow as before.

My guess is that these files were created when dedupe was turned on when I first set everything up which is now turned off (because deletes were slow as hell) which is causing the issue.

Any ideas how to resolve this? Any better solutions to creating a new dataset (without dedupe) then moving everything from one to the other?

 

[rs225]

Well, that makes sense. I think the answer is 'no', so it comes down to how much deleting you have ahead of you. :) And it may also be the case that the penalty will be paid unless you destroy the entire pool. Deleting the dataset all at once would still trigger the same access pattern.

 

[Zombie Twiglet]

Well the migration has begun, wish me luck!

Out of interest does anyone know why dedupe would have extremely slow deletes? And why turning it off would cause the issue to linger?

Thanks for the help folks.

 

[wblock]

Dedup, like cloning, means you end up with blocks that are used by more than one thing. It's not just a list, but something that must be traversed. The more fragmentation and multiple things using those same blocks, the slower it gets. At OpenZFS, there was a talk on improving the deletion speed of cloned datasets. Some of those deletes took minutes, and they had code to fix that. But it's not in ZFS yet.

Dedup is only a net gain in specific circumstances. Kind of like encrypted volumes or SLOG devices.