Deduplication is hitting performance way too hard

[IceBoosteR]

Hello out there,

I got some problems and I think this is related to the deduplication implementation of FreeNAS. But before I open a Bug for that, I want to hear your opinion.
First of all, I am aware of anything I am doing, and that deduplication can have its risks and some performance decrease. So far, so good. But in my cases, this is hitting performance too much.
I got 2 systems with different hardware and FN versions, where I can reproduce the problem:

1st Server:
Xeon E3-1225v3
24GB RAM
4x4TB WD Red
FreeNAS 9.10.2-U6

2nd Server:
2x Xeon L5630
56GB RAM
14x450GB HDD
FreeNAS 11.0-U2

So I got deduplication enabled on some datasets. I saved some gigabytes of storage and dedup table is small ~1GB in RAM max.
When I am copying data to a dataset when dedup is ON or VERIFY, the connection drops every minute, sometimes every 30 seconds, sometimes every 2 minutes. Depends on the data transferred. When the connection drops, I mean that NO read or write happen to that dataset at this specific moment, and Windows shows transferspeeds of 0kb/s instead of 100+MB/s. For 15-30 seconds this state is hold and after that time, the copying resumes (photo attached).
On the other hand, I got a ZVOL vor ESXi datastores. I also enable dedup here for saving storage on the VMs. But when heavy write is done, sometimes the VMs freezes and shows me iSCSI error, that they are waiting for the connection. This can also be found in the log of FN, that the iSCSI has I/O problems.
This is not a CPU or RAM problem, also I think the disks are not the bottleneck has normal transfers has always 100+MB/s so I assume it's the deduplication or the implementaion of this.

Now don't tell me that I should not use dedup, but any helpful comment would be appreciate - because the workaround is to disable dedup ;)

 

[m0nkey_]

Now don't tell me that I should not use dedup

Which is exactly what I will tell you, as you're more likely to save more space by using compression. :)

 

[IceBoosteR]

Which is exactly what I will tell you, as you're more likely to save more space by using compression. :)

Hi,
it is depending on files. I am sure, when I am using compression and dedup on multiple windows VMs, I will save also storage because of multiple 1:1 windows binaries. On the other hand, I use dedup for my music, which cannot be compressed (saves about 14% storage).

Regards
Ice

 

[m0nkey_]

Have you considered increasing the block size of your media datasets? If the data is already compressed, for example movie, increasing your block size to 1024K will actually save you some space. In my case, I saved 5GB space by doing this.

Code:

[root@tardis] ~# zfs get recordsize,used,logicalused tank/media/movies
NAME			   PROPERTY	 VALUE	SOURCE
tank/media/movies  recordsize   1M	   local
tank/media/movies  used		 511G	 -
tank/media/movies  logicalused  516G	 -
[root@tardis] ~# 

 

[m0nkey_]

Additionally, you can't change the block size on the fly. You will need to create a new dataset with the 1M blocks. Now, in order to get your data there, you can't go ahead and use zfs send as the data will be written using the same block size of the source. You must use something like rsync to copy the data over.

 

[m0nkey_]

This is an example of a dataset using 128K blocks. Most of the data is likely already compressed, resulting in more disk space being used.

Code:

[root@tardis] ~# zfs get recordsize,used,logicalused tank/shared
NAME		 PROPERTY	 VALUE	SOURCE
tank/shared  recordsize   128K	 default
tank/shared  used		 378G	 -
tank/shared  logicalused  382G	 -

 

[IceBoosteR]

Hi,

thank you for your suggestion on this.
I have compared this with my pool, and the music is fine (1:1 GBs used <-> logicalused).
But I see that I lost a lot of GB elsewhere:

Code:

[root@freenas] /mnt/tank# zfs get recordsize,used,logicalused tank
NAME  PROPERTY  VALUE  SOURCE
tank  recordsize  128K  default
tank  used  8.29T  -
tank  logicalused  8.41T  -

I am searching for the problematic dataset now ;)
This was very helpful!

 

[m0nkey_]

This was very helpful!

Awesome :)

 

[IceBoosteR]

Okey well I'll test a little bit - but back to the OT please :)

 

[m0nkey_]

Okey well I'll test a little bit - but back to the OT please :)

The typical recommendation for deduplication is not to use it. It typically does not solve an issue, unless you're working with absolutely massive datasets and have 128GB RAM or more.

This is why I headed down the path of increasing the block size, etc.

 

[Chris Moore]

Hello out there,

I got some problems and I think this is related to the deduplication implementation of FreeNAS. But before I open a Bug for that, I want to hear your opinion.
First of all, I am aware of anything I am doing, and that deduplication can have its risks and some performance decrease. So far, so good. But in my cases, this is hitting performance too much.
I got 2 systems with different hardware and FN versions, where I can reproduce the problem:

1st Server:
Xeon E3-1225v3
24GB RAM
4x4TB WD Red
FreeNAS 9.10.2-U6

2nd Server:
2x Xeon L5630
56GB RAM
14x450GB HDD
FreeNAS 11.0-U2

So I got deduplication enabled on some datasets. I saved some gigabytes of storage and dedup table is small ~1GB in RAM max.
When I am copying data to a dataset when dedup is ON or VERIFY, the connection drops every minute, sometimes every 30 seconds, sometimes every 2 minutes. Depends on the data transferred. When the connection drops, I mean that NO read or write happen to that dataset at this specific moment, and Windows shows transferspeeds of 0kb/s instead of 100+MB/s. For 15-30 seconds this state is hold and after that time, the copying resumes (photo attached).
On the other hand, I got a ZVOL vor ESXi datastores. I also enable dedup here for saving storage on the VMs. But when heavy write is done, sometimes the VMs freezes and shows me iSCSI error, that they are waiting for the connection. This can also be found in the log of FN, that the iSCSI has I/O problems.
This is not a CPU or RAM problem, also I think the disks are not the bottleneck has normal transfers has always 100+MB/s so I assume it's the deduplication or the implementaion of this.

Now don't tell me that I should not use dedup, but any helpful comment would be appreciate - because the workaround is to disable dedup ;)

So, you either didn't read any of the recommendations of the forum before enabling deduplication or you ignored those recommendations because you know more about it than the collected knowledge and experience of the forum and yet, here you are now having problems that could have been avoided if you had listened to the documented best practices in the forum.

You need more memory for deduplication to work well, probably more than the hardware you have can accommodate.

 

[IceBoosteR]

So, you either didn't read any of the recommendations of the forum before enabling deduplication or you ignored those recommendations because you know more about it than the collected knowledge and experience of the forum and yet, here you are now having problems that could have been avoided if you had listened to the documented best practices in the forum.

You need more memory for deduplication to work well, probably more than the hardware you have can accommodate.

Well I read it, understand it and I did not get the point why I should get more RAM. Dedup-table is small, 56GB of RAM is more than the "A general rule of thumb is 5 GB of RAM per terabyte of deduplicated storage" and other the other system it is less then 500GB where dedup is enabled.
Yes, I do have problems with it, but this is something which is not critical, and on the other hand does dedup worked very well for me in storage savings. I have never read something in the forum like the behaviour I have.
But if this is an expected behaviour because of less RAM I will go with high compression, not that problem. But strange the behaviour looks to me, because the system does nothing while the copy is on 0kb/s.
Thought with meeting the requirements of the thumb rule, maybe I would see slower read/write speeds, but not a drop to zero...