Dedup

[Yamanipanuchi]

So Dedup

Everyone says not to use it, And from everything I've read it completely makes sense.

Two things I cannot find an answer to.

1. You have Dedup turned on, Is there something that can be watched that will indicate you are getting close to a failure? Memory usage? or tools?

2. You have multiple datasets, You have one with Dedup turned on. It fails. You lose that dataset? Or all other datasets on same drive? Or everything Processor/memory goes up in smoke! (Last one there was just for some humor)

Reason I ask is that I want to try it, Setup an extra FreeNas machine and dump a bunch of active data (Probably VM disks, nothing that will be missed) and see what happens.

 

[DrKK]

The guidance on dedup is that you need absurd amounts of RAM to store the dedup pointer table. Upwards of several GB per TB of storage. Once your data is deduped, you can't 'un-dedup' it really, which is why you would not want to turn it on in general.

However, if you set up a dataset for screwing around, and turn dedupe on for that dataset, there's really no particular harm screwing around, and then blowing away the dataset. At least I can't think of any harm. Try it out if you like :)

 

[SweetAndLow]

I wouldn't go filling your pool very much though while you play around. Also your write performance might suck.

 

[Ericloewe]

I'm afraid of the possibility of this going to hell even with just a dataset deduped. Since the dataset has to be imported with the rest of the pool and the only way to interact with a ZFS pool is with ZFS itself, I have the feeling the potential for danger is still there.

So yeah, don't overdo the experiments or be ready to get more RAM to get everything back.

 

[Yamanipanuchi]

Ill be experimenting on a system made specifically to experiment on. I will not have any data on it of specific importance. AKA the VM's and data on it will be created specifically for the experiment.

But I want also be able to watch the statistics as the system most likely will degrade (Because I will be trying to do so)

 

[cyberjock]

Found this with some googling. It should help you with your test system...

http://serverfault.com/questions/533877/how-large-is-my-zfs-dedupe-table-at-the-moment

I had a bunch of info on dedup experiments from when I did some testing on a test box "just to see dedup in action". I wouldn't say it was overly enlightening. It's pretty much exactly how it's described around the forums and the documentation:

• its dangerous
• it performs well, then it suddenly doesn't.
• once the performance has gone off the cliff, MB/sec throughput can be a luxury and shouldn't be expected.