need tips on tuneing my freenas , quad core with 32gb ram 13 disk

[sy5tem]

hello,

i have pretty much finished my "super nas" . with FREENAS, i have used a amd quad core, with 32gb of ram (non ecc) , also have 4x320gb raidz1 , 4x2tb raidz1, 3x1tb raidz1, 2x64gb ssd raid0.

i clicked autotune, and its added some entry in the tunable:

Code:

vfs.zfs.arc_max : 22985134663
vm.kmem_size : 25539038515
vm.kmem_size_max : 31923798144

now here is my zpool list

Code:

NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
SSD       115G   109G  6.23G    94%  1.00x  ONLINE  /mnt
htpc     1.81T   472G  1.35T    25%  1.00x  ONLINE  /mnt
storage  7.25T  2.05T  5.20T    28%  1.16x  ONLINE  /mnt
test     1.16T   148G  1.01T    12%  1.00x  ONLINE  /mnt

and container

Code:

NAME              USED  AVAIL  REFER  MOUNTPOINT
SSD               113G      0    31K  /mnt/SSD
SSD/ssd           113G  4.44G   109G  -
htpc             1.78T   213M    31K  /mnt/htpc
htpc/recording   1.78T  1.32T   472G  -
storage          1.50T  3.69T   256K  /mnt/storage
storage/backup   57.8G  3.69T  57.8G  /mnt/storage/backup [deduplication on]
storage/private   662G  3.69T   662G  /mnt/storage/private
storage/pub       592G  3.69T   592G  /mnt/storage/pub
storage/storage   222G  3.69T   222G  /mnt/storage/storage [high gzip 10]
test              413G   459G   272K  /mnt/test
test/vmstorage    413G   761G   111G  -

since i just turned on deduplication , and now copying back my machines backup,

do i need to change anything in the system or tunable to allow enouf memory for deduplication ?

thank you



also , using dedub , what append if the system would lose power sudently ? is zfs rebuilding the DDT back into memory after a reboot?

 

[cyberjock]

/facepalm. Did you read the release notes for FreeNAS? In particular the part about deduplication...

You are so taking a risk by using deduplication. I hope you have backups.

Edit: I guess I should try to help you with your concern. With regards to tuning, you can do some searching on the forums for what some people have used. But there really isn't anyone better at tuning FreeNAS for your exact configuration and use than you. It's alot of trial and error(and alot of reading). There's no "cheat sheet" for stuff you can use that will work for everyone. If there was, that would be FreeNAS' default settings.

 

[jgreco]

If you architect your backup system correctly, ZFS dedupe will buy you nothing that Phil Karn's dupmerge can't handle too, at basically zero risk to your ZFS system.

 

[sy5tem]

/facepalm. Did you read the release notes for FreeNAS? In particular the part about deduplication...

You are so taking a risk by using deduplication. I hope you have backups.

Edit: I guess I should try to help you with your concern. With regards to tuning, you can do some searching on the forums for what some people have used. But there really isn't anyone better at tuning FreeNAS for your exact configuration and use than you. It's alot of trial and error(and alot of reading). There's no "cheat sheet" for stuff you can use that will work for everyone. If there was, that would be FreeNAS' default settings.

yes i read the readme of freenas, about memory and stuff, i tough that 32gb was enough that's why i went and made 1 dataset with deduplication on, a backup directory fro my windows machine.

i used autotune to tune the system. i have no idea what is good or not good, and why i need to tweak .. i first tough that the system would see the 32gb and use it at will... but nope. that's why i come here and ask these question, take note that i started this year with linux and freebsd . i readed like a madman on an island, to learn all these thing ... lol

that's why i am now searching for some pointers because i honestly don't wanna learn the intricate of the system to this point, this is hardcore!

If you architect your backup system correctly, ZFS dedupe will buy you nothing that Phil Karn's dupmerge can't handle too, at basically zero risk to your ZFS system.

.. hum that's something to install on top of freenas? its seem's nice

 

[jgreco]

No, it's something that administrators have used for years. It takes advantage of hard links to achieve similar effects, and when combined with intelligent use of rsync-based backup methods, you can accomplish file backups without tons of wasted space. The downside is that you're using hard-linked files, which means that if you corrupt the data in a file, all trees that reference that file magically gain the corruption. However, this has worked fine for nearly 20 years for many people - and that long ago, this thing was running on teeny tiny little systems with a few MB of RAM.

So the first time you run something like make_snapshot.sh it will make its single-client tree, but then you run dupmerge to merge that into your tree from all your machines, and then further runs of your snapshot tool are fairly straightforward. Run dupmerge periodically to handle new files.

All of that works on machines that do not require many gigabytes of extra RAM for dedupe tables.