What are your ARC statistics?

kspare

Guru
Joined
Feb 19, 2015
Messages
507
We have about 30 vms on here, as well as our offsite backup data. We're building another san to get the vm's away from the backup data.

  • Put your data type(s) here...
  • 14.3TiB / 36.2TiB (ESXi)
  • 11.1GiB / 696GiB (freenas-boot)
  • 46.52GiB
    (Adaptive)238.34 (MRU: 43.62GiB, MFU: 2.90GiB) / 64.00GiB
  • Hit ratio -> 81.80% (higher is better)
  • Prefetch -> 23.49% (higher is better)
  • Hit MFU:MRU -> 63.62%:32.86% (higher ratio is better)
  • Hit MRU Ghost -> 0.40% (lower is better)
  • Hit MFU Ghost -> 0.82% (lower is better)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
If it's just the lack of ram, well, it just is.

It just is.

I wouldn't do 16GB of RAM with 45x1TB drives, and I'm betting yours are bigger than 1TB...
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It just is.

I wouldn't do 16GB of RAM with 45x1TB drives, and I'm betting yours are bigger than 1TB...

In theory it ought to work but in practice it'll be spectacularly slow. Even with 1TB'ers.
 

Icy

Cadet
Joined
Mar 8, 2015
Messages
2
  • Home Media/Backup
  • 1.64TiB / 7.25TiB (datastore1)
  • 1.24GiB / 7.44GiB (freenas-boot)
  • 5.53GiB
    (Adaptive)19.93 (MRU: 5.19GiB, MFU: 344.33MiB) / 8.00GiB
  • Hit ratio -> 97.53% (higher is better)
  • Prefetch -> 10.21% (higher is better)
  • Hit MFU:MRU -> 80.24%:1.79% (higher ratio is better)
  • Hit MRU Ghost -> 0.02% (lower is better)
  • Hit MFU Ghost -> 0.07% (lower is better)
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Looks pretty good aside from prefetch, is that an indicator of an issue?
  • 2.24GiB / 136GiB (freenas-boot)
  • 5.41TiB / 130TiB (storage01)
  • 1MiB / 3.62TiB (testvol)
  • 90.55GiB
  • (Adaptive)450.71 (MRU: 84.89GiB, MFU: 5.66GiB) / 128.00GiB
  • Hit ratio -> 78.38% (higher is better)
  • Prefetch -> 0.01% (higher is better)
  • Hit MFU:MRU -> 99.51%:0.47% (higher ratio is better)
  • Hit MRU Ghost -> 0.03% (lower is better)
  • Hit MFU Ghost -> 0.40% (lower is better)
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Gee, I really need to fix the adaptive thing, it's annoying :rolleyes:

Wow, you're definitely the winner of the lowest prefetch :D

I don't know if it indicates an issue but I can see you have a pretty low hit ratio too so I'm guessing you use this server for some VMs because you should have enough RAM given the size of your pool to have a higher hit ratio I think. Maybe the L2ARC do more bad than good? but you should wait for an answer from a more experienced member ;) (cyberjock? you're here? :p)
 
Last edited:

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Gee, I really need to fix the adaptive thing, it's annoying :rolleyes:

Wow, you're definitely the winner of the lowest prefetch :D

I don't know if it indicates an issue but I can see you have a pretty low hit ratio too so I'm guessing you use this server for some VMs because you should have enough RAM given the size of your pool to have a higher hit ratio I think. Maybe the L2ARC do more bad than good? but you should wait for an answer from a more experienced member ;) (cyberjock? you're here? :p)

It has 128GB of RAM, I have another thread in the performance section regarding an issue I have but I won't drag that in here.

Right now there is 15-20TB of sql database backups being transferred to that volume, nothing else yet.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Ah ok, I think I understand what happen: backup server = almost no reads and no reads = no use of prefetching (but need confirmation to be sure) :)

For those who wonder the other thread is this one ;)
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Ah ok, I think I understand what happen: backup server = almost no reads and no reads = no use of prefetching (but need confirmation to be sure) :)

For those who wonder the other thread is this one ;)

Yeah no reads yet really, just copying stuff over :cool:
 

ethebubbeth

Cadet
Joined
Nov 4, 2014
Messages
8
  • Media and Backup
  • 34.9TiB / 65TiB (SDF1)
  • 49.21GiB
    (Adaptive)111.22 (MRU: 42.03GiB, MFU: 7.18GiB) / 64.00GiB
  • Hit ratio -> 97.72% (higher is better)
  • Prefetch -> 81.75% (higher is better)
  • Hit MFU:MRU -> 72.74%:5.06% (higher ratio is better)
  • Hit MRU Ghost -> 0.08% (lower is better)
  • Hit MFU Ghost -> 0.83% (lower is better)
 

itchiTrigger

Dabbler
Joined
Jan 17, 2015
Messages
23
I almost believed you... or not :D

You didn't put the pool name in argument (yeah, I know, it's not checked, but the script doesn't crash so... I'm just a little bit lazy right now...) and it handle only one pool (I've done it quick...). <-- edit: no need to put anything in argument now and it's multi-pool © capable :p

I'll probably edit the script later to add multi-pool capability ;) <-- edit: done :p² happy now? :p³

@BigDave no big deal :) But you should learn how to make and execute a script, mini-tuto:
  • Create a file and copy-paste the content of the pastebin in it, save, quit the editor
  • Make it executable chmod +x theNameOfYourFile
  • To execute it simply do ./theNameOfYourFile
  • NB: for the scripts we use the .sh extension by convention. Be careful to not execute any script found on the net without reading it before to ensure that there is no dangerous commands in it.
Where is the best place to save the scripts? Iv'e seen many different opinions but I would like to hear from the guru's.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Personally I use the root user to SSH (yes, I know, not recommended, whatever :P) so by convenience I put them in /root/ but many users put them on a pool. There is two problems with that I think: if one of your scripts need to be executed before the pools are mounted at startup it'll not work... If the pool fail badly and one of your script is used to tell you if something is wrong on the server (likely, because most of the scripts here are used to do monitoring...) it'll not work...

Besides, I don't like to mix scripts with data purely from a file organisation point of view (it's like mixing video files with your bank files for example, just seems not logical).

The drawback with putting scripts on the system drive is if you re-install you need to re-copy the scripts too, but with the 9.3 update system it's not a problem anymore :)

I also answered this question here with other details ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
For scripts you want to have on an ongoing basis, create a dataset for them and any data they require.
 

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
FN 9.2.17 (yes, I resisted FN 9.3 on this box))
96gb ram
tank1: cifs and 1.8TB iscsi file extend for vsphere. 12 2TB drives as stripe mirror
tank1-1: raidz1 (yes, not z2): backup space for replication to external box


[*]4.37TiB / 10.9TiB (tank1)
[*]3.74TiB / 5.44TiB (tank1-1)
[*]75.19GiB (MRU: 69.63GiB, MFU: 5.56GiB) / 96.00GiB
[*]Hit ratio -> 86.70% (higher is better)
[*]Prefetch -> 32.26% (higher is better)
[*]Hit MFU:MRU -> 68.59%:23.96% (higher ratio is better)
[*]Hit MRU Ghost -> 0.68% (lower is better)
[*]Hit MFU Ghost -> 4.12% (lower is better)

vsphere is not performing quit as it could, will add an m.2 SSD as L2ARC +more ram after the migration to FN 9.3 has been done.
 

Sean Coston

Contributor
Joined
Jul 25, 2014
Messages
128
Hi I am pretty much an OpenBSD / FreeNAS noob, but I'm on my 4th or 5th server rebuild since my first try summer 2014. This is a home server, no "crown jewels" in the data.

I have a question about my setup and why my ZFS Hit ratio is so low.

I started with NAS4Free and then, when I had problems getting some jails up and running, migrated to FreeNAS.
Somewhere along the line I decide that the old repurposed machine I was using wasn't cutting it and bought a dedicated server. It made all the difference, and I couldn't be happier.
Anyway, I have a Lenovo TS140 server with an i3-4130 @ 3.4GHz. I have 20 GByte of DDR3 ECC RAM

I have two pools.

The one called CFS_Media is a stripe of 3 disks - one 2-TByte WD Blue HD and two 500-GByte Maxtor drives. No redundancy because the data is not critical - just a bunch of music, movies, TV series, etc.
The second pool is called CFS and has all my family's important files. It is simply a mirror of two 1-GByte that I backup externally once a week.
This pool also houses all my jails -- plex, couchpotato, owncloud, transmission, and two minecraft servers for my boys. Only couchpotato and transmission are plugin jails. The others manually installed.

So from what I can tell, my server functions perfectly. God I love Plex! Owncloud gives me peace of mind as I've got it set up to sync all of our user files from all of my family's accounts across all of our 4 computers.
I know you are not supposed to use owncloud as part of a backup strategy, but I think it works for our laptops and desktops in case of hard drive failure on one of those... and if one of my kids opens up an email with CryptoLocker virus or something - i've at least got my weekly backups.

Anyway, I digress... my problem is that all over this forum, people say that my ZFS ARC hit ratio should be between 90 and 100%, yet mine is not. Also, the best I can tell, my server never uses any swap but always reserves 10 GByte RAM for it.
Despite my server apparently functioning perfectly, you all would say that it is not. But for what I'm doing, 20 GByte of DDR3 ECC RAM should be enough I would think.

Below are my numbers. Can someone with more knowledge about this help me figure out if there is anything I can optimize?

upload_2015-3-18_19-23-41.png


[*]Put your data type(s) here...
[*]86.0GiB / 928GiB (CFS)
[*]794GiB / 2.71TiB (CFS_Media)
[*]943MiB / 14.9GiB (freenas-boot)
[*]2.23GiB (MRU: 9.09GiB, MFU: 9.09GiB) / 20.00GiB
[*]Hit ratio -> 81.14% (higher is better)
[*]Prefetch -> 2.34% (higher is better)
[*]Hit MFU:MRU -> 96.89%:0.87% (higher ratio is better)
[*]Hit MRU Ghost -> 0.00% (lower is better)
[*]Hit MFU Ghost -> 0.00% (lower is better)
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
This thread is meant to post you ARC stats, not to ask questions about your problems (because the goal is to have a bunch of stats to analyse, not a few stats posts embedded in ten times more questions/problems posts...). I will still answer this time but, please, the next time (that counts for everyone) create your own thread (and just add a link to it here with your stats if you want), thanks ;)

So, it's not a big problem to have a hit ratio between 80 and 90 % instead of 90 to 100 %, it's just that you'll probably benefit of more RAM. If the performance is ok to you then don't change anything ;)

However your ARC, MRU and MFU sizes are very weird. I don't know if it's a bug but it must be (because 9.09 + 9.09 doesn't give 2.23... and the MRU size is exactly equal to the MFU size, coincidence?).

As an aside, please don't cut the list tags when you copy-paste the output, it's why the [*] aren't parsed by the forum in your post and the previous one too, thanks :)

Edit: I've edited the script to fix the "(adaptive)" problem, it should not be displayed anymore if you have a L2ARC ;)
 
Last edited:

Sean Coston

Contributor
Joined
Jul 25, 2014
Messages
128
Thanks for your reply despite my "forum etiquette indiscretions." I had just rebooted. Maybe the numbers for MRU and MFU are real? When the ARC is reset does it arbitrarily set them as the same dataset?

like i said... I'm a noob. don't hold it against me :)
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Well, just after a reboot it's normal that the stats aren't great, it takes days (and even weeks on light loads) for the cache to warm up ;)

"When the ARC is reset does it arbitrarily set them as the same dataset?" this doesn't mean anything, I can't figure out what you want to say, can you explain please?
 

Sean Coston

Contributor
Joined
Jul 25, 2014
Messages
128
Just wondering if after a reboot whether your script calculates the MRU and the MFU as being the same because they are the same... there have not been enough reads to properly establish the values yet and the system reports the entire cache as both MRU and MFU.
But really, as you can see, I don't know enough about any of this to be able to speak intelligently about it. My apologies for confusing people.
 
Top