Do I need more ram on production server

blckhm

Dabbler
Joined
Sep 24, 2018
Messages
42
Hello everyone,

A few months ago, we transfered our all static contents to new freenas system which build on supermicro 6048.

Our specs are;
- Supermicro X10DRL-i
- Intel(R) Xeon(R) CPU E5-2630L v3 @ 1.80GHz
- DDR4 ECC 53207MB (cause we virtualized, actually 64gb)
- 10GbaseT network
- LSI3008 HBA
- 20x 8TB Seagate Archive HDD !!!BEWARE!!! SMR sucks! (we used only for backups)
- 4x 12TB WD Gold (serving static contents like jpeg, png, gif etc.)

+ 4 more 12TB drives on the way to expand the pool.

And our ZFS reports are;

Screen Shot 2019-03-28 at 13.31.38.png

When I read the charts, everything seems to be cached from ARC. And our hit ratio populated due Frequently Used pattern.
I think we don't need more memory capacity or any L2ARC under these loads, do I?

Our arc summary looks like that by the way;

Code:
System Memory:

    0.03%    15.95    MiB Active,    1.56%    809.18    MiB Inact
    95.59%    48.41    GiB Wired,    0.00%    0    Bytes Cache
    2.15%    1.09    GiB Free,    0.67%    346.41    MiB Gap

    Real Installed:                56.00    GiB
    Real Available:            92.79%    51.96    GiB
    Real Managed:            97.46%    50.64    GiB

    Logical Total:                56.00    GiB
    Logical Used:            96.65%    54.12    GiB
    Logical Free:            3.35%    1.88    GiB

Kernel Memory:                    3.96    GiB
    Data:                99.05%    3.92    GiB
    Text:                0.95%    38.67    MiB

Kernel Memory Map:                50.64    GiB
    Size:                22.08%    11.18    GiB
    Free:                77.92%    39.46    GiB
                                Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
    Storage pool Version:            5000
    Filesystem Version:            5
    Memory Throttle Count:            0

ARC Misc:
    Deleted:                12.83m
    Mutex Misses:                2.66k
    Evict Skips:                2.66k

ARC Size:                77.78%    38.61    GiB
    Target Size: (Adaptive)        77.73%    38.59    GiB
    Min Size (Hard Limit):        12.50%    6.21    GiB
    Max Size (High Water):        8:1    49.64    GiB

ARC Size Breakdown:
    Recently Used Cache Size:    40.78%    15.75    GiB
    Frequently Used Cache Size:    59.22%    22.87    GiB

ARC Hash Breakdown:
    Elements Max:                2.36m
    Elements Current:        72.84%    1.72m
    Collisions:                4.58m
    Chain Max:                6
    Chains:                    154.29k
                                Page:  2
------------------------------------------------------------------------

ARC Total accesses:                    833.83m
    Cache Hit Ratio:        97.58%    813.62m
    Cache Miss Ratio:        2.42%    20.21m
    Actual Hit Ratio:        97.54%    813.30m

    Data Demand Efficiency:        82.23%    13.89m
    Data Prefetch Efficiency:    16.98%    7.32m

    CACHE HITS BY CACHE LIST:
      Most Recently Used:        3.12%    25.37m
      Most Frequently Used:        96.84%    787.93m
      Most Recently Used Ghost:    0.19%    1.52m
      Most Frequently Used Ghost:    0.07%    609.25k

    CACHE HITS BY DATA TYPE:
      Demand Data:            1.40%    11.42m
      Prefetch Data:        0.15%    1.24m
      Demand Metadata:        98.44%    800.91m
      Prefetch Metadata:        0.01%    42.24k

    CACHE MISSES BY DATA TYPE:
      Demand Data:            12.22%    2.47m
      Prefetch Data:        30.07%    6.08m
      Demand Metadata:        46.97%    9.49m
      Prefetch Metadata:        10.75%    2.17m
                                Page:  3
------------------------------------------------------------------------

                                Page:  4
------------------------------------------------------------------------

DMU Prefetch Efficiency:            200.34m
    Hit Ratio:            5.03%    10.07m
    Miss Ratio:            94.97%    190.27m

                                Page:  5
------------------------------------------------------------------------
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Looks probably reasonable. The ARC hit in particular is spectacular.

So here's the part people don't think about.

Is the pool itself busy? ARC and L2ARC make the most difference on a system where the pool is TOO BUSY.

Run zpool iostat for awhile and WATCH. Especially reads, for this question.

From the CLI type "zpool iostat <yourpoolname> 1"

The interesting bits are the last four (and I specifically prefer the last two) columns, read and write. If the pool is NOT busy, such as you are seeing lots of zeroes, no amount of additional RAM will make your system faster. If you are seeing mostly numbers that end in K, it is unlikely that more RAM will make your system faster. I am guessing one of these two cases applies to you.

We have a variety of rules of thumb about RAM:disk ratios, the historical one being that there should be a GB of RAM for every TB of disk. That puts you solidly in the "should have 256GB RAM" range. But that's a guideline, and it's meant for a fairly active and busy server, and it's really more of a rule that's intended for lower amounts of RAM and disk. If you have a lazy archival/backup server, you might be okay on less. My own feeling is that it's a bit tight, and if time passes and the pool seems slower as data is removed and new data is written, you may start experiencing slowdowns due to ZFS needing to retain more metadata in ARC in order to do a good job of allocating space. I am reasonably certain this won't be a major problem at 256GB of RAM. But DDR4 is pricey. Personally, I'd prefer 128GB for your system. But if it seems happy at 64, just be *aware* that it might slowly degrade and that someday, hopefully when DDR5 is out and DDR4 becomes "cheap", you might need to fill it out.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I would think that slow 1.8GHz processor is the problem.
Are you doing SMB?
 
Joined
Jul 3, 2015
Messages
926
Do you find those SMR drives fall out of the array from time to time for no apparent reason? I had a play with a few a while back and would often have that happen?
 
Joined
Jul 3, 2015
Messages
926
Do you find those SMR drives fall out of the array from time to time for no apparent reason? I had a play with a few a while back and would often have that happen?
Sorry, forget this question. I just looked back at one of your old threads when exactly this was mentioned and I commented on it.

That's old age for you ;)
 

blckhm

Dabbler
Joined
Sep 24, 2018
Messages
42
Looks probably reasonable. The ARC hit in particular is spectacular.

So here's the part people don't think about.

Is the pool itself busy? ARC and L2ARC make the most difference on a system where the pool is TOO BUSY.

Run zpool iostat for awhile and WATCH. Especially reads, for this question.

From the CLI type "zpool iostat <yourpoolname> 1"

The interesting bits are the last four (and I specifically prefer the last two) columns, read and write. If the pool is NOT busy, such as you are seeing lots of zeroes, no amount of additional RAM will make your system faster. If you are seeing mostly numbers that end in K, it is unlikely that more RAM will make your system faster. I am guessing one of these two cases applies to you.

We have a variety of rules of thumb about RAM:disk ratios, the historical one being that there should be a GB of RAM for every TB of disk. That puts you solidly in the "should have 256GB RAM" range. But that's a guideline, and it's meant for a fairly active and busy server, and it's really more of a rule that's intended for lower amounts of RAM and disk. If you have a lazy archival/backup server, you might be okay on less. My own feeling is that it's a bit tight, and if time passes and the pool seems slower as data is removed and new data is written, you may start experiencing slowdowns due to ZFS needing to retain more metadata in ARC in order to do a good job of allocating space. I am reasonably certain this won't be a major problem at 256GB of RAM. But DDR4 is pricey. Personally, I'd prefer 128GB for your system. But if it seems happy at 64, just be *aware* that it might slowly degrade and that someday, hopefully when DDR5 is out and DDR4 becomes "cheap", you might need to fill it out.

Thanks a lot for your replies guys!

When I started to watch my pool, I can't see a lot of K's on read and write. Almost every line ends with 'M'. Hahaha!
So I should consider to upgrade my memory to 128GB asap according your reply.

here is the sample output;

Code:
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2       8.29T  13.5T    312    209  22.8M  3.66M
tank2       8.29T  13.5T    311     55  16.2M   598K
tank2       8.29T  13.5T    135  1.46K  6.49M  28.0M
tank2       8.29T  13.5T    145     23  2.64M   376K
tank2       8.29T  13.5T    134     17  6.09M   222K
tank2       8.29T  13.5T    169     86  7.87M   828K
tank2       8.29T  13.5T    213     24  10.7M   301K
tank2       8.29T  13.5T    143  1.38K  5.24M  28.1M
tank2       8.29T  13.5T    225      9  7.99M   131K
tank2       8.29T  13.5T    243     16  16.5M   222K
tank2       8.29T  13.5T    415     47  34.6M   424K
tank2       8.29T  13.5T    188     35  7.22M   269K
tank2       8.29T  13.5T    181    872  7.68M  12.0M
tank2       8.29T  13.5T    261    522  13.9M  15.4M
tank2       8.29T  13.5T    398      5  31.6M  75.2K
tank2       8.29T  13.5T    140    154  6.06M  3.02M
tank2       8.29T  13.5T    206    131  12.0M  3.17M
tank2       8.29T  13.5T    203    570  5.42M  7.21M
tank2       8.29T  13.5T    129    915  2.35M  24.6M
tank2       8.29T  13.5T    197      9  6.02M   139K
tank2       8.29T  13.5T    160     37  6.34M   349K
tank2       8.29T  13.5T    114     83  3.39M   716K
tank2       8.29T  13.5T    189    213  10.1M  2.52M
tank2       8.29T  13.5T    220  1.06K  14.2M  23.5M
tank2       8.29T  13.5T    198     28  6.74M   331K
tank2       8.29T  13.5T    153      9  6.81M   107K
tank2       8.29T  13.5T    154      6  6.87M  90.2K
tank2       8.29T  13.5T    260     17  10.7M   198K
tank2       8.29T  13.5T    298  1.41K  19.0M  27.9M
tank2       8.29T  13.5T    195      8  11.6M   114K
tank2       8.29T  13.5T    229     75  12.3M   716K
tank2       8.29T  13.5T    344     25  21.7M   176K
tank2       8.29T  13.5T    129      3  3.39M  39.6K
tank2       8.29T  13.5T    199  1.34K  7.26M  28.1M
tank2       8.29T  13.5T    254     16  5.67M   192K
tank2       8.29T  13.5T    211     94  8.19M  2.09M
tank2       8.29T  13.5T    163    158  5.11M  3.90M
tank2       8.29T  13.5T    194      7  9.47M  83.2K
tank2       8.29T  13.5T    237  1.28K  13.0M  27.7M
tank2       8.29T  13.5T    122     16  3.01M   212K
tank2       8.29T  13.5T    215     18  8.24M   169K
tank2       8.29T  13.5T    176     13  9.18M  83.2K
tank2       8.29T  13.5T    142     10  6.87M   162K
tank2       8.29T  13.5T    116  1.30K  4.74M  26.1M
 
Last edited by a moderator:

blckhm

Dabbler
Joined
Sep 24, 2018
Messages
42
I would think that slow 1.8GHz processor is the problem.
Are you doing SMB?

No, just NFS. BTW I don't have any problem with the configuration. I'am just curious about the question that I've asked on title. Do I need more memory on server. I could not find an appropriate answer to my question according these metrics and charts.
 

blckhm

Dabbler
Joined
Sep 24, 2018
Messages
42
Sorry, forget this question. I just looked back at one of your old threads when exactly this was mentioned and I commented on it.

That's old age for you ;)
No problem, I've started to use these 20 disks as glacier storage for backup purposes on my system.
If I bring these drives online under heavy write ops, drives can't handle write operations due to SMR technology. And suddenly detaches the one of them after few days and the detached disk will selected randomly.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Thanks a lot for your replies guys!

When I started to watch my pool, I can't see a lot of K's on read and write. Almost every line ends with 'M'. Hahaha!
So I should consider to upgrade my memory to 128GB asap according your reply.

Maybe. Maybe not. With more traffic, there's more potential for benefit, but if it doesn't seem sluggish, it's still quite possible you wouldn't feel much or any benefit.
 

blckhm

Dabbler
Joined
Sep 24, 2018
Messages
42
Maybe. Maybe not. With more traffic, there's more potential for benefit, but if it doesn't seem sluggish, it's still quite possible you wouldn't feel much or any benefit.
what about l2arc?

should I consider to use l2arc with these metrics after upgrade to 128GB memory?

we have an intel optane nvme ssd 480GB, is it suitable for l2arc purposes?
 

blckhm

Dabbler
Joined
Sep 24, 2018
Messages
42
After we made memory upgrade on server from 60 to 120GB, some of disk charts decrased by half.

Screen Shot 2019-04-05 at 13.24.09.png Screen Shot 2019-04-05 at 13.24.24.png Screen Shot 2019-04-05 at 13.24.45.png Screen Shot 2019-04-05 at 13.25.09.png Screen Shot 2019-04-05 at 13.25.26.png


iostats on pool
Code:
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank2       8.40T  13.4T     75    201  3.65M  3.54M
tank2       8.40T  13.4T    178    671  17.9M  12.9M
tank2       8.40T  13.4T     41    274   590K  2.88M
tank2       8.40T  13.4T     48     19  1.91M   214K
tank2       8.40T  13.4T     51     42  1.92M   520K
tank2       8.40T  13.4T    131     11  8.22M   127K
tank2       8.40T  13.4T     55  1.09K  3.76M  20.4M
tank2       8.40T  13.4T     22      4   631K  70.5K
tank2       8.40T  13.4T     40      5   910K   114K
tank2       8.40T  13.4T    510      0  59.1M  19.6K
tank2       8.40T  13.4T    298      0  31.9M      0
tank2       8.40T  13.4T    101  1.02K  4.59M  18.9M
tank2       8.40T  13.4T     50      8  2.57M   162K
tank2       8.40T  13.4T     93      4  6.25M   111K
tank2       8.40T  13.4T    240    144  26.3M  2.45M
tank2       8.40T  13.4T     20     67   404K   629K
tank2       8.40T  13.4T     46  1.12K   697K  21.0M
tank2       8.40T  13.4T     61      6  2.89M   119K
tank2       8.40T  13.4T     53      7  3.67M   103K
tank2       8.40T  13.4T     34    147  1.12M  2.55M
tank2       8.40T  13.4T     90      0  2.35M      0
tank2       8.40T  13.4T     34    954  1.45M  17.3M
tank2       8.40T  13.4T     87     11  1.16M   186K
tank2       8.40T  13.4T     32      6  1.55M  77.7K
tank2       8.40T  13.4T     61      6  3.69M  98.0K
tank2       8.40T  13.4T     58     14  2.31M   188K
tank2       8.40T  13.4T    131    934  5.59M  16.5M
tank2       8.40T  13.4T     55    145  1.20M  2.62M
tank2       8.40T  13.4T     75      3  1.95M  42.3K
tank2       8.40T  13.4T     62      0  1.46M      0
tank2       8.40T  13.4T    139      2  4.23M  31.7K
tank2       8.40T  13.4T    146  1.21K  5.56M  24.1M
tank2       8.40T  13.4T     63     74  1.43M   729K
tank2       8.40T  13.4T     47     35  1.67M   400K
tank2       8.40T  13.4T     76      0  2.86M      0
tank2       8.40T  13.4T     16      2   290K  39.2K
tank2       8.40T  13.4T    168  1.05K  17.3M  20.6M
tank2       8.40T  13.4T     80      1  1.40M  19.8K
tank2       8.40T  13.4T     75      0  1.04M      0
tank2       8.40T  13.4T    106      4  4.51M  47.1K
tank2       8.40T  13.4T     37      0   495K      0
tank2       8.40T  13.4T     28   1009   886K  19.0M
tank2       8.40T  13.4T     77     63  2.17M   582K
tank2       8.40T  13.4T     67    181  2.90M  2.80M
tank2       8.40T  13.4T     29      5   573K  64.1K
tank2       8.40T  13.4T     41    350   891K  7.38M
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
what about l2arc?

should I consider to use l2arc with these metrics after upgrade to 128GB memory?

we have an intel optane nvme ssd 480GB, is it suitable for l2arc purposes?

Well I think you're seeing that this isn't always an exact science, and that sometimes "try it and see" is the best answer. Does it feel faster? In the end, that's a significant issue in my opinion. You can knock it down to almost all-zero on the reads by adding sufficient ARC/L2ARC, but at significant cost, and if it isn't making it "feel" better. there might not be a point.

So a few comments for you to consider.

As you increase ARC and/or L2ARC, and the volume of reads is reduced, you are also increasing the number of IOPS available for write -- because the pool isn't so busy with reads. People tend not to realize that this means more ARC/L2ARC can improve write speeds.

Your Optane SSD is wasted on L2ARC. It'll certainly do the job, but you could do about the same job at a fraction of the cost with a nice cheap consumer drive.

NOT a recommendation as I haven't tried them, but I would be tempted to pick up one, or probably better yet a pair, of WD Black 500GB WDS500G3X0C which are only around $125. With 300TBW endurance, it's possible that you might burn through them within the warranty period, but SSD prices are in freefall again and you'd be able to replace a failed unit with something better in three or four years. We used to suggest not blowing past a 5:1 L2ARC:ARC ratio, but more recent versions of ZFS have better utilization of ARC for the L2ARC pointers and you should be able to go heavier without a penalty.

Whether or not it's worth it to do that is hard to know. You could always drop the Optane in and see how it worked, and then replace it with cheaper SSD if you liked the results.
 
Top