L2ARC Max Size?

kspare

Guru
Joined
Feb 19, 2015
Messages
507
We have 4 servers with 256gb ram, we have been running intel 750 1.2tb nvme but upgrading to 2tb intel 3700. Since then i've had one storage server crash out of no where. Is there really a limit to how big the l2arc can be?
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
Yes. The pointers to L2ARC entries must be in RAM. So the larger the L2ARC, the less L1ARC, (aka ARC). It's balancing act. The rule is to max out memory first, before considering L2ARC.

I don't have an idea what the maximum ratio should be. Your original was about 5 to 1, (1.2TB L2ARC to 256GB RAM). The replacement made it 8 to 1.

Of course, your crash might be un-related to size.

Some new OpenZFS features are coming, (or arrived?), that can help;
  • Compressed ARC, (I don't remember if that was L1 or L2), uses the same compressed data as the source.
  • Allocation classes, allows some interesting SSD options like small files on SSD, meta-data on SSD, etc...
 

kspare

Guru
Joined
Feb 19, 2015
Messages
507
I ready about some of those features coming, I think we will just OP the L2 down to 768gb. Looks like we need to do a little testing.

May even try 640gb 128GB*5....that should leave 128gb to the ARC to do its thing without any competition from the l2arc index for memory?
 
Top