How large a pool would you run with 32GB of RAM?

[Jailer]

So this is the situation I'm facing. I'm getting to the point where I need to plan on an increase in storage space again. Being the digital hoarder that I am I'm currently up to 61% used space on my current pool of 6x8TB drives in a RAIDZ2 configuration. Best Buy currently has a few Easy Store deals going on and my plan was to purchase an additional 6x8TB drives and shuck them to add to my pool. But they also have a REALLY good deal on 18TB drives and I'm tempted to go that route.

My problem is my current system is maxed out on RAM at 32GB. I don't want to spend the money to upgrade my system for more RAM support but I'd really like to take advantage of the sale on those 18TB drives. I do have a few jails running on my current system so I'm afraid that I'll run into memory constraints and possibly performance issues if I add 6 more 18TB drives. I do have another older system sitting behind me that does have 64GB of memory but it's old, an X9SRI with a E5-2680 V2. 4 of the 14 SATA ports on that board are SATA2 to give you an idea of how old it is.

So my question to anyone who has bothered to read through this thread this far is: How large of a pool would you feel comfortable running on 32GB of memory?

 

[morganL]

So this is the situation I'm facing. I'm getting to the point where I need to plan on an increase in storage space again. Being the digital hoarder that I am I'm currently up to 61% used space on my current pool of 6x8TB drives in a RAIDZ2 configuration. Best Buy currently has a few Easy Store deals going on and my plan was to purchase an additional 6x8TB drives and shuck them to add to my pool. But they also have a REALLY good deal on 18TB drives and I'm tempted to go that route.

My problem is my current system is maxed out on RAM at 32GB. I don't want to spend the money to upgrade my system for more RAM support but I'd really like to take advantage of the sale on those 18TB drives. I do have a few jails running on my current system so I'm afraid that I'll run into memory constraints and possibly performance issues if I add 6 more 18TB drives. I do have another older system sitting behind me that does have 64GB of memory but it's old, an X9SRI with a E5-2680 V2. 4 of the 14 SATA ports on that board are SATA2 to give you an idea of how old it is.

So my question to anyone who has bothered to read through this thread this far is: How large of a pool would you feel comfortable running on 32GB of memory?

Rule of thumb is:

1GB per HDD => up to 32 drives as a maximum

For performance, its a difficult question and depends on the need for cache hits. For archive, its not needed. For virtualization storage. you need more ARC/L2ARC. For file storage, single clients don't need much caching, if there are 10s of clients, cache helps.

 

[Arwen]

It depends on the usage. For mostly archival use or read once, even 100TB pool probably would be fine. And if it is for just one or 2 people for occasional usage, same thing, 32GB of RAM should be fine.

Plus, a persistent Metadata only L2ARC / Cache device can speed up somethings, yet does not have to be 3 way Mirrored like a Special vDev should be when using RAID-Z2. Though Metadata only in a L2ARC / Cache device can use more RAM for entries. This is because the Metadata would be smaller than actual data, thus more entries.

Note that what I mean by read once, is like watching movies, but not the same one over and over, (like a child might do).

And all bets are off if using De-Dup.


So, it all comes down to usage and what you expect.

 

[Etorix]

Best Buy currently has a few Easy Store deals going on and my plan was to purchase an additional 6x8TB drives and shuck them to add to my pool. But they also have a REALLY good deal on 18TB drives and I'm tempted to go that route.

If the alternative plan is to replace the current drives, I'd note that 6*18 TB is not much larger than 2*6*8 TB (= 6*16 TB) anyway. Either plan would face similar memory issues, but having less drives, of larger capacity is the better plan for noise and power use.

I do have a few jails running on my current system

How much RAM do these jails use?

Rule of thumb is:

1GB per HDD => up to 32 drives as a maximum

That's an interesting variation on another rule thumb which says "1 GB RAM per TB of storage". Or is it another thing entirely?

 

[Jailer]

It depends on the usage. For mostly archival use or read once, even 100TB pool probably would be fine. And if it is for just one or 2 people for occasional usage, same thing, 32GB of RAM should be fine.

Note that what I mean by read once, is like watching movies, but not the same one over and over, (like a child might do).

Yeah that pretty much sums up my use case, although I do act like a child occasionally so.........

If the alternative plan is to replace the current drives, I'd note that 6*18 TB is not much larger than 2*6*8 TB (= 6*16 TB) anyway. Either plan would face similar memory issues, but having less drives, of larger capacity is the better plan for noise and power use.

My original intent when building this server was to start with 6 drives and add 6 more at a later date. That was about 7 years ago and I'm still at 6 drives. I'd hate to replace the current drives as they are only a couple years old but that thought has crossed my mind. There is also the added financial commitment or maintaining and replacing 12 drives vs 6 as well. The power consumption is less of a concern to me as the total cost of ownership of a large number of drives.

How much RAM do these jails use?

No clue, I've never checked.

 

[Etorix]

So, one (or few) users(s), apparently 1 GbE network (10 GbE, it itself, requires more RAM). If the jails are light, you can certainly run an expanded pool on that (6*18TB, 2*6*8TB or even 6*8+6*18).
Replacing is by vdev, so in a 6*8+6*18 pool, the probable next step would be to replace the six 8TB drives, due to ageing or due to lack of space—not to replace twelve drives in one go. So, regardless whether you choose to add a vdev or to keep a single vdev pool and replace the drives, go for the best deal per TB; do not stick to 8 TB drives because you happen to have 8 TB drives now.