RAM sufficient for this 64TB (SMR) Server? SSD sizes for cache/log?

[IamSpartacus]

I've got two identical UnRAID servers at home full of 8 x 8TB Seagate SMR drives and 2 x 480GB SSDs for cache. While I had originally assumed FreeNAS would not be a good application for SMRs, I've been hearing of some users who've used them in RAIDz2 vdevs with pretty solid performance. So my plan is to take my backup UnRAID server (VM) and convert it into a FreeNAS server (VM) for testing to see how the performance compares over the next few weeks. If the performance turns out to be poor I'll probably start looking towards saving for some PMR drives.

My question is, one major difference between UnRAID and ZFS is the RAM requirements to attain good performance. My UnRAID server has 32GB of RAM which is probably 4 times as much as I need for UnRAID. On the other hand, for a 64TB (48TB usable) ZFS array it may not suffice. For those who have worked with very large arrays, how much of an impact will RAM have and will 32GB even suffice.

Server specs:

SuperMicro X10SDV-7TP4F (Xeon D-1537)
32GB DDR4-2133 Registered ECC RAM
Dual 10Gb SFP+ ports
LSI2116 HBA
8 x 8TB Seagate SMR drives
2 x 480GB Intel 730 SSDs

Type of data being stored:

70% Media
10% Backups
10% Surveillance (thinking of putting two WD Purples in a ZFS mirror for this)
10% Personal Data (Documents, Pictures, Software, etc.).

 

[Ericloewe]

It should be okay.

 

[IamSpartacus]

It should be okay.

Okay as in it will function but crave more performance or the difference between 32 and 64GB of RAM will not be noticeable in real world applications?

 

[Dice]

Okay as in it will function but crave more performance or the difference between 32 and 64GB of RAM will not be noticeable in real world applications?

From my experience there is no sensation of "linear increase" in performance related to amount of RAM. Either it works fine, or it bogs down. Nothing in between.
If there is enough RAM for the current usecase performance will not be augmented particularly by adding RAM.
On the other hand, when RAM is lacking, you'll experience the server bogging down to a crawl a while in during an operation and ...a while after to figure out what parts of the pool you use the most frequently.
This said - "once per migration" mega-file-shuffles are doomed to show signs of not enough RAM. However, once migrated and the file moves are done, the system will work just fine with a lesser amount of RAM.
The RAM utilization is <that> dependent on user patterns.

For some reference, I started out on 16GB RAM with a 48TB RAW pool. Transfers and reorganization via CIFS would bog down to a crawl and give all sorts of signs of problems. However, never crashed, just took a good while longer than anticipated. Then I upgraded to 48GB RAM and performance significantly increased. However, I still managed to have the machine bog down about 1/5 as often prior to upgrading RAM, when hitting it stupidly hard with huge directories moving around (not done for testing, but for actual organization).
Now that those moves are complete, I happily run the box virtualized on ESXi with 32GB dedicated RAM.

Your 32GB will <be fine> but you cannot really judge the 'every day experience down the line' when exhausting a fresh system with huge "once in a lifetime" transfers. ZFS is pretty smart. But needs a little bit of real user pattern for leverage.

 

[SweetAndLow]

The ram rule it's just a suggestion. After you get past 16GB ram things start to mellow out. You could probably run a 100tb server with that much ram as long as the demand was low from users.

Sent from my Nexus 5X using Tapatalk

 

[IamSpartacus]

Quick questions on SSD sizes for ZIL / L2ARC devices.

I have 2 x 480GB Intel 730 SSDs and a 64GB SuperMicro SATADOM SSD that I can use. I was thinking 64GB SSD for the ZIL and one of the 480GB SSDs for the L2ARC. Thoughts?

I mainly do large file transfers (10-100GB) to and from my server.

 

[SweetAndLow]

Don't bother with a slog or l2arc. You probably don't need them. Slog is great for running vms I've NFS. L2arc is good after you maxed out your system memory.

Slog needs to be high wire write endurance and have battery backup. L2arc can hurt performance if you don't have enough memory.

Sent from my Nexus 5X using Tapatalk

 

[Spearfoot]

Quick questions on SSD sizes for ZIL / L2ARC devices.

I have 2 x 480GB Intel 730 SSDs and a 64GB SuperMicro SATADOM SSD that I can use. I was thinking 64GB SSD for the ZIL and one of the 480GB SSDs for the L2ARC. Thoughts?

I mainly do large file transfers (10-100GB) to and from my server.

Unless you're providing block storage to run virtual machines, you won't need a SLOG device. Still... I'll quote myself on the subject of SLOG devices from another thread:

"Regarding ZIL SLOG devices: these need to have particular characteristics: power protection, low latency, fast write speed, and high durability. Though they'll 'work', your DC S3520s aren't well-suited for this purpose. The S3500-series SSDs are optimized for reads; the S3700-series are optimized for writes, which makes them a better choice as a SLOG device. The good, better, and best Intel selections for SLOG devices run sorta like this:

• Good: Intel SSD DC S3700 Series SATA, inexpensive, 10 drive writes per day
  
• Better: Intel SSD 750 Series PCIe, mid-level price, 70GB writes per day
  
• Best: Intel SSD DC P3700 Series PCIe, costly, 17 drive writes per day

All three have the power protection, low latency, fast writes, and high durability you need in a SLOG device."

An L2ARC device may be handy in your situation and your Intel 730s would serve will for this purpose. Caveat: the standard advice is to max out the system RAM before adding an L2ARC, i.e., you'll gain more benefit from adding RAM than from adding an SSD. RAM is much faster than an SSD, so this makes sense.

Good luck!

 

[IamSpartacus]

Don't bother with a slog or l2arc. You probably don't need them. Slog is great for running vms I've NFS. L2arc is good after you maxed out your system memory

Sent from my Nexus 5X using Tapatalk

Unless you're providing block storage to run virtual machines, you won't need a SLOG device. Still... I'll quote myself on the subject of SLOG devices from another thread:

"Regarding ZIL SLOG devices: these need to have particular characteristics: power protection, low latency, fast write speed, and high durability. Though they'll 'work', your DC S3520s aren't well-suited for this purpose. The S3500-series SSDs are optimized for reads; the S3700-series are optimized for writes, which makes them a better choice as a SLOG device. The good, better, and best Intel selections for SLOG devices run sorta like this:

• Good: Intel SSD DC S3700 Series SATA, inexpensive, 10 drive writes per day
  
• Better: Intel SSD 750 Series PCIe, mid-level price, 70GB writes per day
  
• Best: Intel SSD DC P3700 Series PCIe, costly, 17 drive writes per day

All three have the power protection, low latency, fast writes, and high durability you need in a SLOG device."

An L2ARC device may be handy in your situation and your Intel 730s would serve will for this purpose. Caveat: the standard advice is to max out the system RAM before adding an L2ARC, i.e., you'll gain more benefit from adding RAM than from adding an SSD. RAM is much faster than an SSD, so this makes sense.

Good luck!

Thanks for the advice guys. I'll work on upping RAM before going down the L2ARC route. Looks like SLOG won't be necessary given my workload.