is 16GB non-ECC RAM enough?

[Schakal_No1]

I am currently running 6x3TB on Raid-Z2 with only 2x4GB non-ECC RAM, which is obviously stretching it.
I want to expand the system to 9x3TB Raid-Z2 and install 120GB SSD-Cache and a 10Gbit Nic (array will be rebuild from scratch).
My Board (X10SAE) supports up to 4x8GB ECC and I am considering the following options:
- Option 1: Install two additional 4GB or 8GB modules of non-ECC RAM (~80-120€)
- Option 2: Suck it up and buy 32GB ECC-RAM (~300€)

Obviously the second option is preferred, but a lot more expensive.
My use-Case is the following:

- All personal use, 1-3 clients connected max
- Video streaming (plex, 4k), backups/archive
- for video streaming 1Gbit Speed is enough
- for backups/archive I'd like to have more than 1Gbit, 5Gbit would be good and 10Gbit perfect

 

[wblock]

It sounds like 16G of ECC RAM would be 150€. I'd rather have 16G of ECC than 32G of non-ECC. You are in the fortunate situation of the motherboard already supporting ECC.

You can watch how much is used. If the system swaps much, then buy more. But at that point, you'll know it's necessary through experimentation rather than just guessing.

 

[Arwen]

Don't buy or consider a SSD for L2Cache. The rule of thumb is to max out your RAM first, then, if you still have read performance issues, consider a L2Cache drive.

That said, there are some improvements coming to OpenZFS in regards to L1 & L2 Cache;

1. Compressed cache, (using the original dataset's algorythm)
2. Reduced overhead for the L2Cache's directory, (which must exist in RAM)

But, we won't see those improvements in FreeNAS for a while. So back to the original rule of thumb, max out your RAM first.

 

[Schakal_No1]

Don't buy or consider a SSD for L2Cache. The rule of thumb is to max out your RAM first, then, if you still have read performance issues, consider a L2Cache drive.

I already own the SSD and won't be using it otherwise. Would you still advice against using it?

 

[MrToddsFriends]

I already own the SSD and won't be using it otherwise. Would you still advice against using it?

Please read the ZFS primer if you didn't already do so ("L2ARC does not reduce the need for sufficient RAM. In fact, L2ARC needs RAM to function.").
https://doc.freenas.org/11/zfsprimer.html

- for backups/archive I'd like to have more than 1Gbit, 5Gbit would be good and 10Gbit perfect

5Gbit: Please note that there is no support for Aquantia (-based) NICs in FreeBSD/FreeNAS.
https://forums.freenas.org/index.php?threads/new-cheap-asus-xg-c100c-nic.56160/

 

[Schakal_No1]

5Gbit: Please note that there is no support for Aquantia (-based) NICs in FreeBSD/FreeNAS.
https://forums.freenas.org/index.php?threads/new-cheap-asus-xg-c100c-nic.56160/

I wanted to use Mellanox MNPA19-XTR. I know they are not supported by FreeNAS but I found some people saying the FreeBSD-driver works if you manually use it in FreeNAS.

 

[Arwen]

Intel, Broadcom and others are starting to support 2.5Gbps & 5Gbps speeds. But at present, 10Gbps is the best supported speed higher than gigabit.

I want to see 25Gbps over copper become the new norm!

 

[Schakal_No1]

Intel, Broadcom and others are starting to support 2.5Gbps & 5Gbps speeds. But at present, 10Gbps is the best supported speed higher than gigabit.

MNPA19-XTR is 10Gbit, when I talked about 5Gbit I meant the actual transimssion rate when accounting for raid speed.

 

[Johnnie Black]

I wanted to use Mellanox MNPA19-XTR. I know they are not supported by FreeNAS

They are supported at least since FreeNAS 9.10, I'm using them.

 

[Robert Thomspon]

You could just wait for the inevitable crash of the US economy, and I'll happily sell you 16GB of ECC RAM for $100 USD.... figure wait a few weeks, won't cost more than €8-10.