NAS Hardware Update

[vincentj]

Hi. I recently bought a new motherboard (Asus ws c246 pro), new seagate ironwolf NAS hdd (7x2TB in Raidz2), and soon I will get 64GB of ECC RAM and a NvME Gen3. This nvme (WD RED 250GB SN700 (amazon)) I intend to use as L2ARC cache (bearing in mind that this NAS will have many small/medium sized files and I use SMB). Will I benefit from having this new setup with nvme L2ARC? Thank you.

 

[artlessknave]

Will I benefit from having this new setup with nvme L2ARC?

unlikely. l2arc is the kind of thing that if you don't already know you will benefit from it, you probably wont.
64 GB of RAM is the absolute minimum generally recommended before even *considering* l2arc. RAM (ARC) is your cache. it's dramatically faster than l2arc. that's why it's called level 2 arc. its not as good as arc. usually you max the RAM first, THEN add l2arc if the zfs stats indicate it could be useful.

why buy a workstation motherboard instead of a server motherboard? you lose out on ipmi and management, at the very least. these are designed, surprisingly...for workstations, not server.

you didn't get a CPU?

 

[vincentj]

my budget allowed for this. I have an intel i3-6100 cpu. Sure, obviously the ram cache is always better, but it will inexorably fill up...so instead of fetching data from the hard drives again I thought I'd put l2arc inside my system. 64gb of ram is the maximum i can fit...at the moment

 

[vincentj]

also, I had to think about adding the SLOG as well. for the moment i have seen the intel dcs3700 ssd but it is only 240gb. I also thought, in this case, of an nvme ssd. Advice on this?

 

[artlessknave]

SLOG is usually even less useful than L2ARC and even more misunderstood. a SLOG only needs to be about 6-10GB (unless you have some ludicrous amount of RAM maybe). SLOG will ONLY apply to SYNC writes, while SMB, one of the most common file sharing protocols, is incapable of sync writes on its own.
SLOG will eat most SSDs and spit them out; SLOG is 100% writes, as it is a backup of the transaction groups stored in the write cache - RAM. the only time it would ever be read is a catastrophic occurrence that interrupts a write AND must be rebuilt for the integrity of the data or pool, but this is nearly impossible to do with normal file sharing, as the CoW nature of ZFS ensures that writes either completely succeed or completely fail. this kind of write failure is most common with things like VM's, which write constantly to disk and zfs cannot tell what is and isnt important and what was and wasn't lost.

also note that a a failed SLOG can put the pool at risk.

L2ARC will use RAM to store it's indexes. RAM that could be caching data instead.

misconstruing either SLOG or L2ARC can make your pool slower and more at risk.
both of these, again, should only be added after you run the system for awhile and have the stats to show that either would add value rather than just making things worse or costing money to do nothing.

there is a moderate amount of misinformation around about these, particularly SLOG. I have seen a video where an IX rep themselves incorrectly called SLOT a write cache; it is NOT a write cache.

 

[Etorix]

I have an intel i3-6100 cpu.

Not compatible with a C246 motherboard. You need a Core i3-8100/8300/9100/9300 or a Xeon E-2100/2200 to go with your motherboard, or a C236 motherboard to with your CPU.

 

[vincentj]

SLOG is usually even less useful than L2ARC and even more misunderstood. a SLOG only needs to be about 6-10GB (unless you have some ludicrous amount of RAM maybe). SLOG will ONLY apply to SYNC writes, while SMB, one of the most common file sharing protocols, is incapable of sync writes on its own.
SLOG will eat most SSDs and spit them out; SLOG is 100% writes, as it is a backup of the transaction groups stored in the write cache - RAM. the only time it would ever be read is a catastrophic occurrence that interrupts a write AND must be rebuilt for the integrity of the data or pool, but this is nearly impossible to do with normal file sharing, as the CoW nature of ZFS ensures that writes either completely succeed or completely fail. this kind of write failure is most common with things like VM's, which write constantly to disk and zfs cannot tell what is and isnt important and what was and wasn't lost.

also note that a a failed SLOG can put the pool at risk.

L2ARC will use RAM to store it's indexes. RAM that could be caching data instead.

misconstruing either SLOG or L2ARC can make your pool slower and more at risk.
both of these, again, should only be added after you run the system for awhile and have the stats to show that either would add value rather than just making things worse or costing money to do nothing.

there is a moderate amount of misinformation around about these, particularly SLOG. I have seen a video where an IX rep themselves incorrectly called SLOT a write cache; it is NOT a write cache.

So could all this be useless? I saw that IX Systems sells most of the nas with used nvme memory like l2arc or/and slog. If this could cause problems or maybe no benefit at all, why are they being put?

PS: I thought of inserting a SLOG because I wanted to speed up writing on samba (I have 3 users who use it) since I have the pool in raidz2 and therefore there is no increase in writing performance.

Not compatible with a C246 motherboard. You need a Core i3-8100/8300/9100/9300 or a Xeon E-2100/2200 to go with your motherboard, or a C236 motherboard to with your CPU.

Yes, I saw. I have to change the cpu. Advice on this too?

 

[artlessknave]

So could all this be useless? I saw that IX Systems sells most of the nas with used nvme memory like l2arc or/and slog. If this could cause problems or maybe no benefit at all, why are they being put?

because they know how to use it and how to set it up. the fact that you are even asking about them the way you are tells me you do not. most people do not, because they think SLOG = write cache.

PS: I thought of inserting a SLOG because I wanted to speed up writing on samba (I have 3 users who use it) since I have the pool in raidz2 and therefore there is no increase in writing performance.

SLOG will NOT "speed up write on samba". SLOG is not a write cache. ZFS already caches writes - to RAM, in what are called transactions groups, which are flushed to disk around every 5 seconds or so. this is why more RAM is best, it increase both the READ and WRITE cache sizes available for ZFS to accelerate your storage.

1. SLOG only applies to sync writes.
2. samba does not do sync writes. you can force it by setting sync at the dataset level but...the next point applies.
3. most of the time SLOG makes your writes slower, because ZFS caches the write to RAM (the write cache) THEN has to clone that write to SLOG, then reports that the write was sucessful, THEN writes it into the pool. as disk is nearly always slower than RAM, this means sync writes are slower - the difference is these writes become *safer*; if the RAM copy is lost, it can be repeated from SLOG. this is most useful for something like a VM datastore, but at this point those should just go on SSD anyway, and not really need a SLOG either.

 

[Etorix]

So could all this be useless? I saw that IX Systems sells most of the nas with used nvme memory like l2arc or/and slog.

That should be "with L2ARC and/or SLOG as option", not as default.
SLOG is only useful for sync writes, and even then sync writes with the best SLOG money can buy are still WAAAY slower than async writes.
File serving through SMB is unlikely to involve sync writes.

As you have 64 GB RAM, and your proposed L2ARC is a reasonably sized 250 GB, you should be able to add a L2ARC with no harmful effects (contrary to those who build with 16 GB and a 2 TB "cache"…). But it's not obvious whether it would have beneficial effects. L2ARC would not be necessary if the 64 GB RAM already make for a sufficient ARC. And L2ARC would not be useful if the workload is totally random and any given file is unlikely to be requested twice in succession.
The cautious approach would be to build without a L2ARC, put the NAS "in production" for some time and then look at the output of arc_summary: This will tell whether a L2ARC would be of actual use. (And, yes, you can always add a L2ARC later on.)

Yes, I saw. I have to change the cpu. Advice on this too?

For use with ECC, any Core i3-8000/9000 or Xeon E-2100/2200 you may find for a reasonable price. SMB serving to a handful of clients does not require many cores, so the low end of the range will do.

 

[vincentj]

Thanks. And how can I increase the write speed (or at least do a balancing)? i'm in raidz2, so i only get performance regarding reading, not writing.

 

[ChrisRJ]

@vincentj please read the resources on the basic aspects of ZFS, those will give you the background to a good design. You can find them in the "Recommend readings" in my footer.

It should also be noted that you have provided very little information about your use-case:

• How many users?
• How many in parallel?
• What programs will be used?
• How does your network look like?
• What client machines do you have?
• and much more

TrueNAS is an enterprise system. That means it comes with a lot of power but also complexity. If you have seen YouTube videos about how great and seemingly easy to set up it is, forget those. 95% of those videos are entertainment only and often contain misleading or even wrong information. A notable exception would be content from Tom Lawrence.

 

[vincentj]

I mainly use samba, but not only. I have 3 users plus cameras that save data. My network is gigabit, but for the server I have two cards in bonding, always gigabit (this will change when I change the network, moving to a 10Gigabit one and I want to be already prepared for this change, thus avoiding, perhaps, modifying the pool!

 

[Etorix]

I mainly use samba, but not only.

Those who can most help you want to know exactly what's hidden behind this "not only".

My network is gigabit, but for the server I have two cards in bonding, always gigabit

This possibly does not work the way you think it should.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

moving to a 10Gigabit one and I want to be already prepared for this change, thus avoiding, perhaps, modifying the pool!

To saturate a 10 Gb/s link you'll need multiple vdevs.
A single 7-wide raidz2 has the IOPS of a single drive (oops!) and, at best, the throughput of 5 drives.

 

[artlessknave]

As you have 64 GB RAM, and your proposed L2ARC is a reasonably sized 250 GB, you should be able to add a L2ARC with no harmful effects (contrary to those who build with 16 GB and a 2 TB "cache"…).

I see your 2TB cache....and raise you a 12TB L2ARC, with 64GB of RAM!

Reddit - Dive into anything

www.reddit.com

 

[ChrisRJ]

I mainly use samba, but not only. I have 3 users plus cameras that save data. My network is gigabit, but for the server I have two cards in bonding, always gigabit (this will change when I change the network, moving to a 10Gigabit one and I want to be already prepared for this change, thus avoiding, perhaps, modifying the pool!

As @Etorix already eluded to, this is by far not enough information. Relax, you are not alone in this ;-).

Let me give you an example about the level of detail needed. To convert things to the analog world, your question is the equivalent of "what tires should I get for my vehicle". I asked back what you want to do with the vehicle and you respond "I primarily ride a bicycle". There are two challenges for me with this answer.

First, you tell me that you also use other vehicles that also need tires but neither mention which kinds nor how often or for what exact purpose. What if you live in the middle of nowhere and need a truck ready 24x7, although you rarely use it? If I don't know about the truck, I will not mention that it needs other tires than a bike and you would be left stranded sooner or later.

Second, you don't tell me what you want to do with your bicycle. Downhill racing would require a mountain bike. Or do you want a racer? Or an urban bike to do some shopping? Or drive around your kids? SMB is the bike here, and it is a huge difference, whether you want to save an office document, stream a video, or do 8k video editing.

Lastly, I want to come back to the already mentioned complexity. If you wish to have a great experience and good value for money with your TrueNAS, you must invest a lot more time than what was needed to write a response of less than 3 lines. As the saying goes: garbage in, garbage out. If you do not spend the time needed, you will have a pretty bad experience.

So again, please read the resources I mentioned above!

As a reference point for the time you should invest: When I built my current NAS 3 years ago, it took me about 3 months to come to a solution. And that was on top of more than 10 years with TrueNAS, 15 years with ZFS, and 28 years of building servers.

If you do not want to spend that amount of time, that is ok. But be prepared for less than ideal results.