20 user office - freenas as primary file server

[vinjoza]

Hi. I am happy to be a new part of the Freenas community as of like 2 weeks ago!

BACKGROUND - INFO
To give a little background about me and my needs: I run a small office, under 20 people, and also I am the main IT guy unfortunately. We do electrical design, so our PCs normally have several files open in parallel, MS office and Autocad being the most common. File size can go up to 20-30 MB per file (or more, but that is rarely).
In addition we have some programs which have database files stored on the server (it's not running a database per-say, it's internal program database, only many small files need to be stored and shared between several users).
What we do is mount various network drives to our users and they open files directly via CIFS share. This way we avoid double versions and keep everyone up-to-date with latest files.

Seeking something simple and almost zero-maintenance, we have so far relied on Synlogy DS 412+ which has served us quite well, but recently it has become a bit too slow dealing with so many files in real-time. After research, I decided against a new top-end Synology, but rather took a spare workstation we had and tried turning it into a Freenas file server. We are talking about a HP 290 G2 MT Business PC, i5-8500 3 GHz, 8 GB RAM, gigabit LAN. Upgraded with a 1 TB NVMe for storage, SATA SSD 120GB for filesystem (had one spare), fired it up and we are basically using it for past 2 weeks. Speed is awesome, Freenas seems very low-maintenance, it has many options if we need to, it joined our domain without problems. So All is great!

QUESTION:
I want to ensure backup, redundancy, safety of files. In Synology I had a RAID5 configuration. That one is still runing and I was planing to use it for secondary backup of my main Freenas pool. But I would like some additional redundancy, and am thinking about also setting up Snapshots. I would add some Caviar Red 7200 in RAID1 as a second pool or something, or maybe some SATA SSDs? Idea is to have a backup drive where I would do hourly shadow copy (if possible and if it wouldn't kill off my performance of my NVMe main drive, seems unlikely) and then do snapshots on that volume? We really don't need hourly - it would be a big shame if we lost a day's work but not a complete disaster.

So What I want to accomplish, in order of priority:
- Ensure security/backup of data against hardware failure. We have an identical PC which will be readily available to stick hard drives in if this one fails, and simply boot up again.
- have a nightly backup at minimum (once a day)
- have an hourly backup if possible (to reduce possibly lost work)
- have a daily snapshot in some way and keep daily 1-2 weeks. I would then keep weekly snapshot for 2 months, and monthly for 1 year. Annual forever.
- have a hourly snapshot if possible. That would be kept only like 24/48 hours
- to have snapshots readable via CIFS from my windows machines as Windows Previous Versions (I hear that's possible, it would be most convenient way)
- set up some backup to a separate machine (Synology prefered)
- set up WebDav cloud backup (i have a 1TB cloud service included from my ISP which I now use via Synology, so i can set up a backup to Synology, and then to cloud from there)

What hard drives would you recommend I buy for secondary pool? Do I need Raid0 or is a single, larger hard drive enough? SSD or server-line HDD? How to accomplish all this? Am I an idiot for not setting up RAID1 on the main pool from the start?

I know it's a long post, but I assume people here have similar or parallel set-ups and shared some of my ideas and concerns at one point or another, and I REALLY will appreciate any and all help I can get!

 

[Samuel Tai]

1. Actually, first you need to do some homework on the capabilities and terminology of ZFS and FreeNAS. A good start is the ZFS primer in the FreeNAS Guide. https://www.ixsystems.com/documentation/freenas/11.3-U3.2/zfsprimer.html
2. Second, you will be running a full server, with all the responsibilities and obligations that entails. You need to consider a more robust grade of hardware beyond spare PCs you happen to have lying around with oddball assortments of random disks for daily production. (If you want to tinker to get familiar with operating FreeNAS and the quirks of the various subsystems, e.g., Samba in your case, then by all means, use what you have on hand.) Here, you should consult the Forum Resource on FreeNAS hardware: https://www.ixsystems.com/community/resources/hardware-recommendations-guide.12/
3. You've started thinking about replication and data recovery. What happens if both the FreeNAS and the Synology crash? Do you need to worry about data security in case of legal discovery? Do you need encryption at-rest? Stripes (RAID0) are a bad idea for anything except scratch space on SSDs with data loss prevention. Mirrors (RAID1) are better for data safety. You also need to design your pool for your use case now and in the future, so consider options like spare drives, cache devices, and possibly dedicated log devices.

 

[vinjoza]

Samuel,
thank you kindly for your reply. I did go through the Hardware recommendations guide, but they are very general and not very up-to-date. Clearly, as it is not a document that can be maintained continuously.
Further, I am not unfamiliar with main terms. Thank you for the links, I will study them further. Don't think that I have just set something up "from scrap components lying around". If you look more closely, it's a mATX PC with intel H370 chipset, one m.2/NVMe port and 4 SSD ports. For our needs, where I will need 1 system drive, one main storage drive (if that is best) and one or two backup drives, that is more than enough. I do not have full redundancy but that, for us, is absolutely not necessary. Having a clone to run up is good enough, but with a saved freenas config even a different config would be up and running within 30 minutes for sure. I am more than fine with that, we can afford to lose a day every 5-10 years, if it comes to that.

What we cannot lose is years worth data. So we have the mentioned cloud backup, as well as manual, off-grid periodic backup to an external drive which is kept at a remote location. That part we have covered, and in my 12 years here we developed some procedures that keep us out of danger's reach.

My main question is Freenas, specific suggestions to approach my 1st level redundancy, as well as possibility to combine with snapshots. We actually have full drive daily snapshots now, kept for a week on Synology, plus monthly kept for a year.

About RAID1 i fully agree, stripping makes no sense here, even NVMe itself is a big overkill. But it does have a 2 mil hour MTBF rating, for whatever it is worth, and price difference to a regular SSD is negligable.

Originally I wanted to forego the RAID for main drive. Why? Mirroring raid would decrease performance (granted, it might not be even noticeable in this setup), but more than that, it only adds the single layer of redundancy. My idea was to put a 2nd SSD drive, 2 TB and make it it's own pool. Then make a backup on it, possibly shadow copy if we would do hourly (as it would imply copying while files are being worked on), and do snapshots in some fashion as described. 2-3 TB would give space to snapshots, and Synology would be a safety pin backup in-office, with cloud and off-site, off-grid manual backup would be a final fail-safe.

My main question is, in this setup, what hard drive set up would you recommend, and possibly what backup/snapshot arrangement? SSD or some enterprise in this sense?

 

[Samuel Tai]

OK, this is a good amount of pre-planning. How much daily volume do you see?

 

[rvassar]

Actually RAID will substantially increase performance if you design the pool correctly. Mirror RAID devices can execute two read operations at the same time. Parity RAID will generally read somewhat faster, but write at the speed of a single disk. Once you scale up to multiple vdev's these bits of trivia become force multipliers, as ZFS will round-robin between vdev's to balance the load. This is why there are so many 300Gb & 600Gb 15k rpm SAS drives still floating around. Not a lot of storage by today's standards, but pre-SSD, they were used to build monster IOPS via RAID.

 

[vinjoza]

OK, this is a good amount of pre-planning. How much daily volume do you see?

Could you help me - I am trying to figure out how to get the value of daily volume?

 

[ornias]

Could you help me - I am trying to figure out how to get the value of daily volume?

Writes (gb) and reads (gb) per day would suffice.
As well as peak datarate requirements (peak gb/s) and peak iop requirements.

 

[vinjoza]

Writes (gb) and reads (gb) per day would suffice.
As well as peak datarate requirements (peak gb/s) and peak iop requirements.

Thank you. Could you help me figure out where to get that info from the Freenas?

 

[vinjoza]

So if i go with zpool iostat, i get the following:
capacity operations bandwidth
pool alloc free read write read write
-------------- ----- ----- ----- ----- ----- -----
EIB-FILESERVER 393G 551G 0 10 6.76K 169K

Is bandwith averaged over some time? Is it every second, 169 KB is written?

 

[vinjoza]

OK so I have about 200-500 MB of daily write. Basically, most of this is because currently, our users like to make a fresh daily copy of all files they do - in reality I expect average of <100 MB of new data per day, after we would switch to snapshots.

Plan is to make a nightly backup to a second pool, which would be comprised of a RAIDz1, 2 pieces of 4TB WD Red SSD SA500 series in mirror arrangement. That pool would be a secondary share for non-important (and less-frequently accessed) files, as well as a place where a nightly backup of the main drive (the NVMe 1 TB one) would go. There, on 2x4TB in mirror pool, I plan to keep a daily snapshot of the backup folder which would be ran-over nightly. I plan to keep 5 years worth of daily snapshots there. In my evaluation, that is 2000 snapshots (less than 10k), and approx 200 GB of additional space. Even if it is 1 TB it's fine (if my calculation is off by factor of 5).

On the main drive, the NVMe one, I would have an hourly snapshot kept for 10 workdays over 10 working hours - so 100 snapshots, which is maybe 20 gigs - even 100 gigs for snapshots is more than acceptable.

I am aware that there is a risk of losing daily work, but, as stated, this is acceptable.

Am I doing this right? Reading thru theory is one thing, getting feedback from the real world users is another - i can really use your help guys?

 

[Heracles]

took a spare workstation we had and tried turning it into a Freenas file server. We are talking about a HP 290 G2 MT Business PC, i5-8500 3 GHz, 8 GB RAM, gigabit LAN.

A first No Go for enterprise production. You should use actual server gear for an enterprise class server.

- Ensure security/backup of data against hardware failure. We have an identical PC which will be readily available to stick hard drives in if this one fails, and simply boot up again.

You need to enforce the 3 copies rule for that (see my signature). What you describe involves only 2 copies, both of them being onsite and online. You will not survive a physical incident like fire and some logical incidents could also take down both of your copies at once.

- have a nightly backup at minimum (once a day)
- have an hourly backup if possible (to reduce possibly lost work)

Once the 3 copies rule has been designed, you can have it run at whatever frequency you need.

- have a hourly snapshot if possible. That would be kept only like 24/48 hours

This is very easy and you can do more snapshots if you have the storage for them. Here, I have 5 generations of snapshots :
15 minutes - kept for 3 days
1 hour - kept for 1 week
1 day - kept for 90 days
1 week - kept for 18 months
4 weeks - kept for 5 years

Design your plan in a way you have less than 1,000 snapshots at a time. To search which snapshot you need when you have too many is a pain...

What hard drives would you recommend I buy for secondary pool? Do I need Raid0 or is a single, larger hard drive enough? SSD or server-line HDD? How to accomplish all this? Am I an idiot for not setting up RAID1 on the main pool from the start?

I would go with Seagate IronWolf, to be sure not to end up with some SMR drives. Also, never do anything without redundancy, even more important for an enterprise class solution like the one you are looking for.

Good luck designing your system and better be safe than sorry here.

 

[vinjoza]

A first No Go for enterprise production. You should use actual server gear for an enterprise class server.

Thank you. I was thinking about this.

You need to enforce the 3 copies rule for that (see my signature). What you describe involves only 2 copies, both of them being onsite and online. You will not survive a physical incident like fire and some logical incidents could also take down both of your copies at once.

Once the 3 copies rule has been designed, you can have it run at whatever frequency you need.

Actually, we would have 4 copies. One on 2nd pool, another on Synology, third on the cloud (that's one remote and presumably 400 km away) nightly, and 4th is manual USB drive for manual periodic backup, approx monthly, and kept away from location. Correct me if I'm wrong tho

This is very easy and you can do more snapshots if you have the storage for them. Here, I have 5 generations of snapshots :
15 minutes - kept for 3 days
1 hour - kept for 1 week
1 day - kept for 90 days
1 week - kept for 18 months
4 weeks - kept for 5 years

Design your plan in a way you have less than 1,000 snapshots at a time. To search which snapshot you need when you have too many is a pain...

OK, I get this, sounds like a plan.

I would go with Seagate IronWolf, to be sure not to end up with some SMR drives. Also, never do anything without redundancy, even more important for an enterprise class solution like the one you are looking for.

Good luck designing your system and better be safe than sorry here.

My idea is to do everything on SSDs, as nowdays they seem to have very good expected MTBF and TBW. The price difference just isn't that much and speed is great. HDD I would only use for a backup, like I plan to use the Synology now.

When you say redundancy, you would by all means go for RAIDz1 (or any type of) mirror for primary share pool?

Thank you for a very clear, to-the-point and helpful reply!

 

[Samuel Tai]

When you say redundancy, you would by all means go for RAIDz1 (or any type of) mirror for primary share pool?

See @jgreco's excellent https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/.

Z1 isn't recommended, as it can only tolerate the loss of a single drive in the pool. Basically, @jgreco's rule of thumb is Z2 for datasets that are mostly read-only, and mirrors for datasets that are write-heavy.

 

[Heracles]

Actually, we would have 4 copies.

No problem doing more than the required 3. 3 is a minimum, not a maximum.

One on 2nd pool,

That one should not be counted. It is online, onsite and in the same system. 0 in 3. This copy does not provide much more than what the original first copy offers.

another on Synology,

That one can barely be considered as copy No 2 : Online and Onsite but in a different system...

third on the cloud (that's one remote and presumably 400 km away) nightly,

That one is your first clear protection. Clearly Offsite. Because it is sync over the network, it is Online. So last need is for a copy that will be clearly offline.

manual USB drive for manual periodic backup, approx monthly, and kept away from location.

That one will indeed be offline, as long as the USB storage is not used for anything else or kept plugged in a system.

So Yes, you end up with the 3 copies rule enforced properly.

When you say redundancy, you would by all means go for RAIDz1

No... RaidZ1 should not be considered as offering redundancy. RaidZ2, RaidZ3 and mirrors are the way to achieve redundancy. If you have a 4 drives array with RaidZ1 and need to rebuild it, the process will require to read back 3 drives without any single error. Should a single error happens, it may corrupt the pool because there is no redundancy to detect and fix it.

 

[ornias]

No problem doing more than the required 3. 3 is a minimum, not a maximum.

You forget to mention the fact external storage providers ALSO often include backups.
In those cases 1 onsite copy and one offsite copy that gets backuped by your solution provider might be enough for low to medium risk data.

 

[Heracles]

You forget to mention the fact external storage providers ALSO often include backups.

Should you have access to these offline backups, should they be kept for long enough and you are able to ask the provider to restore it, then I agree that you can count it. Should you only have the 30 days of undelete from Dropbox, that I do not consider as good enough even if Dropbox takes offline backups.