Backing up Windows Servers to FreeNas with an air gap to circumvent ransomware attacks.

[Dixeritis]

I am a Sys Admin for a school district, this is my first job in the IT field so I am definitely more of a Jr admin. Two weeks before school started we were hit with ransomware, this is only after me being in the field for just over a year. As you can imagine I had my work cut out for me and 8 months later my plate isnt any lighter. Our saving graces are that the Director got our email moved to O365 and we had a DC left in Azure before the attack. That was it. We lost EVERYTHING. We were "operational" in two weeks (two 80hour weeks, pay stops at 35 hours fyi) we just had no bells or whistles.

Here we are 8 months later, in a department with no budget so I am looking for a free backup solution. I have VERY minute experience with FreeNas, but had some hands on with UnRaid but after some research I learned that FreeNas is a significantly better storage solution. Here is what it is running on (spare server we have):
2x Xeon E5-2680v2
384gb ECC Ram
1x280gb SSD Boot
12x4 TB Hdds
1 Pool with 2x6 VDevs

The title describes what I am trying to do, an automated backup solution with an air gap so ransomware can not affect it. Like I said I have no idea on how to accomplish this so any and all help is welcome.

 

[netprince]

What kind of data are you trying to backup on the windows servers? File servers or application services running on a VM?

I backup file servers by scheduling weekly rsync/robocopy to a freenas SMB share, then in freenas I schedule regular snapshots of the backed up data in case I need to recover from ransomware. I make sure to have snapshots for at least 6 weeks (sometimes 12 weeks) to make sure I have at least some clean snapshots if ransomware is reported.

For servers running inside VMs (for me its hyperV), I export all the VMs weekly using powershell and then copy them over to a freenas SMB share. Once they are on a freenas box you can use snapshots again to keep older versions of the exports, at least 6 weeks, more if you have the storage.

The important part is that the freenas snapshots are read-only, so the ransomware cannot encrypt them directly. Ransomware will encrypt your servers, and you will backup your encrypted servers as scheduled, so you must use the snapshots to recover data.

Also you want to make sure you keep snapshots around long enough to recover if nobody reports the ransomware for a few weeks.

Hope that helps.

 

[danb35]

The title describes what I am trying to do, an automated backup solution with an air gap so ransomware can not affect it.

Automated with an air gap is very difficult to accomplish. However, a regular snapshot schedule makes ransomware recovery trivial.

 

[Dixeritis]

That sounds simple enough, we have no local fileservers so we our pretty light on on data. I specifically would like to capture the VMs from Hyper-V, yes, the servers themselves do not run anything other than hyper-v. Would you be able to link the powershell script for the automatic exportation?

 

[netprince]

I didn't really use a script so much as just cobbled together some commands from various google searches. The part that exports the VMs is just a one liner, something like on this page...

Export a Running Virtual Machine Using PowerShell

Exporting a Hyper-V VM (virtual machine) creates a full copy of that VM. This can be used as an easy way to create an ad-hoc backup or an archive. An export can also contain all of the existing che…

www.robertborges.us

more details...

Export-VM (Hyper-V)

Use this topic to help manage Windows and Windows Server technologies with Windows PowerShell.

docs.microsoft.com

 

[Dixeritis]

When I try to export directly to the FreeNAS server, I am getting a "General Access Denied Error" with the account having full control rights to the share. Is the best route to export to the server followed by robocopy and delete the servers copy after the transfer is done? Also are you doing this manually for every server, have a timer, or are using psexec? Thanks again.

 

[netprince]

Yeah the best way is to export the VM locally then copy it over to the SMB share, I've never managed to get it working by directly exporting to the SMB share. I wasted a LOT of time trying to make the direct export to SMB share work.

I am using task scheduler to start the powershell script. Its been a while since I set all this up... I have a vague memory of having to add my FreeNAS box to the domain, then give the system accounts of my hyperv servers access to the SMB share. (that may not be required, I cant remember for sure)

 

[seanm]

The important part is that the freenas snapshots are read-only

Well, they are read-only, but they are also deletable. So if a bad guy™ gets root on your FreeNAS he can simply delete snapshots too.

danb35 is right that automated and airgapped together is hard. I guess you could automated a replication from one FreeNAS to another, say scheduled for noon on Mondays, and just phsyically plug/unplug the backup server's network cable every Monday morning/evening.

 

[Newfoundland.Republic]

One approach could be to put your FreeNAS server on a seperate subnet and firewall where the firewall rules would allow the FreeNAS server to access the Windows Servers but not the reverse. You could then pull the data of the Windows Servers into the FreeNAS box.

 

[Dixeritis]

Firewall is out of my Dominion, it is however on its own subnet with some of the other servers. We are a smaller district with only 8 locations and two door to door techs. When I am back at my office I will play around with some basic scripts and the task scheduler as suggested. My freenas server is joined to the domain, pool is encrypted with snapshots taken weekly and retention set at 6 weeks as suggested again until I see just how much data we have. Counting physical machines I have 43 servers. Not sure if it is possible but it would be nice to set freenas to give the account write access on the same schedule, which would slow another attack.

 

[danb35]

So if a bad guy™ gets root on your FreeNAS he can simply delete snapshots too.

True, though that's dealing with a separate attack vector. You could secure that by restricting management (web GUI/SSH) to a separate management LAN.

 

[Newfoundland.Republic]

Can you work with your networking colleague on setting up some VLANs and firewall rules? Get some good separation between your backup and your servers?

 

[Dixeritis]

Our firewall and switch configs are managed, but both or the company's are extraordinarily easy to work with. So if I can prove to my boss that this is a viable option and works well I am more than sure he will allow me to go about making the necessary changes. Although 80% of our network is vlan'd the ransomware got nearly everything. But to further specify with the firewall the ideal situation would be block everything but the middle man/whatever needs to write to it, correct?

 

[Newfoundland.Republic]

Hi @Dixeritis - VLANs by themselves do nothing without firewall rules and/or ACLs (access control lists) between those VLANs. Without those controls there is no barrier to access between the VLANs and subnets if there is routing between them..

Chat with the firewall folks and tell them that you need a way that the FreeNAS box needs to connect to the servers and but not the reverse. If you are using SMB shares, then TCP 445 should work (if I remember correctly). You will also have to give your workstation access to the FreeNAS box (over TCP 443 (HTTPS), preferrably). Don't forget, you need to keep good computer hygene on your workstation, too!

Edit: Some additional thoughts:
1. If your school district has not done so already, you should have a security expert take a good look and see if only the ransomeware got in. Given you are a school district, you likely have a great deal of PIA (Personally Identifiable Information) on your servers. This could be a legal issue.
2. In conjunction with a security expert, you should work with your networking folks to review the current network architecture to reduce the risk of something like this happening again.

 

[Newfoundland.Republic]

1x280gb SSD Boot

You don't need a 280GB SSD boot drive. FreeNAS only needs a fraction of that amount (and you cannot use it for anything else). Take a look at something like a couple of Kingston A400 120GB (still much larger than needed but cost effective) and mirror them for redundancy. You will still need to backup your configuration (including the encryption keys if you are using encryption - if that key is lost you have wonderfully secure data with zero utility, you can't get it back).

 

[Newfoundland.Republic]

12x4 TB Hdds

Take a look at WD Red NAS drives.

 

[Dixeritis]

When I meant, no budget, I mean no budget. Anything that is paid for is done by general funds and it is arm and legs to get that done. I believe the drives in the servers are seagate constellations, I only used the SSD because it was what I had available, I do have the encryption key saved. The ransom was not data specific, but we were targeted, it is called Emotet. We have already had insurance and a forensic team come through and "clean" our systems and more importantly determine if data was stolen (was not). Most of our network has been reimaged and we now have Defender ATP with Malwarebytes (switching to Sophos). I believe I have misspoken and the 280 is some sas drives in raid 1. I will have to verify that before I get too far into configuring this.

 

[blueether]

280 is some sas drives in raid 1

You really don't want hardware raid and freenas - make sure you have a HBA card or the like (LSI with IT mode firmware flashed)

 

[Dixeritis]

This is some older hardware, I couldn't figure out how to bypass the controller so each drive is a singular R0. I am definitely going to dig into more today since everyone has confirmed that is not only viable but actually a good thing to do.

 

[Newfoundland.Republic]

You really don't want hardware raid and freenas - make sure you have a HBA card or the like (LSI with IT mode firmware flashed)

I second @blueether - FreeNAS should not use the hardware RAID; not even in individual RAID-0 configuration. Some have reported issues with pass-through on RAID cards with SMART info, etc. Use an LSI card (you can get them cheap on eBay) flash to IT mode (many tutorials online) and let FreeNAS take care of the RAID using RAID-Z2, etc.

You want to do all you can to protect your backups!