Help with Dell R720XD setup and opinions

[Kirpan]

Hi Guys,


New to the forum so here goes. I had a Baffalo terastion which died recently (Luckily I had the data stored elsewhere) and I started to look for a better NAS option. I thought surely there must be something out there better than the Baffalo which has very limited help if you start getting Kernal issue (Booting firmware). So I started to look at synology, but thought for the spec and the price I am sure I can do better. Then I researched some more and stabled on Freenas. Finally I light OS Nas system which seems to have alot of redundancies should anything go wrong. As I am familiar with building PC's I thought I am sure I can do a better job. So I started to look and was looking at some atx server boards which buy the time I added the cost up was around the £1800 mark.


So I finally came across a Dell R720XD with the below Specs:


2x Intel Xeon E5-2670 V1

1100w twin redundancy power supplies

64gb Ram

Dell H310 Raid controller which has pass thorough (IT Mode) so Freenas can control all Hard drives

8x 6TB Sas drives 7.2K Total 48TB

2x 10GB ports and 2x 1GB port



2x USB for now



I am upgrading the rear 2bays so I can add 2x ssd drives,



I think if I have the ssd drives on the Front and use the dell H310 Raid controller I’m not sure if freenas will work.



I have purchased a Dlink DGS-1510-28X 28 port 1GB with 4x 10GB ports switch,



My Plan



I have a Dell machine I9 9900 which has a Nvidia Quadro 2200 graphics card. This will be my plex server and I will add a 10gb pci card which has 2 ports. 1 port will go to the switch and the other directly to the Dell R720XD so I think it will be faster as it’s a direct link.



Same on the R720XD 1x 10GB port to the switch and other to the Dell plex server.



My Questions



Whats the best way to set the pools up, looking for 1 large volume which again not sure if that’s best and what Size will I get.



How many disks can fail before I lose any data in

Raid2Z

Raid3Z



How will the speed be in

Raid2Z

Raid3Z



What is the point of having 2x SSD, will freenas manage the redundancy for those 2 drives if one fails?



How easy is it if my system dies to move the drives to other machine and restore the data.



Should I install Freenas 11.3 or Truenas 12 however not sure how stable Truenas 12.0 is. Or is there a Trunas 11.3 I can download?



Should I buy a LSI card twin port which should I need to recover the drives in a emergency or should I buy another Dell 720XD shell for around £250



Please give me your thoughts on the system and let me know if I’m thinking to much into this.



Thanks in advanced!

 

[Heracles]

Hi,

My Questions

Whats the best way to set the pools up, looking for 1 large volume which again not sure if that’s best and what Size will I get.

"Best" can be answered only once the need is completely defined... Will you host VMs in that pool ? How many users will work in the NAS at the same moment ? ... ... ...

Typical answers for 8 drives are either a single Raid-Z2 vDev (6 drives of usable space ; 36 TB total) or a set of 4 mirrors (4 drives of usable space ; 24 TB total).

How many disks can fail before I lose any data in

Raid2Z

Raid3Z

First, they are Raid-Z2 and Raid-Z3, not Raid-2Z nor Raid-3Z

A Raid-Z2 vDev will survive the loss of 2 drives while Raid-Z3 will survive the loss of 3 drives. In these cases, they drop to 0 redundancy. Should anything goes wrong before re-silvering is over, you can loose you data. It is recommended to replace and re-silver your pool before it drops to no redundancy. That is also why Raid-Z1 is not recommended anymore. By definition, you will rarely be able to re-silver it until it drops to 0 redundancy.

What is the point of having 2x SSD, will freenas manage the redundancy for those 2 drives if one fails?

Yes, if you put them in a mirror, FreeNAS will manage the situation. It will run from the last one that survived and will re-silver the replacement to restore the mirror once you replaced the failed one.

For boot pool, it is not recommended anymore to do it with mirrors. It is considered as overkill. Boot mirrors were recommended for USB sticks.

How easy is it if my system dies to move the drives to other machine and restore the data.

It is as easy as re-installing FreeNAS and importing your config file. You will need to backup your FreeNAS config for that of course. Should you end up without backups, the data will be safe but you will have to re-do all your config like your shares.

Note that this is true if the pool is NOT encrypted. Should you wish to threaten yourself with such a self-inflicted ransomware, know that the risk to loose your data is much MUCH higher.

Should I install Freenas 11.3 or Truenas 12 however not sure how stable Truenas 12.0 is. Or is there a Trunas 11.3 I can download?

Here, I stay with 11.2. I rely on my FreeNAS as strong, professional, reliable and safe storage. Everything I have is in my FreeNAS servers. For that, I am not in any hurry to upgrade to the latest version. I intend not to stay behind too long, but I will not be an early adopter for sure.

Should I buy a LSI card twin port which should I need to recover the drives in a emergency or should I buy another Dell 720XD shell for around £250

The moment FreeNAS sees the drives directly in your first setup, any setup where FreeNAS will see the drives directly will be able to import the pool. An example of what not to do is someone in the forum who created a series of logical drives in his raid controller, each with a single physical drive. These logical drives were then presented to FreeNAS. FreeNAS must handle the physical drives itself.

I do not see anything about backups in your plan... No matter how strong a single FreeNAS server can be, it is still a single server and a single point of failure. See the 3 copies rule in my signature about a complete backup strategy.

 

[Kirpan]

Hi,



"Best" can be answered only once the need is completely defined... Will you host VMs in that pool ? How many users will work in the NAS at the same moment ? ... ... ...

Typical answers for 8 drives are either a single Raid-Z2 vDev (6 drives of usable space ; 36 TB total) or a set of 4 mirrors (4 drives of usable space ; 24 TB total).



First, they are Raid-Z2 and Raid-Z3, not Raid-2Z nor Raid-3Z :)

A Raid-Z2 vDev will survive the loss of 2 drives while Raid-Z3 will survive the loss of 3 drives. In these cases, they drop to 0 redundancy. Should anything goes wrong before re-silvering is over, you can loose you data. It is recommended to replace and re-silver your pool before it drops to no redundancy. That is also why Raid-Z1 is not recommended anymore. By definition, you will rarely be able to re-silver it until it drops to 0 redundancy.



Yes, if you put them in a mirror, FreeNAS will manage the situation. It will run from the last one that survived and will re-silver the replacement to restore the mirror once you replaced the failed one.

For boot pool, it is not recommended anymore to do it with mirrors. It is considered as overkill. Boot mirrors were recommended for USB sticks.



It is as easy as re-installing FreeNAS and importing your config file. You will need to backup your FreeNAS config for that of course. Should you end up without backups, the data will be safe but you will have to re-do all your config like your shares.

Note that this is true if the pool is NOT encrypted. Should you wish to threaten yourself with such a self-inflicted ransomware, know that the risk to loose your data is much MUCH higher.



Here, I stay with 11.2. I rely on my FreeNAS as strong, professional, reliable and safe storage. Everything I have is in my FreeNAS servers. For that, I am not in any hurry to upgrade to the latest version. I intend not to stay behind too long, but I will not be an early adopter for sure.



The moment FreeNAS sees the drives directly in your first setup, any setup where FreeNAS will see the drives directly will be able to import the pool. An example of what not to do is someone in the forum who created a series of logical drives in his raid controller, each with a single physical drive. These logical drives were then presented to FreeNAS. FreeNAS must handle the physical drives itself.

I do not see anything about backups in your plan... No matter how strong a single FreeNAS server can be, it is still a single server and a single point of failure. See the 3 copies rule in my signature about a complete backup strategy.

You absolutley right. I was going to set up freenas then I was going to add another standing drive where I can back up my data to a mirror storage. I needed to know how much data I was going to get before I could work out the amount of data.

A Raid-Z2 vDev will survive the loss of 2 drives while Raid-Z3 will survive the loss of 3 drives. In these cases, they drop to 0 redundancy. Should anything goes wrong before re-silvering is over, you can loose you data. It is recommended to replace and re-silver your pool before it drops to no redundancy. That is also why Raid-Z1 is not recommended anymore. By definition, you will rarely be able to re-silver it until it drops to 0 redundancy.

So if 2 drives fail at the same time will I lose all my data, as it sound like if you lose a drive you need to wait until the other drives get in sync before you can lose another drive. Am I right in syaing that.

 

[Heracles]

So if 2 drives fail at the same time will I lose all my data

Not that you WILL loose your data. It is that you MAY loose your data. Raid-Z2 will survice the loss of 2 drives. If it happens, you replace the drives, re-silver the 2 new drives and everything go well, you will not loose anything.

The risk is if something goes wrong during the resilvering process.

If a Raid-Z2 loses 1 drives, or a Raid-Z3 loses 2 drives, they still have one for redundancy. If something goes wrong during the process, they will go to that last redundant drive and fix it, ensuring the resilvering process ends up perfect. It is that capability that you loose when your vDev drops to 0 redundancy.

It is not that you --automatically-- loose it all when you drop to 0 redundancy. It is that you --risk-- to loose it all.

 

[NickF]

I am of the mindset that backups are important than multiple disks of redundancy for small pools. The number of people have have 4 or 5 drive RAID Z-2 pools is silly too me. I have a RaidZ1 on my production FreeNAS of 3x 10TB drives for my Plex stuff. If you are storing movies and you are using RAID Z-2 or RAID Z-3 with less than 8 drives, you are silly. That being said, you are looking at a server with 6TB SAS drives. What is the cost per tb on those drives? Why SAS for this purpose? If they are a good price, power to you! But when you can buy 10TB drives or even 14TB drives on sale for $159-$199 each...why?
Alot of SAS Drives I have in my fleet, specifically a whole lot of 4TB and 600GB drives from a retired HP SAN, don't actually support SMART monitoring. That is not uncommon with SAS. So despite better error handling with SAS, I would prefer to get an alert on a SATA drive that has a bad sector that FreeNAS knows about before the drive totally fails, than just have a SAS drive drop out all together.

Enterprise SAS drives have their place, just as RAID Z2 and RAID Z3 have their place. I use them all the time, but not for Plex. Just backup your data and have a plan in place to pivot your storage subsystem so that you can still watch movie son Plex in case of an epic failure.

I also think that you can MORE than get away with a single CPU and 32GB configuration if the reseller you are looking at can accommodate that for you.

Either way I also have 3 copies of all of my movies/tv shows for plex. I used ZFS send/receive to send it to an HP MicroServer in my house that also has 3x10TB Drives. That MicroServer then does send/recieve to an offsite location at a friends house to a FreeNAS box that is virtualized on ESXI and has an LSI HBA passed through to it that connects to a 12 bay disk shelf that is filled with 4TB drives.

You could do the same thing with an old HP Z220 workstation that you can buy for ~$100 on ebay and some shucked hard drives. You don't need 10-gigabit for the backup appliance either.
RAID is not a backup.



My two cents. Spend your time and money wisely.

 

[pschatz100]

I am of the mindset that backups are important than multiple disks of redundancy for small pools. The number of people have have 4 or 5 drive RAID Z-2 pools is silly too me. I have a RaidZ1 on my production FreeNAS of 3x 10TB drives for my Plex stuff. If you are storing movies and you are using RAID Z-2 or RAID Z-3 with less than 8 drives, you are silly.

I would not agree with this statement. To me, RaidZ2 is the minimum at which I am comfortable.

For all the time and effort I put into preparing and uploading data, whether it be irreplaceable personal data or movies for Plex, the comfort factor of having an extra disk of redundancy is worth the cost. If you ever had a disk in a RaidZ1 pool fail, then you know the anxiety you get when there is no redundancy until the bad disk is replaced and resilvering completed. I had a resilver fail once, and the feeling was terrible. I was able to obtain another new disk, burn it in and then complete resilvering - so at the end of the day, I did not lose any data but I was very nervous for the week or so it took me to get everything sorted out. Since then, I have configured my pools as RaidZ2.

Of course, what you say about backups is also true. RaidZ is not a substitute for a good backup strategy. But there are risks associated with restoring backups as well. For all the money and time invested in a system, an additional disk of redundancy is a small investment that goes a long way towards making me feel better about my setup.

 

[NasKar]

@Heracles would you mind providing the details of how you setup your ZFS replication over site-to-site VPN?

 

[NickF]

I would not agree with this statement. To me, RaidZ2 is the minimum at which I am comfortable.

For all the time and effort I put into preparing and uploading data, whether it be irreplaceable personal data or movies for Plex, the comfort factor of having an extra disk of redundancy is worth the cost. If you ever had a disk in a RaidZ1 pool fail, then you know the anxiety you get when there is no redundancy until the bad disk is replaced and resilvering completed. I had a resilver fail once, and the feeling was terrible.

But that is the thing. I have, and I am advocate having, a full second local copy of this class of data. For exactly the reason you cite. I have ZERO interest in trying to recreate my collections or trying to restore photos and home movies that I have digitized. But the fact remains spending an extra $200 on an extra disk literally buys a second barebones appliance to use as a backup target.

The fact remains that if a disk drops out on you on your main system, there are a host of reasons why that could happen. It could be normal wear and tare on the spindles, and that is the most likely cause. But your SAS card could be intermittently failing, or your RAM could be screwy, or your CPU or motherboard or power supply could be causing isssues. Trying to resilver on an unstable system could be catastrophic. So having that second system already in production where you can KNOW the data is safe is significantly more important.

Of course, what you say about backups is also true. RaidZ is not a substitute for a good backup strategy. But there are risks associated with restoring backups as well. For all the money and time invested in a system, an additional disk of redundancy is a small investment that goes a long way towards making me feel better about my setup.

I am unaware of what "risks" are at play? Lets say during resilvering, my production box utterly fails. I build an entire new pool. I shift Plex to point to a share on the backup box so I can watch my stuff and I am back up and running. Then, If I have ZFS on both ends and I am under normal circumstances just shipping a snap from production to backup, all I am doing is just flipping that around, and shipping the snaps from backup to the "new production".
I see no risks here.

But let me ask you the question. I had a SuperMicro box in a closet at work running in an environment far too hot for it to be in. It caught on fire. I didnt have backups of the data on that server. No amount of redundancy in that box would have saved it. If money matters, why would good backups not be more important than additional redundancy in each node?

All that being said-- I am not saying Z2 and Z3 are not good investments in some cases. But only when alot of spindles are in play or with more sensitive data than DVD rips. And if money isn't a factor, or you already have a good backup strategy in place--THEN you can work on making each node more robust. All I am getting at here is this type of stuff is about order of operation. Get a system in prod, get your backups in prod, then you can do whatever you want to beef it up.

Like right now, I have a single point of failure going to my ESXI host. Both it, and my FreeNAS box have been running since November with a single uplink going to a single switch. Just this past week I started buying all the parts necessary to build a failover stack of my switch. Then I am going to buy dual (or even maybe quad) 10G network cards to build out redundancy for my systems that span across the failover cluster. Rome wasn't built in a day, and I am not myself a multi-million dollar company. I am just some dude who is hording data in his basement. Making small, iterative changes is the only way I can afford to do what I want to do.

 

[Heracles]

how you setup your ZFS replication over site-to-site VPN?

Hi NasKar,

On my main site, I am using pfSense as my Internet firewall. That one is the VPN server to which end points connect to.

On my satellite sites, including the one at my father's place where my DR FreeNAS is, I use D-Link DIR-880 routers that I flashed with DD-WRT. An OpenVPN client is one of the feature offered by DD-WRT. So these are configured to connect back to my pfSense and advertise their subnets.

I could have stop it here and call resources on each sites by IP address but I never liked hard coding, either IP hard coding or any other form. For that, I also have a DNS server for my internal resources. Every system is registered in the DNS with their internal names (domain .lan) mapping to their internal IP address.

That way, my main FreeNAS server just needs to call my DR FreeNAS using its local domain name and will reach it. Of course, I opened the required ports in the firewalls at both ends for the socket to go through.

To configure the replication itself, I used the semi-automatic option offered by FreeNAS. I created a replication token on the DR server and configured the replication task on the main server using that token.

I did first replication while both servers were on my site. That allowed me to do that much bigger sync over LAN. I then moved the DR server to my father's place, modified the DNS server to point the new internal IP address for the server and replication came back. It only needed to catch a few snapshots instead of re-syncing the entire pool.

The VPN being completely external to FreeNAS, it is also completely transparent. From FreeNAS' point of view, it is doing basic and standard replication over LAN.

 

[NasKar]

Thanks Heracles,
I also have a pfsense OpenVPN server on my main system that I use to remotely dial into my server at home. Can I setup openVPN on the backup Freenas (not in a jail) and have it replicate over the VPN to the remote site? Similar to how I have OpenVPN working in a jail. I'm trying to minimize the footprint of equipment at the remote location.

I have an old ASUS router that I flashed to DD-WRT yrs ago before I started using pfSense, so maybe that's an option.

Do you have a UPS at your DR FreeNAS location or just your Main Server?

 

[Heracles]

It caught on fire. I didnt have backups of the data on that server. No amount of redundancy in that box would have saved it. If money matters, why would good backups not be more important than additional redundancy in each node?

That is why a complete backup strategy requires the 3 copies rules explained in my signature...

Once your infrastructure is complete and provides these 3 copies, one extra drive to upgrade Raid-Z1 to Raid-Z2 is not much.

Also, out of the 3 copies, it is possible for the same one to be both offline and offsite. Tape backups are such a copy. In this case, a physical incident will force you to your offline copy, so one that is not as up-to-date as the online copy was. If you go with an online but offsite and onsite but offline, you do not have this problem. A physical incident will have minimal impact because the offsite copy was sync online, so very accurate. A logical incident contained in the first server will also be of minimal incident. Only a logical incident that will propagate from the main server to the DR server will force you to the not-so-up-to-date offline copy. In such a terrible scenario, the lowest of all probabilities, that copy will still be more then welcomed.

 

[Heracles]

I'm trying to minimize the footprint of equipment at the remote location.

It may be possible, but I discourage having FreeNAS in any kind of first line role like that.

I have an old ASUS router that I flashed to DD-WRT yrs ago before I started using pfSense, so maybe that's an option.

Absolutely and I would prefer that one.

You can also look at some small pfSense appliances like the SG-1100. I in the process of replacing my father's DD-WRT with one of these. That will improve my DR capabilities by adding things like HAProxy in the gateway on the DR site.

 

[Heracles]

Do you have a UPS at your DR FreeNAS location or just your Main Server?

Hi again,

I do have some UPS at my father's place. They are a great protection for basically everything. He even had a very clear demo recently. About at the end of March, well after people have been asked to isolated themselves because of the virus, they had a power outage. When power came back, a big power surge happened and one of his neighbor had is cable modem toasted by that. He ended up disconnected for a few days. My father did not suffered anything like that thanks to its power protection.

For a little while, I only had a very good and professional power bar from APC instead of a UPS. I consider this as the minimum. But as of now, indeed, I have a UPS on site for that DR server.

 

[NasKar]

Absolutely and I would prefer that one.

How would you test it before bringing it the remote location. Should I put the remote FreeNAS on a different VLAN at my house?
Also I noticed that you have less HDD capacity at each level of backup. Do you choose to only backup parts of the Main Server?

 

[Heracles]

How would you test it before bringing it the remote location. Should I put the remote FreeNAS on a different VLAN at my house?

Hi again,

Say your internal network is 192.168.1.0/24.
I would create a VPN using subnet 192.168.2.0/24 for its tunnel and use 192.168.3.0/24 as the internal LAN for the ASUS.

Then, I would plug the ASUS' WAN interface in your main LAN.
Next, you configure OpenVPN and route 192.168.3.0/24 over it.
You then plug your DR FreeNAS in the LAN part of your ASUS.

Once that is done right, your FreeNAS server in your LAN should reach the server at 192.168.3.x without any change:
--The main server will see that 192.168.3.x is not within its own subnet, so will route it using its default gateway
--pfSense will recognize 192.168.3.x as the subnet reachable with the OpenVPN gateway at 192.168.2.x, so will route it there
--Once in the ASUS, the packet is forwarded to what is its local LAN.
--Reverse path will do the same.

I noticed that you have less HDD capacity at each level of backup. Do you choose to only backup parts of the Main Server?

Indeed, there are things that I do not backup. I have a pretty large iSCSI storage for my ESXi server and its VM. That space is not backed up. For a VM like my pfSense, I backup only its small config file inside the replicated dataset. The VM itself is not backed up in any other way. For my database server, I do an SQLDump of its content but again, do not backup the actual VM. I also have a lot of test / lab VM that are not backed up in any way.

I also have a Graylog server for log formatting and archiving, as well as Q-Radar Community Edition for monitoring the security over my entire network (main site and satellites). These two have huge storage that are not backed up either. I will lost my log history should I suffer a catastrophic failure on my main server. I do not have to comply with official regulation like SOX, so I do not mind loosing these logs.

As for Thanatos, that one is indeed a little short and I would like to have more. Still, as of now, it has plenty of space and can hold all I need. I still have a lot of time before my backup needs reach its limits and I will have replace it by then. At that time, I will also be sure I can have it on Raid-Z2. For now, it would already requires a very catastrophic event for me to be forced to that last resort. So for that event happen at the very random moment Raid-Z1 fails on me is of low enough probability for me. When I will be able to do better, I will. But for now, I consider the setup as enough.

 

[danb35]

Should I install Freenas 11.3 or Truenas 12 however not sure how stable Truenas 12.0 is. Or is there a Trunas 11.3 I can download?

TrueNAS 12 is NOT stable--it's pre-Alpha code at this point. The current stable release is 11.3, and I think most of us are finding that it is pretty stable at this point. TrueNAS 11.3 is only available as a paid product from iX.

 

[NickF]

That is why a complete backup strategy requires the 3 copies rules explained in my signature...

Once your infrastructure is complete and provides these 3 copies, one extra drive to upgrade Raid-Z1 to Raid-Z2 is not much.

Also, out of the 3 copies, it is possible for the same one to be both offline and offsite. Tape backups are such a copy. In this case, a physical incident will force you to your offline copy, so one that is not as up-to-date as the online copy was. If you go with an online but offsite and onsite but offline, you do not have this problem. A physical incident will have minimal impact because the offsite copy was sync online, so very accurate. A logical incident contained in the first server will also be of minimal incident. Only a logical incident that will propagate from the main server to the DR server will force you to the not-so-up-to-date offline copy. In such a terrible scenario, the lowest of all probabilities, that copy will still be more then welcomed.

I agree with you, and that was what I was trying to illustrate to pschatz and for the benefit of OP.
I see a trend on these forums where people either underbuild it build with unsupported junk...or utterly overbuilding with high end Enterprise hardware with no disaster recovery plan. Trying to get people to understand that backups matter.

 

[NasKar]

@Heracles with your help I have been able to create an OpenVPN client in the DD-WRT. I have the WAN on the DD-WRT connected to my local LAN and the LAN of the DD-WRT connected to a laptop just to setup. I'm able to access my FreeNAS on my local network (192.168.5.x/24) from the laptop on a different subnet (192.168.1.x/24). I would like to have bidirectional access to the remote system thru the VPN.

1) Would I have to create a additional OpenVPN server on the DD-WRT and client on my pfSense?
2) Should the pfSense VPN server be peer to peer? Can I use shared key on the DD-WRT?
3) I tried to use an old large pfSense with this setup in peer to peer shared key on the Main pfSense but I'm only able to ping from the remote laptop to my main LAN servers and not from my main LAN to the remote laptop. Any thought on what I did incorrectly?

 

[Heracles]

1) Would I have to create a additional OpenVPN server on the DD-WRT and client on my pfSense?

No. Tunnels work both ways, so a single is enough.

2) Should the pfSense VPN server be peer to peer? Can I use shared key on the DD-WRT?

No, the VPN should be Remote Access for a site-to-site.

3) I tried to use an old large pfSense with this setup in peer to peer shared key on the Main pfSense but I'm only able to ping from the remote laptop to my main LAN servers and not from my main LAN to the remote laptop. Any thought on what I did incorrectly?

Your setup is probably working this way because the DD-WRT is directly connected to both segments, so from laptop to FreeNAS, you are not going through the VPN. My bad here, I should have recommend you to create a separate network inside your pfSense so DD-WRT and FreeNAS are not in the same network. That way, it will work only thanks to the VPN or not working at all.

 

[Kirpan]

Thanks for the info guys, I have gone with Raid2Z and will get another storage as a backup with 3 or 4x 10gb drives which should be surfice. Currently the system is flyig but I have a issue with my 10gb card working only at 3gb.

The 6TB SAS drives were the same price as the sata so I went woth SAS.

not sure how to confirm weather SMART feature is working or not in Freenas. Anyone know how to confirm this?