Updating from Nexenta to a new, upgraded FreeNAS build

[danb35]

Last I read, USB 3.0 worked as well as USB 2.0. Neither was recommended for long term attachment of storage devices, though.

 

[Chris Moore]

i've spent an hour looking for FreeNAS documentation that states USB 3.0 isn't reliable, works intermittently or isn't recommended for FreeNAS but i can't find where i read it. the post was from a few years ago, as i recall, i was hoping USB3.0 might be reliable when used in FreeNAS at this point. is it?

What are you trying to do with this?

 

[Jatrabari]

Last I read, USB 3.0 worked as well as USB 2.0. Neither was recommended for long term attachment of storage devices, though.

Can you elaborate? Because I am considering a backup solution for my server as external USB drives...

 

[Chris Moore]

Can you elaborate? Because I am considering a backup solution for my server as external USB drives...

I have tried using USB with FreeNAS. It caused the server to crash and reboot. I don't suggest it. The problem appears to be that the controller will randomly drop the connection and FreeNAS is really funny about drives that disappear when they are being used.

 

[Jatrabari]

I have tried using USB with FreeNAS. It caused the server to crash and reboot. I don't suggest it. The problem appears to be that the controller will randomly drop the connection and FreeNAS is really funny about drives that disappear when they are being used.

So just connecting one USB drive would cause a crash? In my case the external drive would not be a part of any pool but a independent disk to backup the most important files. Would then a cloud backup be the only, most reliable option if USB is not reliable and a backup pool can't be inside the same machine as the main pool?

 

[loch_nas]

A crash would be caused if connection to the USB drive is lost. USB is just unreliable crap.
USB stands for Unreliable Serial Bus.

In my eyes eSATA is a much better choice than USB. It's a shame that eSATA is not very common anymore.

 

[Chris Moore]

So just connecting one USB drive would cause a crash?

No, not just the act of connecting it. The crash comes when you are using it and it suddenly pops out of existence (as far as the NAS knows) because some component of the USB interface just dropped the ball. When a drive is suddenly removed from the system, it can do unexpected things. It was worse historically, but may be more tolerant of it now. YMMV

 

[Ericloewe]

A crash would be caused if connection to the USB drive is lost. USB is just unreliable crap.
USB stands for Unreliable Serial Bus.

In my eyes eSATA is a much better choice than USB. It's a shame that eSATA is not very common any more.

The bus itself is fine, especially from 3.0 onward. The problem is that it won by being cheap, which means everyone is going to race to make the cheapest SATA/USB bridge they possibly can.

Those crap controllers are ultimately the problem.

 

[Chris Moore]

Would then a cloud backup be the only, most reliable option if USB is not reliable and a backup pool can't be inside the same machine as the main pool?

There is nothing saying that a backup drive or pool can't be in the same system as the main storage pool. click on the button in my signature for "Show: Emily-NAS". I have a backup pool. It is a 4 drive RAID-z1 pool using 5TB drives where my main pool was origninally two vdevs of 6 drives, 2TB each, at RAID-z2. I upgraded one of the vdevs to 4TB drives and now my backup pool needs to grow to keep up with the main pool capacity wise, but I have time.
The point is, all my drives are connected by SAS even though they are SATA drives. There is a way to do what you want, but USB is not the best way and it can cause system instability.

 

[Chris Moore]

PSU frying everything in the server, someone kicking the server, drunk/sleepy/inattentive admin destroying everything, fire, etc.

@Jatrabari , Some of those things can only be defended against by having an offsite backup. A local backup in the same server would be a defense against some accidents, and you can easily eject (offline without erase) the backup disks and take them away if you want to move them offsite. You can also setup a cloud based backup, but most of those solutions are going to have a monthly cost involved.

 

[Chris Moore]

I don't have the hardware configurations of all the other users memorized, but I do remember that there are several that insert disks into the regular hot-swap bays, establish a pool, make a backup of the primary data pool, offline the backup pool and then remove those disks to either move them offsite or put them into a fire-resistant box. Don't despair, there are many options.

 

[loch_nas]

The bus itself is fine, especially from 3.0 onward. The problem is that it won by being cheap, which means everyone is going to race to make the cheapest SATA/USB bridge they possibly can.

Those crap controllers are ultimately the problem.

Thnx! You're right, I shouldn't blame the bus itself. Somehow I have the impression that it got worse since USB 3.0, but maybe I just have been unlucky with crappy hardware until now. Only 1 external drive I have works as I would expect, it's in a not so cheap external enclosure (~40 €) I put a former internal disk into. All other USB 3.0 stuff (even card readers) are absolutely unreliable.

 

[Ericloewe]

Well, USB 3.0 introduces a few challenges, which are bound to trip up the El-Cheapo USB/SATA bridge designers in Shenzhen:

• The signalling is much higher frequency
• There are several differential pairs instead of just one half-duplex pair
• You're kinda expected to implement useful additions like USB-Attached SCSI Protocol. I kinda expect the inexperienced engineer at Super China Happy Sun to not do a good job with that.

Additionally, OSes have to support the fancy new stuff. That took a while, and FreeBSD wasn't quite there until 10.x. The good news is that the new xHCI USB driver protocol is meant to last for the foreseeable future, so that USB 3.1 doesn't need a new stack to sit alongside EHCI and the old USB 1.1 somethingHCI.

 

[Jatrabari]

A crash would be caused if connection to the USB drive is lost. USB is just unreliable crap.
USB stands for Unreliable Serial Bus.

In my eyes eSATA is a much better choice than USB. It's a shame that eSATA is not very common any more.

No, not just the act of connecting it. The crash comes when you are using it and it suddenly pops out of existence (as far as the NAS knows) because some component of the USB interface just dropped the ball. When a drive is suddenly removed from the system, it can do unexpected things. It was worse historically, but may be more tolerant of it now. YMMV

So USB as a backup solution is out for sure as I don’t want to worry about any instability or other complications regarding USB. How does this incompatibility relate to USB sticks that are used and have been used as boot devices?

@Jatrabari , Some of those things can only be defended against by having an offsite backup. A local backup in the same server would be a defense against some accidents, and you can easily eject (offline without erase) the backup disks and take them away if you want to move them offsite. You can also setup a cloud based backup, but most of those solutions are going to have a monthly cost involved.

I don't have the hardware configurations of all the other users memorized, but I do remember that there are several that insert disks into the regular hot-swap bays, establish a pool, make a backup of the primary data pool, offline the backup pool and then remove those disks to either move them offsite or put them into a fire-resistant box. Don't despair, there are many options.

Well, as I see it, there are 4 options for backup without a second backup FreeNAS server:

1. External USB drive for main backup (not reliable, not an option)
2. External SATA port for an external drive (don’t have this in the MB or have to get a extra card just for this)
3. Backup pool in the same system as the main pool (can do this but it just feels wrong, would have to have a secondary backup (offsite, hot swap)
4. Encrypted cloud backup such as Backblaze B2 (not that expensive with my total data amount but would like to avoid if possible, not sure how this works in FreeNAS)

IMO, option 3 is the most probable one in my situation, even though there are the risk that Ericloewe mentioned, supplemented with either hot swap or cloud offsite backup. I had abandoned this option before but think I will consider this again…

Are there any other options that have not been mentioned?

Here is also updated configuration. I am still deliberating on the size of the PSU. 550 W is calculated for 6 drives but 650 W could be enough for 9 if I decide to implement the backup pool + offsite hotswap...

MB: Supermicro X10SRL-F
CPU: Intel Xeon E5-2630v3 8x 2.40GHz or Intel Xeon E5-1650 V4 6x 3.50GHz
Heatsink: Noctua NH-U9DX i4
Memory: 64 GB ECC (4x 16GB Samsung M393A2K40BB1 2x 32GB Crucial CT32G4RFD424A DDR4-2400)
HBA: LSISAS9201-16i SAS card (will be flashed into IT firmware)
HDDs: 4 TB WD Red x 6 as Raidz2
Case: 12x 5,25” bays for up to 6x Icydock MB153SP-B 3 in 2 SATA Internal Backplane Modules for easy access and hotswap (plan to add more HDDs in the future for expansion up to 16 drives)
Power: Enermax EMD625AWT Seasonic Prime Ultra 550W or 650W
Boot device: 16/32 GB SSD 2x Supermicro 32 GB SATA DOM mirrored

 

[Chris Moore]

Here is also updated configuration. I am still deliberating on the size of the PSU. 550 W is calculated for 6 drives but 650 W could be enough for 9 if I decide to implement the backup pool + offsite hotswap...

Best to go with the bigger power supply to start because it doesn't use extra power to have a supply with more max capacity and then you don't need to change it later. In fact, it you want to go to 16 drives at some point, you should probably go ahead and get the power supply for that now. Larger supplies are usually more efficient, but do pay the extra for a gold rated supply.

 

[Jatrabari]

Best to go with the bigger power supply to start because it doesn't use extra power to have a supply with more max capacity and then you don't need to change it later. In fact, it you want to go to 16 drives at some point, you should probably go ahead and get the power supply for that now. Larger supplies are usually more efficient, but do pay the extra for a gold rated supply.

Actually I did some calculations and with a 6-disc 4 TB raidz2 pool I will get about 12 TB of usable space and that is plenty with my data consumption and needs for years. And when the present pool is close to 80 % mark, I can just upgrade the pool by swapping the HDDs with bigger ones. So 9 HDDs (6 main pool, 2 backup pool and 1 hotswap) will be enough for me.

About the PSU, the 550 W has been calculated according to jgreco's guide with 1.25x margin and that is right on the money as to 9 HDDs + rest of the components but the idle power is below 10 % which I have been concerned about. But as I have read in this forum and also Ericloewe mentioned that it's more important to scale the PSU by the startup current demands and I don't mind about the extra 100 W of elbow room. I am planning to go for the Titanium PSU as they don't cost much more compared to Gold PSUs.

But does it benefit the PSU as less wear and tear and longer lifespan if it is used at low power levels or can there be some foreseeable problems?

 

[Chris Moore]

So 9 HDDs (6 main pool, 2 backup pool and 1 hotswap) will be enough for me.

Sorry, for some reason I thought you had planned more drives.

the 550 W has been calculated according to jgreco's guide with 1.25x margin and that is right on the money as to 9 HDDs

Sure, not arguing at all with that calculation. I still think it best to have the extra headroom as a precaution because I don't like to stress the equipment.

But does it benefit the PSU as less wear and tear and longer lifespan if it is used at low power levels

I don't claim that it will make the supply last longer because I have no evidence of it, but that is my intuitive thought, if it is less stressed, it should last longer and, more important in my mind, be reliable.

 

[droeders]

Well, as I see it, there are 4 options for backup without a second backup FreeNAS server:

1. External USB drive for main backup (not reliable, not an option)
2. External SATA port for an external drive (don’t have this in the MB or have to get a extra card just for this)
3. Backup pool in the same system as the main pool (can do this but it just feels wrong, would have to have a secondary backup (offsite, hot swap)
4. Encrypted cloud backup such as Backblaze B2 (not that expensive with my total data amount but would like to avoid if possible, not sure how this works in FreeNAS)

IMO, option 3 is the most probable one in my situation, even though there are the risk that Ericloewe mentioned, supplemented with either hot swap or cloud offsite backup. I had abandoned this option before but think I will consider this again…

Are there any other options that have not been mentioned?

Here's another option using some drive caddies from Addonics:

https://forums.freenas.org/index.php?threads/how-to-backup-to-local-disks.51693/#post-428001

Based on the post, @NASbox sounds pretty happy with it.

 

[NASbox]

Here's another option using some drive caddies from Addonics:

https://forums.freenas.org/index.php?threads/how-to-backup-to-local-disks.51693/#post-428001

Based on the post, @NASbox sounds pretty happy with it.

Yup! All the benefits of an external USB drive, with all the benefits of a native disk. Just make sure you have a GOOD hot swap caddy (most are NOT built for this type of use, they are for quick recovery after the occasional disk failure NOT routine planned hot swaps!)

I'm still struggling a bit with scripting, but the data is not high turnover, so at this point I'm not doing it very often. As I get proficient with ZFS snapshot replication, I'll get a much more automated system-or you could just rsync (I used to, but ZFS Send/Receive is MUCH faster and more accurate). Anyway you'd have the same problems with USB drives (plus a bunch more, like a potentially unreliable drive.)

 

[Chris Moore]

Case: 12x 5,25” bays for up to 6x Icydock MB153SP-B 3 in 2 SATA Internal Backplane Modules for easy access and hotswap (plan to add more HDDs in the future for expansion up to 16 drives)

@droeders , the OP was already planning to use Icydock MB153SP-B units to get hot-swap bays. Unless the plan changed even more since the original post.