BUILD A little guidance - which SuperMicro/Atom, offsite backups, other questions

[TimNZ]

Hi all,

I've recently been put onto FreeNAS by a good forum I'm part of, the system and this community here looks fantastic! I've read a LOT of FAQs, threads, blogs, recommendations, some wikipedia etc to upskill myself but there's a fair bit to learn. I have a pretty good idea what a vdev and zpool are, I know you can't add drives to a vdev but you can create more vdevs and add to a zpool. I've never used FreeNAS, but I figure it's capable and mature so it'll be fine. I'm fairly technical, some Linux experience, background as a developer, software engineering degree with electronics and experience building computers, Amazon certified solution architect.

Note that I'm in New Zealand so getting hold of good hardware can be difficult. SuperMicro boards aren't sold here at reasonable prices, so I have to import to NZ via a reshipper, which is expensive and can be slow. I may be able to get gear from Australia though. The Fractal 304 case that costs about $50 in the US costs $200 here, for example, but I can get a cheap enough case. It would be cheaper and easier to go with a standard Asus/Gigabyte board and an i3 but I want something durable, reliable and well understood by the community.

Motivation: I'm starting to experience bit rot on my NTFS/Win7 system (md5 checksum failures) so I'm wanting to build a home NAS based on ZFS so my important data can be housed securely. Commercial NAS's don't have the data integrity I want. I'll want 2-4TB of data that's secured well (RAID Z1), along with another 2-4TB that I don't need protected - single drive ZFS or even NTFS if it's supported. Everything is backed up offsite to disk and critical data to Crashplan so if the whole lot fails, burns, etc, I'm not super concerned, so RAID Z2 isn't required for me. I'll probably go 3 x 3TB disks just because I like having free space, probably 1-2 WD Red, one Hitachi, perhaps a Seagate but I've been burned with them before.

Data integrity is my main concern, not performance. Typical clients will be watching one 720p movie at a time (two maybe, one day) and processing 500+ RAW images at a time using Adobe Bridge - though most likely I will do my photo processing off local SSDs. I don't work with video. Because I want reliability I will likely use a SuperMicro motherboard and 4-8GB ECC RAM. I'm unlikely to transcode anything on the server. There's a chance I'll want to virtualise one day, so the server can do other things for me since it's on all the time.

I have some specific questions (sorry, there's a bit of a list):
1. Which SuperMicro board should I get? I've had the A1SRi-2758F Atom based board recommended but there are so many it's hard to choose. I've read disparaging remarks about the A1SRi-2758F. I'm fairly convinced that an Atom CPU will be up to the task, i3 etc not needed.
2. Do these atom CPUs need cooling? If so do they come with a cooler?
3. Is any specific brand/model of RAM recommended? I'll read the hardware compatibility list once I choose a motherboard.
4. I will have offsite backups, but bit rot or copy problems can occur, and silent corruption of copying/files shouldn't be mirrored. Can anyone recommend the best way to back up a FreeNAS system offsite? Ideally in a format that's easily accessible from Windows if I need to recover a single or multiple files - fat32 or ntfs is easiest, and a backup program can help. Right now I use Robocopy but if I corrupt a data file that corruption is mirrored. The backups could run from my windows machine or the server.
5. Does the case matter much? I can't get the Fractal cases for reasonable prices in NZ. Vibration I'll want to control, I'll keep heat down, and I'll have a good modular power supply. I can't find a compact case with good drive storage, I'll likely have to go mid tower.
6. Is it pretty easy to add more SATA ports via an expansion card on those Atom boards?
7. Does having one RAID Z1 pool then 1-2 other existing drives (NTFS, FAT, or ZFS) sound reasonable? I read somewhere only ZFS is supported in later versions, which I haven't researched yet.
8. I won't need the NAS on all the time. Is there an easy way to have it either power up/down at specific times, or will power saving be enough? Having an Atom CPU means lower power consumption so not a huge deal.

I'll no doubt have other questions as I learn. Thanks for any advice or opinions :)

 

[Jailer]

For your stated usage you're going to want more horsepower than what that Avaton board will provide. I'd look at no less than a xeon and a good x10 based supermicro board.

You're also going to need more memory. I wouldn't go less than 16GB. 8GB is the minimum recommended.

You can add individual drives to your system after your initial pool is created but don't add them to your existing pool. Create a new pool for each individual drive(s) you add. If you add a single drive to an existing pool you just created a single point of failure that can wipe out your entire pool if that drive fails.

Make sure work your way through the sticky threads for your hardware and pool setup so you get started off on the right foot.

 

[Ericloewe]

You don't want a C2758, you want a C2750 (the former is for network stuff like routers).

The Atoms are passively cooled, but require reasonable airflow.

If you're going with a Supermicro X10 motherboard, there's a guide for the memory you'll need (Link is in my sig).

With ZFS, assuming ECC RAM, there will be no invisible bit rot. With proper pool design, it'll all be corrected, as well. The ideal scenario for offsite backups is a similar/identical machine that is replicated to.

The cases are always a very personal choice. Just get something from a reputable brand with good cooling for the drives. Or get a case with a ton of 5.25" bays and add hotswap bays. Or get something rackmounted, budget, noise and available space permitting.

The Atoms are limited in PCI-e connectivity, but it's possible to add an HBA (bandwidth may be limited and you lose all other expansion). A Xeon E3 system would allow for a lot more connectivity (HBA + 10GbE NICs or several HBAs). A Xeon E5 has a crapton of PCI-e connectivity, but is also quite a bit more expensive.

Only ZFS is supported, everything else is for importing data only. RAIDZ2 is strongly recommended for data integrity, as RAIDZ1 is prone to fail during rebuilds.

FreeNAS is designed for 24/7 operation, so frequent/automated power on/off events are not easy to manage and are unsupported.

 

[jgreco]

I'll want 2-4TB of data that's secured well (RAID Z1), along with another 2-4TB that I don't need protected - single drive ZFS or even NTFS if it's supported.

RAIDZ1 isn't "secured well." With the size of drives these days, you're taking a gamble that the remaining data on the remaining drives is completely retrievable and valid after a disk fails. Will you have a cold or warm spare on hand? If so, just add it to the pool NOW and do RAIDZ2. If you won't have a cold or warm spare on hand, then you have to remember that any problem that arises between the time you experience the first failure and the time you manage to get that replacement drive AND complete the resilvering operation into the pool is an unrecoverable error.

RAIDZ2 is the minimum sane level of protection. RAIDZ3 is better yet. Some of us do RAIDZ3 plus a warm spare, meaning we can initiate a resilver the moment we notice a failure, PLUS a cold spare on hand.

Everything is backed up offsite to disk and critical data to Crashplan so if the whole lot fails, burns, etc, I'm not super concerned, so RAID Z2 isn't required for me.

Strongly suggest you rethink the RAIDZ2. Nothing against Crashplan but are you sure they'll be around when you need them? Will you get your bits back in a timely fashion?

I'll probably go 3 x 3TB disks just because I like having free space, probably 1-2 WD Red, one Hitachi, perhaps a Seagate but I've been burned with them before.

Data integrity is my main concern, not performance. Typical clients will be watching one 720p movie at a time (two maybe, one day) and processing 500+ RAW images at a time using Adobe Bridge - though most likely I will do my photo processing off local SSDs. I don't work with video. Because I want reliability I will likely use a SuperMicro motherboard and 4-8GB ECC RAM. I'm unlikely to transcode anything on the server.

8GB is the bare minimum fro FreeNAS.

There's a chance I'll want to virtualise one day, so the server can do other things for me since it's on all the time.

Then step up to Xeon now. No one's virtualized FreeNAS on an Avoton.

I have some specific questions (sorry, there's a bit of a list):
1. Which SuperMicro board should I get? I've had the A1SRi-2758F Atom based board recommended but there are so many it's hard to choose. I've read disparaging remarks about the A1SRi-2758F. I'm fairly convinced that an Atom CPU will be up to the task, i3 etc not needed.

Find a nice C2750 board, not the 2758.

2. Do these atom CPUs need cooling? If so do they come with a cooler?

Yes. Heatsink. Make sure air flows across them.

3. Is any specific brand/model of RAM recommended? I'll read the hardware compatibility list once I choose a motherboard.
4. I will have offsite backups, but bit rot or copy problems can occur, and silent corruption of copying/files shouldn't be mirrored. Can anyone recommend the best way to back up a FreeNAS system offsite? Ideally in a format that's easily accessible from Windows if I need to recover a single or multiple files - fat32 or ntfs is easiest, and a backup program can help. Right now I use Robocopy but if I corrupt a data file that corruption is mirrored. The backups could run from my windows machine or the server.
5. Does the case matter much? I can't get the Fractal cases for reasonable prices in NZ. Vibration I'll want to control, I'll keep heat down, and I'll have a good modular power supply. I can't find a compact case with good drive storage, I'll likely have to go mid tower.
6. Is it pretty easy to add more SATA ports via an expansion card on those Atom boards?

No. You typically get zero or one expansion slots. You can stick an HBA in there for more connectivity but then you're slot-full.

7. Does having one RAID Z1 pool then 1-2 other existing drives (NTFS, FAT, or ZFS) sound reasonable? I read somewhere only ZFS is supported in later versions, which I haven't researched yet.

The others are only there for importing data. Think of it as ZFS-only and you'll be on the right track.

8. I won't need the NAS on all the time. Is there an easy way to have it either power up/down at specific times, or will power saving be enough? Having an Atom CPU means lower power consumption so not a huge deal.

I'll no doubt have other questions as I learn. Thanks for any advice or opinions :)

You can make the system shut down at scheduled times but we frown upon this; it's hard on the hard drives and you're more likely to get failures.

 

[TimNZ]

Thanks all, except for zambanini. I think that's the least friendly first reply I've ever had on a new forum and I've been on plenty. I've done a lot of reading a the guidance I've asked for is part of my research.

I know ZFS has no bit rot, that's the sole reason I'm investing :) I do have a budget to consider, and please keep in mind this is for a home system for storing backups and rarely accessed data, some photos and low res video, there will be very little load and I have excellent offsite and online backups. I design backup systems assuming my house will burn down at the least convenient time, so if the whole system fails and all data is lost it's not a big deal to me, I can restore from backups. If this system gets too expensive it won't be done at all and I'll have to live with NTFS and bit rot, so finding a moderate priced solution is important. A standard PC like the one in this article is still an option too - knowing and understanding the risks, with good backups, make it viable.

I was surprised to find the Xeon systems similar priced to the Atom's if I buy from the US - say a SuperMicro MBD-X9SCL-B for $150 (or MBD-X10SLM-F-O for $180) and the Xeon E3-1220 for $200 is about the same price as the SuperMicro Atom CPU/motherboard combination - would that CPU/MB be a good combination? I see X10 boards mentioned, they're only slightly more expensive. The reviews of the iXsystems FreeNAS mini with the Avaton Atom seems excellent, but not for virtualisation or expandability, but let's throw that requirement out. How is power usage between the Xeon and Atom systems? Interested to hear if anyone favours the Atom over Xeon.

RAID Z2 would be nice but in practice is beyond my initial budget given the 5 disk requirement. I understand more limited repair capability, and the chance of drives failing during a rebuild - that's why I have backups. I just want to be able to detect errors and repair simple errors. Honestly single disk ZFS that just tells me "file xyz has failed" would probably be good enough for me to restore from backup, any repair capability is a bonus.

I will aim for at least 8GB of RAM, but reviews I've read have said 4GB is actually ok. Given all we'll be doing is storing backups, browsing photos, and streaming medium res video the system really won't be taxed, and we don't hit the same data over and over so caching would have limited benefit.

Curious why scheduled downtime hurts drives. Hard drives spin down after a given idle period, which I assume FreeNAS does, so rebooting is much the same as idling for a drive.

So in summary my follow up questions would be:
a) Interested in Xeon vs Atom opinions, given no virtualisation required, power usage important, and unlikely to expand beyond say 8 disks max.
b) Assuming I go Xeon is the MBD-X9SCL-B motherboard ok or is the cheapest of the X10 range ok?
c) Given drives spin down when idle how is scheduled system off time a problem?

 

[Jailer]

Another option to save cost is an i3 instead of a Xeon. And I would still go with this over an Avaton system. You'll be working an Avaton system harder for the same end use, especially running plex, so the power savings aren't going to be there IMHO. But that's just my opinion.

I still say you should go with 16GB of RAM given your intended usage. 4GB isn't going to cut it, trust me I've tried. All kinds of odd unexplainable things start happening with too little RAM.

By default freenas does not spin the drives down. The heads park after a certain amount of idle time but the drives stay spun up. It's been posted that frequent spin up and spin down of the drives shortens their life but that's just what I've read. You'll have to speak with the guys in the know on that one, I don't have any experience or knowledge on the matter to make a suggestion either way. For what it's worth though I left mine set to the default setting and leave em spinning.

 

[TimNZ]

Interesting about the drives, I may do some reading. I'll go for 8GB of RAM initially, upgrade to 16GB as required. I won't be running Plex, I use XMBC/Kodi which just accesses files.

i3 sounds like a pretty good option. I guess an i3-4130T (35W, $126) with a SuperMicro X9 1155 motherboard ($143) and Kingston RAM ($100 for 8GB to start with) could be a pretty good combo. I checked the NZ prices, they're at least 50% higher than newegg.

 

[Jailer]

Skip the T part (low wattage). All they do is cripple the speed to keep it in the pre set thermal range and you lose performance as a result. You'd basically be in the same boat as an Avaton. And that one you linked to is a socket 1150 CPU and is not compatible with the x9 series of motherboards. X9=1155, x10=1150.

If you're going to go with an X9 series board and an i3 you'll want this CPU.

If you're going with an X10 series board and an i3 you'll want this CPU.

And if you're REALLY tight on the budget you can look at a pentium processor like this one or this one and save even more.

Keep in mind all those suggestions I just made were the lowest cost option for each. You can of course purchase whatever fits your needs.

 

[TimNZ]

Ok, no T part, gotcha. I'll make sure the motherboard and CPU are the same socket - stupid mistake there, was just quickly looking at prices. Probably wouldn't go down to Pentium, i3 pricing is fine.

What's the minimum disks for RAID Z2? I read five somewhere but wonder if 4 is possible. I guess they basically turn into mirrors, so 4x2TB would give you 4TB usable space?

Shame the Lenovo TS140 isn't available here - $239 for the base config with case, psu, motherboard, cpu, 4GB RAM, that's cheaper than anything else around. Unfortunately it costs around US$1200 here in NZ and shipping turns out to be super expensive because of weight.

 

[Jailer]

Take a look at Cyberjocks ZFS guide, it will tell you everything you need to know about setting up your pool.

 

[Ericloewe]

RAIDZ2 needs four or more disks. You can't go wrong with 4, 6 or 10. Other numbers shouldn't cause trouble in a home setting, but don't go over 10.

 

[TimNZ]

Thanks guys, I'll have a read of that guide and consider options. As I said because I have good backups even just detecting errors may be enough.

 

[Apollo]

Ok, no T part, gotcha. I'll make sure the motherboard and CPU are the same socket - stupid mistake there, was just quickly looking at prices. Probably wouldn't go down to Pentium, i3 pricing is fine.

What's the minimum disks for RAID Z2? I read five somewhere but wonder if 4 is possible. I guess they basically turn into mirrors, so 4x2TB would give you 4TB usable space?

Shame the Lenovo TS140 isn't available here - $239 for the base config with case, psu, motherboard, cpu, 4GB RAM, that's cheaper than anything else around. Unfortunately it costs around US$1200 here in NZ and shipping turns out to be super expensive because of weight.

I have had an issue with Lenovo back a few month ago for what seemed to be an excellent deal. I placed the order, then a few hours later came back to me saying they couldn't honor the order as it was due to pricing mistake.
Way to go Lenovo.
Since then I never consider Lenovo as a potential supplier. My way of paying them back.

 

[jgreco]

Thanks all, except for zambanini. I think that's the least friendly first reply I've ever had on a new forum and I've been on plenty.

We can fix that. Come back tomorrow morning before I've had my coffee and I'll serve up some truly cranky. ;-)

If this system gets too expensive it won't be done at all and I'll have to live with NTFS and bit rot, so finding a moderate priced solution is important. A standard PC like the one in this article is still an option too - knowing and understanding the risks, with good backups, make it viable.

Standard PC's (usually) won't do ECC, which is an important part of the data protection strategy. If you go that route, then you might not need RAIDZ2 either. If you want protection against bit rot, take the time to do it right, even if it means putting off a purchase a few months and cutting out a few nights at the bar to save a little extra cash up. It's basically a one-time up-front expense to get a filer that'll serve you well for years to come. I know that's not what you want to hear - no one does - but in the end it sometimes turns out to be the cheapest route when you don't need to replace the darn thing when something bad happens.

 

[TimNZ]

To be honest building an always on server grade computer is seeming like overkill for my situation, especially given the high memory requirements and having to import hardware from the US. I'm considering just using Microsoft's ReFS and Storage Spaces. It's a totally different solution to a ZFS NAS, but will probably solve my actual problem - bit rot. I don't actually need a NAS as my PC is on when I need data, the PC is powerful and has plenty of space for drives, has cooling, etc. I'll ponder this option for a while.

 

[jgreco]

It isn't actually a totally different solution. It's Microsoft's attempt to implement something like ZFS but "better." Because ZFS does have some warts.

http://www.trios.com/career/?section=Whatsnew117

I think this is a fairly good summary of the state of things.

I am not "amazed" by ZFS on average, because in order to get amazing performance out of it you have to throw a lot of resources at it, and with resources comes performance.

I'm willing to grant that ReFS should improve with time, but as it stands the only way to get acceptable performance out of it is to disable integrity checking, which kinda defeats the point.

However, the reality is that to actually get reliability in a filesystem, including things like data integrity validation, you need certain qualities. It's a hard problem. You can't just throw arbitrary software on random hardware and get it. You still need ECC for reliable validation under ReFS, and you still need good hardware to get good network performance. The FreeNAS community is a bit unusual in that people here have done a lot of the work for you, learning, documenting, teaching. We try to put it all out there up-front.

 

[TimNZ]

Agree that ZFS is a better solution, but sometimes "good enough and cheaper" is suitable for the job - detecting and correcting bit rot is my primary concern. I wouldn't use a Storage Spaces parity set, they're super slow, I'd use a mirror set which apparently works reasonably well.

I've run memtestx86 for a good long time without a single bit error, so I'm comfortable taking the risk with my current PC. I suspect ReFS will do fine for me, as I'll be using it very simply - one mirrored pool. If it's not up to the job then I know FreeNAS on a server is a good viable option with good community support if I want to spend the money.

I appreciate the help offered, and if nothing else I learned something and hopefully this thread could be useful to others in future.