I am so green...

[Chris Moore]

It'll be interesting.

I heard a saying once, went something like, "May you live in interesting times," but I don't recall the source, probably a movie...

 

[Elliot Dierksen]

I heard a saying once, went something like, "May you live in interesting times," but I don't recall the source, probably a movie...

I heard that as an ancient Chinese curse.

 

[jgreco]

I heard that as an ancient Chinese curse.

Nasty muppet hecklers should know.

 

[dakotta]

My ears have been burning... there is a lot more activity on this forum than I expected. :)

Thanks everybody for your comments. As usual, a lot to think about.

JohnK -- thanks for the tip about virtual machines. I'll look into it.

The 'overhead' in ZFS is the amount of data the file system stores on disk, like checksum data, that is required by ZFS but it is not your actual data.

The amount of space that you are not supposed to use, 20%, is because the file system writes changes to existing files, or new data into free space first, so you need to have some free space to make the write. After the write is successful, the space where the file was stored before is marked as free, if you don't have a snapshot that is holding that block from being free. When you fill the pool beyond 80%, the performance is reduced because the system is having to spend more time hunting for free space. When you fill it beyond 90%, performance falls off a cliff because the system changes to a whole different algorithm to optimize use of free space. As you approach 100% the system will grind to a halt. The recommendation to keep 20% free space for the copy-on-write nature of the file system has nothing to do with the actual 'overhead' of the checksum data that ZFS creates, it has to do with the ability to use the file system at all.

This has been explained many times on the forum. I am paraphrasing what I read on this forum years ago when I first started investigating using ZFS.

I was thinking that the 20% for COW included the 'overhead'. But then I misread the results of the calculator anyway, so I was wrong on both counts.

For (5) 4 TB drives connected as RAID-Z2, the calculator I linked to suggests 0.192 TB of 'overhead' in addition to 2.362 TB of 'free space' for the filesystem, giving me 9.4 TB of 'useful' data storage. Not the 12 TB I was expecting.

For (6) 4 TB drives, I might get 12.6 TB, not the 16 TB I was expecting.

Thanks for the head's up, Chris. Also thanks for the comment about snapshots adding up quickly.

====================

A couple of the parts I was hoping to buy have already become obsolete (like the Fractal Design Define R5) so I was hoping to buy some hardware soon and then continue reading about ZFS and FreeNAS, but it looks like I've got my priorities backwards. I worked my way through the "Slideshow explaining VDev, zpool, ZIL and L2ARC" and now I'm reading the manual. And after that I'll work my way through the rest of the links in Chris' signature.

This is both a lot harder and a lot easier than I was expecting. It seems like everyone else's job is always so easy... until I try to do it myself. :(

Right now, my thinking is:

I don't have any professional IT experience; I don't have any server experience. But I can read and think, and I have a lot of patience. The hard work has already been done by people who are far smarter than I am. It's looks like every question I could possibly ask (beginner that I am) has already been answered on this forum. Guides have been written covering the most dangerous pitfalls.

So... if I read and take the time to understand... thoroughly test the system before trusting it... and monitor the resulting build... I should be all right.

Time to get out my notebook and get cracking!

Cheers,

 

[Chris Moore]

It seems like everyone else's job is always so easy... until I try to do it myself.

LOL. If you have any questions, someone here will be happy to help.

Time to get out my notebook and get cracking!

Sounds like you have the right attitude. Something to contemplate, you can setup a 'for testing' system using hardware you may have laying around. I would never suggest doing that and trusting your data to it, but it would let you do some learning about how all the features work.

 

[CraigD]

This forum is really good, I had only ever used Windows, read a lot, asked a couple of questions when I got stuck

A Server is a computer that provides services, if you can build a computer you can build a server!

I have been using a Verbatim 4-disk RAID 5 storage array (1.3 TB) for about 8 years and I'm becoming increasingly anxious about the stability of the array and my ability to rebuild it if it should fail. (I have spare disks, but if the circuit board fails I don't think I'll be able to find another.)

FreeNAS primary goal is data integrity, should your hardware fail your pool can be transferred to another system without a rebuild

I've never done anything like this, so there will be a steep learning curve and many hours of breaking/fixing things before I'm happy.

You are ahead of most, It help greatly if you know how to fix problems that could happen (replacing drives, expanding pools, using snapshots and rsync)

My goals are:
1. rock-solid backup strategy, with versioning and off-site backups

So you need TWO freeNAS servers...

2. the ability to stream music and movies from the NAS to anywhere on my local (home) network

A SAMBA share works, I use PLEX

3. the ability to run virtual machines for Windows and Linux (one machine at at time)

I've never done this I can't help but do you really need to do this?

4. the ability to stream music/access data from the NAS to a remote machine on the Internet through a VPN

PLEX will stream remotely (Unless you are double NAT'ed) VPNs can be used

I have some ideas on hardware, but it seems like the first step is to figure out my required storage capacity.

Your hardware choices are solid, and your right pool design is very important

Right now I have around 4 TB of data that I want to back up. If I double that in 3 years and take a 20% hit for overhead, I will need 10 TB of useful storage capacity.

Well you used your last "server" for 8 years, doubling your data every 3 years seem conservative, you need to plan ahead, the most drives you can get in the define R5 is 12(8 in bays, 3 in the 2 5.25" bay, and one on a hanger) I would start with 6 drives then add another 6 when needed

Growing your server(s) over the next 8-9 years is possible if you get the right motherboard, RAM, and CPU

Older doesn't mean inferior, often you get the same performance for less money! You should be able to get an R5 case (Buy it ASAP) I captured a unicorn when I found an Antec 1200 case (holds 16-20 drives)

Have Fun

 

[JohnK]

So you need TWO freeNAS servers...

I will honestly recommend this. I have lost my pool through stupidity once and this saved me big time.

Also, I want to add that when I used 7 drives inside the Fractal Design R4, I had to add a few extra fans to keep my drives from cooking. Also, since moving my installations off my two Fractal Designs Arc systems, I have burned out two power supplies in those boxes. An Antec and Seasonic. Not saying that I would have lost data, but I now prefer redundant power supplies.

 

[Chris Moore]

I posted this a week or two ago, so these exact items may not be available now, watch out for eBay redirecting you to other "similar" listings, but take a look at this to get an idea of the kind of money you can save with used server gear instead of buying new:

CASE: Fractal-Design-Define-R5-FD-CA-DEF-R5-BK-Black-Silent-ATX-Midtower-Computer-Case - - US $89.24
https://www.ebay.com/itm/253026336681

POWER: Corsair-Certified-CS-M-Series-CS650M-650W-80-Plus-Gold-Active-PFC-Modular-Power - - US $64.99
https://www.ebay.com/itm/382130407495

These are older components, but still very powerful.
I use similar components in both my 48 bay primary NAS and in my 24 bay backup NAS that runs ESXi with FreeNAS in a VM.

This one comes with a CPU and fan, but it is really slow, so probably want to replace it:

System Board: Supermicro X9SRL-F Motherboard LGA2011 System Board w/Intel E5-2650L 0 @ 1.80GHz - - US $174.99
https://www.ebay.com/itm/132762499937

Memory: SAMSUNG 16GB Samsung M393B2G70QH0-YK0 PC3L-12800R DDR3L 1600 2RX4 Server Memory RAM - - US $40
https://www.ebay.com/itm/183457371966

If you find that you want a better or quieter CPU cooler, I use this model on two of my systems.
It is only slightly louder than the Noctua cooler I have on my wife's desktop PC.

CPU Fan: Dynatron R27 Side Fan CPU Cooler 3U for Intel Socket LGA2011 (Narrow ILM) - - US $39.59
https://www.ebay.com/itm/401284811045

The CPU that comes in the board above is pretty low speed, so you might find that you need a better one.
You have a lot of options for CPUs to go in this board, but I recently bought one of these for myself:

PassMark score of 13073... If you are wondering... This is the model I use for the NAS I run Plex in.

CPU: Intel Xeon E5-2650 V2 2.6GHz 8 Core 20MB 8GT/s SR1A8 LGA2011 ( Ivy Bridge ) Processor - - US $89.97
https://www.ebay.com/itm/142937685210

For the drive controller, I would suggest a SAS controller, just to get all the drives on a single controller. It works better that way. One SAS controller like this can run up to 256 drives by use of expander controllers. We can talk more about that when you need more drives but this will get you to eight drives to start.

Drive Controller: SAS PCI-E 3.0 HBA LSI 9207-8i P20 IT Mode for ZFS FreeNAS unRAID - - US $69.55
https://www.ebay.com/itm/162862201664

Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

I would suggest one of these SSDs for the boot drive.
These are used data-center drives, but as a boot drive in FreeNAS, it should last as long as the server, if not longer:

Boot drive: Intel SSD DC S3500 Series 2.5" SATA 6Gb/s, 20nm, MLC 80GB - - US $29.99
https://www.ebay.com/itm/273102509397

Thermal Compound: Noctua NT-H1 Thermal Paste Grease Conductive Compound for CPU/GPU - US $6.95
https://www.ebay.com/itm/302624513215

Just rough math in my head, I think that is all under $650.... Still, you might need some odd bits, but it should be simple to get there from here and this should save you a buck or two vs buying new and still do the job for years to come.

 

[Chris Moore]

Not saying that I would have lost data, but I now prefer redundant power supplies.

Speaking of which. If you even think you might end up with a lot of drives, a chassis like this is a pretty decent investment:

Supermicro 24 Bay Chassis SAS846TQ Server 4U - - US $399.99 + $64.15 Standard Shipping
https://www.ebay.com/itm/183457141415

I would buy one if I didn't already have six servers at my house. Two of them are small though, so they shouldn't count...

 

[southwow]

FreeNas with 32 Gigabytes of ram and any decent 4-core (8-thread) CPU will allow all of this. No worries about being new to this. I've been using Unix OS's all of my life and still have much to learn!

Edit:

+1 on @Chris Moore 's suggestion of the 846TQ case. I have two of them (one as my main storage, and the second ready to come online as a JBOD once my offsite backup finishes... later this week or early next week!).

with the TQ, you've got some future-proofing as it is simply a pass-thru without a SAS expander... meaning whatever your SAS cards (and/or expander) do as a transfer rate, the TQ should theoretically do!

Some people say cabling is a nightmare. They're wrong for the most part. Just use some high-quality .75 or 1 meter sheathed SAS breakout cables and it will be clean and a complete breeze!

Also, you eliminate a point of failure: the expander in the backplane. I, personally, prefer to keep my enterprise hardware as non-integrated single-purpose as possible... meaning I have room for an offboard expander, I don't NEED SAS onboard as I can use a separate card, etc.

If my board dies, ANY recent board will work as a replacement as long as it meets the BSD and FreeNAS hardware requirements. If my SAS expander dies; I don't need to change my entire backplane, I just remove 2 screws and add a new one. If my SAS controller dies, I cross-flash another and I'm up and running again in less than an hour.

 

[dakotta]

I posted this a week or two ago, so these exact items may not be available now, watch out for eBay redirecting you to other "similar" listings, but take a look at this to get an idea of the kind of money you can save with used server gear instead of buying new...

Just rough math in my head, I think that is all under $650.... Still, you might need some odd bits, but it should be simple to get there from here and this should save you a buck or two vs buying new and still do the job for years to come.

Chris, you post a lot of links to used equipment on eBay. Have you ever had any trouble with excessively short lifespans or dead-on-arrival equipment?

(The only time I have used eBay was to purchase a NEW cell phone. The antenna never worked and I was never able to get any response from the company that sold it to me.)

Cheers,

 

[Chris Moore]

Have you ever had any trouble with excessively short lifespans

Not on regular computer gear, but I have bought some used hard drives that didn't hold up very well. I buy all my personal computer gear second hand from eBay and have been pretty well satisfied with it all.

dead-on-arrival equipment?

Happened once or twice, but eBay has a very strong return policy and I have been fully refunded every time (even the cost of shipping) so I don't worry about that. It might take me some extra time, but it is worth it for the money I save. For example, my 48 bay 4U rack mount server chassis would have cost me over $5000 if I had got it new, but I picked it up for a small fraction of that and I have six servers in the house right now. Something I may need to adjust because it is taking up too much space in my small home office.

The only time I have used eBay was to purchase a NEW cell phone. The antenna never worked and I was never able to get any response from the company that sold it to me

I try to confine myself to US sellers. They are much more responsive. The thing about eBay policy, they will take the money back from the seller if you file a claim through eBay. If the seller doesn't respond to a message, go to eBay. You can even call them on the phone and talk to a person. I have done it several times, including getting the call escalated to a higher level support person.

 

[jgreco]

Chris, you post a lot of links to used equipment on eBay. Have you ever had any trouble with excessively short lifespans or dead-on-arrival equipment?

(The only time I have used eBay was to purchase a NEW cell phone. The antenna never worked and I was never able to get any response from the company that sold it to me.)

Cheers,

So you buy from eBay for the same reason you run FreeNAS: You want something great without paying full price to some vendor for it. You can absolutely get yourself a nice 12-bay Synology RackStation RS2416+ for $2500 and not have to worry about any of this, but vendor lock-in ....

eBay can be hit or miss for many things, but there's a crapton of stuff that is recycled by companies who specialize in "disposal" of gear. If you look at the seller info, and the seller ratings and feedback, you can quickly come to a conclusion. I buy some stuff used both for our own use here, and also for some customers who are sensitive to price, where having the latest and greatest is not a primary concern. Your IT dollars can sometimes go much farther if you're buying used. I have occasionally had bad RAM, bad flash, bad drives, missing cables, etc., but in ~90% of cases the seller is fine to work with, and it is just an oversight or not-sufficiently-tested (I suspect not-tested-at-all in some cases) type of issue. In a small number of cases I've had to file an eBay claim.

eBay can be strange. Many years ago, I ordered a used laptop for one of the partners for a travel concern where it wasn't going to be used except for that trip, so I didn't want to blow lots of cash on a laptop that'd end up in a closet. We got the laptop. It tested out like a champ, and after a week or two, I went to leave feedback. Seller had vanished. Listing was "not found." A day or two later, I got a stern note from eBay telling me to file a claim against the buyer. A week later, eBay refunded me. They wouldn't explain why any of this happened. I told them I had the sale item in hand and was happy with it. No difference. I started to get concerned that maybe it was stolen and that someone was going to come after me for the laptop. No, nada.

Most of the server gear you find on eBay was used by IT professionals in an appropriate environment and is being retired after a 3-5 year lease or life cycle arrangement. If you get that clear impression from a seller, such as they've got dozens of other IT items, with great feedback, then it's a purchase I'd not be too afraid of making (and do make often enough, spending ~$5K-$15K/yr on eBay vendors most years). I tend to buy drives and flash new if possible though.

 

[MatthewSteinhoff]

Have you ever had any trouble with excessively short lifespans or dead-on-arrival equipment?

DOA, yes. Short lifespans, not that we've noticed.

For the last seven or so years, my company has bought pretty much everything - desktop computers, networking equipment, servers, parts, iPads, etc. - either refurbished or used. Buying from sellers with a solid track history means you're relatively safe. Most of the folks on eBay these days seem to be actual companies that just happen to have an eBay storefront.

My only caveat is not to wait until the last minute to purchase. Our return rate is very low - maybe one of every fifty items. Typical problems are DOA or wrong part. In most cases, the vendor will send us a replacement for DOA items without even wanting the bad part returned. For the discount, you're the quality control department. Often, parts are pulled and mailed without testing.

We've saved enough money over the years that we have a spares on hand we probably couldn't afford if we were buying new. When we do get a dud, we deploy the spare to keep a project moving and then order a replacement.

Cheers,
Matt

 

[JohnK]

Most of the folks on eBay these days seem to be actual companies that just happen to have an eBay storefront.

I find that some bigger companies like Newegg and Superbiiz(imicro) have ebay storefronts. It is worth looking at as sometimes you get better shipping deals

 

[Chris Moore]

I am sure happy to hear that it isn't just me getting gear from eBay.

 

[jgreco]

I find that some bigger companies like Newegg and Superbiiz(imicro) have ebay storefronts. It is worth looking at as sometimes you get better shipping deals

And eBay Bucks.

I am sure happy to hear that it isn't just me getting gear from eBay.

Many of us in the service provider world rely on used equipment purchases. It's like how a new car depreciates 10% the moment you drive it off the lot, except that with lots of gear it can be more substantial than that. In the late 1990's I acquired a pair of Ascend GRF-400 routers (six figures new) for a song because they had a few software glitches early on, which caused their original owner to not totally love them, and then with the evolution of ethernet to 1G and the arrival of the Juniper M40, it became rather affordable to pick them up secondhand, and now one of them acts as my office printer stand, mostly because it's fun to have a printer stand that cost more new than most cars.

 

[southwow]

I find that some bigger companies like Newegg and Superbiiz(imicro) have ebay storefronts. It is worth looking at as sometimes you get better shipping deals

Never had a bad experience with superbiiz (they used to be ewiz). I've been buying stuff from them for at least 12 years. They seem to always have stock for obscure supermicro stuff like wide rails for the older rackmount cases.

I stay away from eBay, though, whenever possible. The exception is a chassis, no reason to fear them. literally all of the dozens of sticks of memory I've ever bought there have been defective unless they were registered.