Sanity Checking new Build

[Stux]

I came across those shortly after posting, but thank you for mentioning them. Looks like they're used for mining cryptocurrencies, so you can cram more GPUs into a box without using slots on the motherboard. (Hint: that's what you want to search for - "pcie mining card" - if anyone stumbles into this thread and has no idea what we're talking about). Pretty neat, and ~$10. I'll have to add that to my list of things to play with. Seems like my design concept is becoming more of a Frankenstein's monster by the minute :)

And just to make it clear to people wondering about the mining cards, you don't need to use their 1x slot to usb3 to 16x slot functionality. Just their molex power to dummy slot functionality :)

 

[Robert Trevellyan]

Ubuntu 16.04 now has ZFS included ... I have no idea how solid it runs or how well it plays with others

Been playing with this for a couple of weeks. It works great, but the part of the setup I like best is LXD. It's like a combination of the best features of jails with some of the benefits of virtualization.

 

[Magius]

I just updated the first post based on some of the info/suggestions that have been discussed so far, and grayed out some of the original considerations/questions that are no longer relevant. It seems that most of my questions have been answered, but there are two more that I wouldn't mind some reassurance on. One was on configuring ashift=12 on 'old' disks with 512B sectors. Purpose is to lay a foundation for future upgrade to larger capacity drives, as I understand ashift cannot be changed after creation time. Any known problems/concerns with this in terms of data integrity? Second question was looking for a list of pool properties that can be configured, such as 'autoexpand=on' and 'atime=off'. Did I miss it in the manual?

Thanks again for all the help!

 

[Robert Trevellyan]

configuring ashift=12

Current releases of FreeNAS will do this on all drives in new pools by default, as far as I know.

'autoexpand=on' and 'atime=off'

As above, autoexpand=on is now the default for new pools.

atime is a property of datasets (including the top level dataset of each pool that shares the name of the pool), so you'll find that in the ZFS Dataset Options section of the manual.

 

[danb35]

It's going to be a very rare case where you'll have any occasion to mess with zfs properties at the command line. You should be able to see all the available properties using man zfs.

 

[Magius]

you'll find [atime] in the ZFS Dataset Options section of the manual.

You should be able to see all the available properties using man zfs.

Thank you both. I did find 'atime' and about a dozen other dataset options in table 8.1.4 in the Storage section of the manual. I noticed that that table did not mention 'autoexpand', which is one of the reasons I suspected there must be more options not listed. I used the search function on the docs page for 'autoexpand' but it found noting. Searching for 'atime', conversely, does bring you to Section 8, so the search function allegedly works...

As you may have seen in my first post, I found an Oracle document listing several options, some of which I couldn't find in the FreeNAS documentation (not that surprising) but similarly there was no mention of 'autoexpand' in the Oracle doc, leading me to believe it's not a complete list either...

Finally, I seem to recall another option related to large blocks (can't remember the exact name) that has come up in some threads, but I don't see that mentioned in the Oracle docs either, and wasn't sure what to search for in FreeNAS docs, since I can't recall the name.

I guess the long and short of it is I'd like to understand what *all* of the options/features/properties/etc. are that I might want to consider tweaking on my pool and/or dataset(s) to best suit my needs. At first I assumed I just couldn't find it, so was sheepishly asking for a pointer to the info. Now I'm thinking such a simple consolidated table might not exist at all? Maybe the general consensus is that the installed defaults are best and tweaking is generally more harmful than good, but I'm the type that likes to understand what's going on under the hood.

I'll definitely check out the zfs man pages as suggested, hopefully that will clear things up. I just don't want to find out about some archaic option 2 years from now that can only be set at pool creation time that would have been great for my use case. Trying to do all that research up front and get it right the first time :)

 

[Ericloewe]

As you may have seen in my first post, I found an Oracle document listing several options, some of which I couldn't find in the FreeNAS documentation (not that surprising) but similarly there was no mention of 'autoexpand' in the Oracle doc, leading me to believe it's not a complete list either...

Don't even look at Oracle documents. It's been a completely different product for many years.

 

[Robert Trevellyan]

I'd like to understand what *all* of the options/features/properties/etc. are that I might want to consider tweaking on my pool and/or dataset(s) to best suit my needs. At first I assumed I just couldn't find it, so was sheepishly asking for a pointer to the info. Now I'm thinking such a simple consolidated table might not exist at all?

You won't find what you're looking for in the FreeNAS documentation. FreeNAS is intended to be used as an appliance, so many decisions have been made for you, including the configuration of many properties and features, and they are not exposed to you by FreeNAS. If you want full control of every aspect of your filesystem, you need to use a different product.

 

[Magius]

You won't find what you're looking for in the FreeNAS documentation. FreeNAS is intended to be used as an appliance, so many decisions have been made for you, including the configuration of many properties and features, and they are not exposed to you by FreeNAS. If you want full control of every aspect of your filesystem, you need to use a different product.

Fair enough. And I hope nobody interprets what I said above as derogatory towards FreeNAS. I'm a long time storage geek, I've held certifications from both EMC and NetApp (whoopdie-doo, I know, just establishing context) and I like to dig around under the hood, to understand and 'optimize' things, even though that sometimes makes them break :p I think it's amazing that something as complicated and powerful as BSD + ZFS + lots more has been integrated into a turn-key solution that 'just works' even for people with no experience outside of mounting a share drive on their Windows PC.

That said, in my mind I had pictured an analogy similar to this (bear with me, this is a stretch): You buy your storage appliance from [iXsystems/NetApp] and it comes with this great NAS OS [FreeNAS(TrueNAS)/DataONTAP] with a great filesystem offering unique capabilities [ZFS/WAFL]. The appliance comes to you pre-configured for 'best practices' and is mostly 'idiot proof' out of the box. You can connect it to your network/domain, toggle on some CIFS/NFS shares, and it will essentially 'just work'. From there, if you really want to, you can poke around, toggle some options, optimize the appliance for your data environment, or potentially blow the whole thing up if you're not careful :D. *That's* what I was hoping to dive into next with FreeNAS, all the dangerous little switches, but it seems some of those inner-workings aren't as clearly documented as I'd thought. The documentation that does exist has been great so far, I might add, I just thought there might be something missing in this area, so I was asking about it to make sure I hadn't overlooked it.

As I said above though, fair enough. I can accept that the devs know their product best, and I'll take your statement at face value that it's intended to be more of a hands-off appliance. In a way that's even a relief, as there's less for me to worry about in terms of "missing something", or remembering what crazy optimizations I made and why a couple years from now. Thanks again!

 

[Robert Trevellyan]

if you really want to, you can poke around, toggle some options, optimize the appliance for your data environment, or potentially blow the whole thing up if you're not careful :D. *That's* what I was hoping to dive into next with FreeNAS, all the dangerous little switches, but it seems some of those inner-workings aren't as clearly documented as I'd thought

Just to be clear, you can poke around with everything using the CLI if you really want to, and there may be good reason to do so in some cases. However, the documentation you should use for doing so is the FreeBSD man pages (zpool and zfs). The FreeNAS documentation is aimed squarely at the GUI user and doesn't address all the technical details of the underlying filesystem, just like it doesn't address all the technical details of the underlying OS.

 

[Magius]

The FreeNAS documentation is aimed squarely at the GUI user and doesn't address all the technical details of the underlying filesystem, just like it doesn't address all the technical details of the underlying OS.

You know, I feel a bit dumb now, but I'm really glad you stated the obvious for me. For some reason, when it came to documentation, my brain was not processing the obvious separation between FreeNAS and FreeBSD. You're right, of course. By wanting to dig deeper behind the curtains of FreeNAS, what I really seem to want to do is learn more about FreeBSD and its implementation of ZFS.

At least now I know where I should have been looking, and with hurricane Matthew headed my way tomorrow (work campus closed), I should have plenty of down time for research. Thanks again!




Sent from my SM-N900P using Tapatalk

 

[Robert Trevellyan]

:)

As you probably know, FreeBSD and ZFS on Linux pull from OpenZFS, so either would be a suitable vehicle for getting your hands dirty with the details of ZFS.

Matthew

 

[Magius]

Hello again folks. I'm having some second thoughts now on my new server build and looking for opinions.

As I put in the first post, my proposed build was going to be an X11SSH-LN4F Skylake board ($240), E3-1230v5 ($275) and 2x16GB RAM ($185). I liked the upgradability to 64GB down the road, compared to the 32GB limit on most last-gen boards, and the plan was to run ESXi with an HBA and two NICs passed through to FreeNAS. The only thing that seemed to be a slight negative was no upgradability in the CPU. Even recent leaks of Kaby Lake Xeons show the E3-1200v6 family limited to 4 cores, with just a slight MHz bump. It would be nice if years down the road I could drop an 8-core chip in this socket and extend the life of the server, but that looks unlikely on this platform...

The inspiration struck when I came across a thread on ServeTheHome tracking the price of the E5-2670, an old Sandy Bridge Xeon from early 2012, showing this ex-$1500 chip is available on eBay today for <$100. The SB architecture is a bit dated for me, but what if I gambled on a similar future with the Haswell/Broadwell platform? I could build the server today w/ a cheap Haswell E5 and count on the fact that in 4-5 years eBay should be flooded with cheap Broadwell Xeons.

I looked for LGA2011-R3 boards with feature parity to the Skylake platform and came up with the X10SRH-CLN4F (single socket) and X10DRL-LN4 (dual socket). Differences compared to the X11 are the loss of an M.2 slot, i350 NICs instead of i210, support for eight RDIMMS (cheaper) vs. four UDIMMs, a couple extra PCIe slots, and the CLN4F even throws in a SAS3008. Either board seems to cost ~$390.

So the thought process is that I could spend $150 extra on the board, drop in 2x16GB RDIMMs ($140) then settle in with a 'placeholder' CPU for a few years. The best candidates I've been able to find are E5-2609v3 ($225, 6c/6t, 1.9 GHz, 85W) or E5-1620v3 ($250, 4c/8t, 3.5 GHz, 140W ouch). Overall this build would cost ~$100 more than the original Skylake build.

The payoff for that $100 theoretically comes in 4-5 years when I should be able to drop in one or two new CPUs, something like the E5-2640v4 (10c/20t, 2.4-3.4 GHz, 90W) or E5-2618L (10c/20t, 2.2-3.2 Ghz, 75W), for pennies on the dollar. In theory the platform can support up to the $4000 E5-2699v4 (22c/44t, 2.2-3.6 GHz, 145W) but I'm trying to be semi-realistic here :D

What do you guys think? Sure it's impossible to predict which Broadwell Xeons will end up on eBay en masse (not likely the 'L' models), or when, or for how much, but does the theory seem sound that I should be able to find 'some' cheap CPU upgrade down the line to make this worth it? Is there some other big drawback that I've overlooked due to the "ooo shiny" of a platform that supports up to 44 cores, 88 threads, and 1TB of RAM? :pIf anyone else has done something similar for similar reasons, I'd be curious if you found a better 'starting CPU' for today, as I'm not 100% thrilled with either of the two Haswell options I've identified. Thanks!

 

[Ericloewe]

Hello again folks. I'm having some second thoughts now on my new server build and looking for opinions.

As I put in the first post, my proposed build was going to be an X11SSH-LN4F Skylake board ($240), E3-1230v5 ($275) and 2x16GB RAM ($185). I liked the upgradability to 64GB down the road, compared to the 32GB limit on most last-gen boards, and the plan was to run ESXi with an HBA and two NICs passed through to FreeNAS. The only thing that seemed to be a slight negative was no upgradability in the CPU. Even recent leaks of Kaby Lake Xeons show the E3-1200v6 family limited to 4 cores, with just a slight MHz bump. It would be nice if years down the road I could drop an 8-core chip in this socket and extend the life of the server, but that looks unlikely on this platform...

The inspiration struck when I came across a thread on ServeTheHome tracking the price of the E5-2670, an old Sandy Bridge Xeon from early 2012, showing this ex-$1500 chip is available on eBay today for <$100. The SB architecture is a bit dated for me, but what if I gambled on a similar future with the Haswell/Broadwell platform? I could build the server today w/ a cheap Haswell E5 and count on the fact that in 4-5 years eBay should be flooded with cheap Broadwell Xeons.

I looked for LGA2011-R3 boards with feature parity to the Skylake platform and came up with the X10SRH-CLN4F (single socket) and X10DRL-LN4 (dual socket). Differences compared to the X11 are the loss of an M.2 slot, i350 NICs instead of i210, support for eight RDIMMS (cheaper) vs. four UDIMMs, a couple extra PCIe slots, and the CLN4F even throws in a SAS3008. Either board seems to cost ~$390.

So the thought process is that I could spend $150 extra on the board, drop in 2x16GB RDIMMs ($140) then settle in with a 'placeholder' CPU for a few years. The best candidates I've been able to find are E5-2609v3 ($225, 6c/6t, 1.9 GHz, 85W) or E5-1620v3 ($250, 4c/8t, 3.5 GHz, 140W ouch). Overall this build would cost ~$100 more than the original Skylake build.

The payoff for that $100 theoretically comes in 4-5 years when I should be able to drop in one or two new CPUs, something like the E5-2640v4 (10c/20t, 2.4-3.4 GHz, 90W) or E5-2618L (10c/20t, 2.2-3.2 Ghz, 75W), for pennies on the dollar. In theory the platform can support up to the $4000 E5-2699v4 (22c/44t, 2.2-3.6 GHz, 145W) but I'm trying to be semi-realistic here :D

What do you guys think? Sure it's impossible to predict which Broadwell Xeons will end up on eBay en masse (not likely the 'L' models), or when, or for how much, but does the theory seem sound that I should be able to find 'some' cheap CPU upgrade down the line to make this worth it? Is there some other big drawback that I've overlooked due to the "ooo shiny" of a platform that supports up to 44 cores, 88 threads, and 1TB of RAM? :pIf anyone else has done something similar for similar reasons, I'd be curious if you found a better 'starting CPU' for today, as I'm not 100% thrilled with either of the two Haswell options I've identified. Thanks!

The big issue is that DDR3 might get more expensive soon.

 

[Magius]

The big issue is that DDR3 might get more expensive soon.

Eric I believe I took that into account and selected all boards and CPUs that support DDR4 instead of DDR3. Do you see a component in particular that I made a mistake on?

My expectation would be to start w/ 2x16GB DDR4 ECC RDIMM, and in the future when I upgrade the CPU to broadwell I can choose to fill up the remaining 6 slots w/ 16GB RDIMMs for "cheap", taking me as high as 128GB. The boards support 128GB LRDIMMs and max 1TB memory, but to me that seems ridiculous :)

 

[wblock]

As you probably know, FreeBSD and ZFS on Linux pull from OpenZFS, so either would be a suitable vehicle for getting your hands dirty with the details of ZFS.

Yes... except that FreeBSD has been running ZFS since 2008, while Ubuntu has only had it in the last year or so. Even after all this time, FreeBSD is still finding obscure bugs and improving integration, so do not expect the Ubuntu version to be equivalent in maturity.

 

[Ericloewe]

Eric I believe I took that into account and selected all boards and CPUs that support DDR4 instead of DDR3

Oh, I misread. I saw "Sandy Bridge" and glossed over the v3/v4.

 

[Magius]

Oh, I misread. I saw "Sandy Bridge" and glossed over the v3/v4.

No problem, there are a lot of variables with these builds and I'm always catching things myself. For example, I was considering compatible i7 Haswell CPUs as a moneysaving option, until I realized they didn't support ECC...

 

[Chris Moore]

Don't even look at Oracle documents. It's been a completely different product for many years.

There is some good information in the Oracle documents because so much of ZFS has not changed. The differences does cause new users some confusion though.
There is at least one other ZFS project out there that I ran across when I was doing research.
It makes administration of these systems a bit of a pain.

 

[Stux]

I used similar logic when selecting an e5 x10 platform.

The e5-1620v4 is approximately equivalent to an e3-1230v5. I wouldn't use the 2609. It's s poor cpu. Low speed low core count.

The next step up is the 1650v4, which is a great 6 core CPU which runs up to 4ghz (iirc).

And then, maybe in a few years you can find an eBay special with 20 cores ;)

I wouldn't bother with a Dual Processor board.