Supermicro FreeNAS

[artlessknave]

Hello,

I am looking to rebuild my NAS, which was built with cheapo hardware and has since died. (https://www.asus.com/ca-en/Motherboards/E45M1M_PRO/)
I want to upgrade to ECC and something with actual performance, plus IPMI because I'm tired of screwing with monitors at the BIOS.
I already have some parts I intend to use so the plan would be to buy mobo+proc from newegg.ca and RAM from Crucial (buying the RAM seems to be troublesome for canada, newegg.ca only has kingston, which I'm not confident in compatibility)

I would like to ask if anyone can see any major problems with my selections.
The first I can think of is airflow in that case, but with no video card, a modded 250mm fan in the side panel, and cages with fans, I believe it will be sufficient.

I'm curious which of those motherboards people would choose and why. I tend toward the MBD-X11SSH-LN4F-O as I don't think the extra x4 PCIE will be of much use to me.

Thank you.

PARTS I ALREADY HAVE:

case: thermaltake armor
http://www.thermaltake.com/products-model.aspx?id=C_00000077

drive cages: Rosewill RSV-SATA-Cage-34 + ICY DOCK ToughArmor MB996SP-6SB ( 6 x 2.5" SATA 6Gb/s )
http://www.newegg.ca/Product/Product.aspx?Item=N82E16816132037
http://www.newegg.ca/Product/Product.aspx?Item=N82E16817994147

Storage: SATA NAS drives (pool seems to be fine, ran a scrub using a spare mobo)

SAS controller: LSI9211-IT+SATA breakout cables (LSI9240(IBM M1015) flashed to IT mode (LSI9211-IT)+SATA breakout cables)
https://www.servethehome.com/ibm-serveraid-m1015-part-4/
PSU: corsair tx750m
http://www.corsair.com/en-ca/enthus...rtified-750-watt-high-performance-modular-psu

PARTS I WOULD NEED TO BUY:

Processor: Intel Xeon E3-1230 v5 SkyLake 3.4 GHz 8MB L3 Cache LGA 1151 80W BX80662E31230V5 Server Processor
http://www.newegg.ca/Product/Product.aspx?Item=N82E16819117613

RAM: CT7982105 - Crucial 32GB Kit (2 x 16GB) DDR4-2400 ECC UDIMM
http://www.crucial.com/ProductDispl...p_category=&parent_category_rn=&storeId=10151

Motherboard:
m.2 + 4 NIC
SUPERMICRO MBD-X11SSH-LN4F-O Micro ATX Server Motherboard LGA 1151 Intel C236
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182996

2 NIC + 2x PCI-E 3.0 x4 (in x8) slot
SUPERMICRO MBD-X11SSM-F-O Micro ATX Server Motherboard LGA 1151 Intel C236
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813183013

 

[Ericloewe]

Please use the default font size unless there's a very good reason to do otherwise.

I tend toward the MBD-X11SSH-LN4F-O as I don't think the extra x4 PCIE will be of much use to me.

The extra two NICs and crippled M.2 slot surely will be of even less use.

 

[Chris Moore]

drive cages: Rosewill RSV-SATA-Cage-34 + ICY DOCK ToughArmor MB996SP-6SB ( 6 x 2.5" SATA 6Gb/s )
http://www.newegg.ca/Product/Product.aspx?Item=N82E16816132037
http://www.newegg.ca/Product/Product.aspx?Item=N82E16817994147

I understand that you already have this hardware, but I am confused by it. What is the reason for using 2.5" drives?

I would suggest a chassis like this, using higher capacity and less expensive 3.5" drives:
http://www.ebay.com/itm/Build-Your-...483625?hash=item23778b7569:g:2D0AAOSwo4pYNgDK

I would not choose either of those boards because they are WAY more money than I would spend. I always go a generation or three older than latest and greatest because I refuse to (for my personal use) buy showroom new. It is all about the price though, and I buy used cars too.

If I had to choose, just from those two options, I would pick this one: X11SSM-F
The reason being, for FreeNAS, there is no justification for the M.2 slot and it costs you a regular expansion slot. I would rather have my PCIe slot so I can put in a 10GB network interface or another disk controller. My servers have all four expansion slots filled. The next upgrade for me will be to an E5 board for more PCIe lanes and the accompanying expansion slots.

Would you like some recommendations on more cost effective hardware choices?

 

[artlessknave]

I understand that you already have this hardware, but I am confused by it. What is the reason for using 2.5" drives?

I would suggest a chassis like this, using higher capacity and less expensive 3.5" drives:
http://www.ebay.com/itm/Build-Your-...483625?hash=item23778b7569:g:2D0AAOSwo4pYNgDK

I would not choose either of those boards because they are WAY more money than I would spend. I always go a generation or three older than latest and greatest because I refuse to (for my personal use) buy showroom new. It is all about the price though, and I buy used cars too.

If I had to choose, just from those two options, I would pick this one: X11SSM-F
The reason being, for FreeNAS, there is no justification for the M.2 slot and it costs you a regular expansion slot. I would rather have my PCIe slot so I can put in a 10GB network interface or another disk controller. My servers have all four expansion slots filled. The next upgrade for me will be to an E5 board for more PCIe lanes and the accompanying expansion slots.

Would you like some recommendations on more cost effective hardware choices?

I have 3 of the 3.5" cages for 12 drives, plus one 6x 2.5" for SSD/laptop system/misc drives (ZFS boot pools didnt play well with my el-cheapo USB drives). this is more than i need at the moment for a cost of 0$
(i have thought about using a server chassis but their shape doesnt really fit with where i have to put it plus the cost for something I just dont need right now - this could be part of a future upgrade but right now want to get something that turns on without spending lots)

part of my costs consideration is actually being able to GET the hardware. shipping to canada can offset any savings from cheaper hardware, and newegg has those in stock ( even that mobo ships from the US rather than canada as is, so it may have customs on it, but companies like newegg with a presence in canada sometimes don't need to go through customs)

i would, by far, prefer relatively current generation hardware. it will get outdated anyway so i dont really like choosing older to start with.
i had looked at
http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182821
but while it has 6 more sata (2x SATA (6Gbps), 4x SATA (3Gbps) it also has 1/2 the max ram, costs more, the equivalent proc costs more, and it has less expansion slots than even the quad-nic board listed above

this system is just for me and partly just because I want it to experiment with. the only thing I'm currently likely to put in the extra (gimped) expansion slot is a GB NIC anyway, and a good 2 port NIC can be in the 100-200$ range anyway. I have nothing to connect a 10gb card to. 2 more LSI controllers would fill all 3 slots and be more drives than any chassis i have can even hold.

that said, if you have some good suggestions I would be interested, even if they don't fit my criteria at this time.

 

[joeschmuck]

You didn't mention what you use this for or how many hard drives you had in your system and the RAIDZ level which are factors. I also don't understand the 2.5" external drive bays but I have no opinion either way on those. With respect to the 3.5" drive bays, with the case you have selected I would just mount the hard drives inside the case and use the 120mm fans to blow air across them. Removable drive bays like you have selected have good reviews however they are just added on costs not required but I think I'd like them too if I had the extra money.

If you have 8 SATA hard drives or less in your system then you don't need the LSI card. If you need 12 SATA ports then I'd recommend the IOCrest SI-PEX40062 four port card, the only thing I don't like about mine is it's not an 8 port card so I have two of them but I use ESXi so it's a bit different that running FreeNAS on bare metal. You won't have to worry about firmware upgrades, it will just work, and save you a few bucks. If you are running ESXi then go with the LSI card, it makes more sense.

I don't personally know if the RAM you selected is on the approved RAM list for the motherboards you listed. I bought my Samsung RAM from Superbiiz at a great price.

 

[artlessknave]

I found the "approved" RAM a pain to trace and find where I could even buy it. that RAM is compatible with the boards according to Crucial.
whether I need the LSI card or not is immaterial, as I already have it. same with all of the drive cages.
I dont WANT the drives mounted inside, as then I have to tear half the monstrosity apart to replace any of them, and its easy to bump cables to OTHER drives
everything was originally in a define-mini
http://www.fractal-design.com/home/product/cases/define-series/define-mini
which is an awesome case (now houses my desktop), but the hard drives are terrible for making them easy to replace (their setup is to connect the cables behind the drives, requiring BOTH side panels to be off to replace any of them - I turned the drives around and plugged them in that way, but THEN putting a side panel on pressed on the cables and caused issues, I eventually just gave up and didnt put the panel on at all)
usage is storage of media, ISO files, experimental files, or anything else i need. performance wasn't really enough to do much more than that. ( I wanted gigabit speeds from a NAS, at the time the prebuilt ones advertised "gigabit" but their procs couldnt do much more than 20mb/s, this one could manage 60mb/s at least)
pool is currently 3 mirrors with 2 extra unattached spare 3TB drives (raidz inflexibility was getting in my way while I was having hardware issues and didnt have funds to fix it better and decided that I liked the simpler layout of just mirroring)
I originally built it with "green" drives (there werent really NAS drives available at the time) and have been replacing failures with reds (started with 4xseagate, 4xWD, ALL seagate's have had checksum errors and got replaced first)

boot mirror-0 298G 2.5"
boot mirror-0 298G 2.5"
boot mirror-0 15G usb
data mirror-0 2.7T wd red
data mirror-0 2.7T wd red
data mirror-1 1.8T wd green
data mirror-1 1.8T wd green
data mirror-2 1.8T wd green
data mirror-2 1.8T wd red
none none 2.7T wd red
none none 2.7T wd red
none none 476939 Meg 2.5"
none none 476939 Meg 2.5"

 

[Chris Moore]

The transfer speed issue that you have is down to your drive layout.

 

[artlessknave]

The transfer speed issue that you have is down to your drive layout.

at the risk of sounding snooty...what transfer speed issue?

 

[joeschmuck]

Saw you just posted your drive count. That is a lot of drives.

at the risk of sounding snooty...what transfer speed issue?

Where you are stating your transfer speeds were 60MB/sec is slow. Large files transfer at just over 100MB/sec for me. Lots of small files transfer much slower of course.

 

[joeschmuck]

whether I need the LSI card or not is immaterial, as I already have it. same with all of the drive cages.

I didn't understand that from your original post. It looked like you were looking to purchase the items you linked to. Maybe if you could be a bit clearer next time it would help.

boot mirror-0 298G 2.5"
boot mirror-0 298G 2.5"
boot mirror-0 15G usb
data mirror-0 2.7T wd red
data mirror-0 2.7T wd red
data mirror-1 1.8T wd green
data mirror-1 1.8T wd green
data mirror-2 1.8T wd green
data mirror-2 1.8T wd red
none none 2.7T wd red
none none 2.7T wd red
none none 476939 Meg 2.5"
none none 476939 Meg 2.5"

I would recommend a few changes here but I'm not forcing you to do anything...

1) Make a backup of your config file and rebuild the boot drive with only a single 2.5" drive. It is a bit of a waste of space using a 300GB drive but it's better than a USB flash drive any day of the week.
2) Remove all but one drive that you are running as a hot spare. This is just putting extra wear on a spare drive which could be used at a later date.
3) What are the 500GB 2.5" drives being used for? If nothing then I'd remove them as well.
4) Come up with a plan to convert over to a RAIDZ2 pool. You would make better use of your hard drive space but the more data you store means it will take more effort to backup that data and then create a new pool.

Some people like mirrors so if you like those and are happy with them, you should keep it.

 

[Chris Moore]

Where you are stating your transfer speeds were 60MB/sec is slow.

Right, @artlessknave it is likely that you are only reading from one or maybe two of your mirrors at any given time and that limits your transfer speed. Even if you were reading from all three mirrors at the same time, it would give your the total maximum transfer speed of three drives.
As an example, in my pool, I am reading from eight drives simultaniously (with some overhead) and the transfer is distributed across two controllers, so the throughput to the drives is much faster. It is not CPU bound as the CPU utilization rarely gets over 20%.

 

[artlessknave]

Right, @artlessknave it is likely that you are only reading from one or maybe two of your mirrors at any given time and that limits your transfer speed. Even if you were reading from all three mirrors at the same time, it would give your the total maximum transfer speed of three drives.
As an example, in my pool, I am reading from eight drives simultaniously (with some overhead) and the transfer is distributed across two controllers, so the throughput to the drives is much faster. It is not CPU bound as the CPU utilization rarely gets over 20%.

ummmm, it was slow because it was an amd fusion. that number was before I converted the whole thing FROM raidz2 to a mirror. i havent been able to use it since it was a mirror as just after that the mobo stopped being able to see PCI devices *at all*
the speed of a previous low power cheapo build has absolutely nothing to do with what the post was for
I care more about stability of the pool and vdev flexibility than raw speed, as well as not having the entire pool at reduced redundancy while raidz stresses all remaining drives to resilver (I can also move drives in the mirror in a way raidz simply cannot do, along with some other benefits), as long as it ISNT 20mb/s like the DNS-323 i had tried out had
the flash drive was only there to limit the boot pool to that size (otherwise any replacements would HAVE to be 300GB, excluding using a smaller SSD), and its only really still there because the whole thing is FUBAR and needs hardware before i can do anything with it


I'm about as bad at explaining this out so that people are answering what I was wanting answered as I expected...
I mainly wanted to know if my hardware choices would explode like a samsung smartphone

 

[Chris Moore]

I'm about as bad at explaining this out so that people are answering what I was wanting answered as I expected...
I mainly wanted to know if my hardware choices would explode like a samsung smartphone

Well, it sounds like you have pretty much decided what you want, regardless of any input that might be provided.

It won't explode.

The only question that remains is the amount of performance that can be had.

If you are so inclined, you can look at my hardware configuration and know this, I have sustained transfer rates of 858M/s with my configuration over a 10GB interface when copying data from one of my two NAS systems to the other. My RAID-z2 configuration can sustain the failure of ANY two disks and I will lose no data. I can replace a disk and resilver the container in about 2.5 hours because it hits internal transfer rate of over 900M/s. I can also be resilvering the array and still have enough horsepower to stream and trans-code video for my PLEX.

I can tell that you didn't want any advice, so that is probably the end of it.

 

[artlessknave]

Well, it sounds like you have pretty much decided what you want, regardless of any input that might be provided.

It won't explode.

The only question that remains is the amount of performance that can be had.

If you are so inclined, you can look at my hardware configuration and know this, I have sustained transfer rates of 858M/s with my configuration over a 10GB interface when copying data from one of my two NAS systems to the other. My RAID-z2 configuration can sustain the failure of ANY two disks and I will lose no data. I can replace a disk and resilver the container in about 2.5 hours because it hits internal transfer rate of over 900M/s. I can also be resilvering the array and still have enough horsepower to stream and trans-code video for my PLEX.

I can tell that you didn't want any advice, so that is probably the end of it.

sorry
you are mostly right in that i wasn't really looking for lots of advice, however, your setup does look pretty sweet.
currently that is so far beyond what i can possibly need right now, as I would have to buy a 10gb switch, 2 10GB NICs, and a whole other computer to really start to make use of that, in addition to the whole storage being *completely* unavailable right now

I seem to be a little unclear of the raidz setup, you have 2 6 disk raidz2 in mirror?
which specific supermicro chassis?

I'm absolutely interested in learning about different possibilities, but right now I'm mostly just looking for minimum to get it to turn on again so I can at least access the disks.

 

[Chris Moore]

I seem to be a little unclear of the raidz setup, you have 2 6 disk raidz2 in mirror?
which specific supermicro chassis?

I'm absolutely interested in learning about different possibilities, but right now I'm mostly just looking for minimum to get it to turn on again so I can at least access the disks.

I did use this setup before I built my 10GB switch, but I didn't buy a switch, I built it using the instructions you can find here:
https://www.youtube.com/watch?v=MgNpI6VAAhI

No, my two 6 disk vdevs are in one pool. Each of the 2 vdevs are RAID-z2 and could sustain two drive failures, so as long as it were the right disks, I could loose 4, but I would never want to be in that position. Each vdev gives me a little over 6TB of usable space after the overhead for parity, so the total 12 disks for storage only gives me a little more than 12TB of share space which doesn't sound much better than mirroring until you look at the resistance to disk failures. I am paranoid about losing my data to a disk failure. In a mirror, unless you make it 3 way mirrors, more than one disk failure looses your vdev and if you have more than one vdev in a pool, it looses your whole pool. Some people think that having a pool span two vdevs is a risk, but I have seven cold spares three of which are already in brackets and could be slotted into the enclosure in a matter of seconds. Then the rebuild only takes a little more than 2.5 hours. The last resilver I did was to replace two drives at once, one in each vdev, which took a little longer than for just one disk.
Here is the zpool status output:

Code:

[root@Emily-NAS] ~# zpool status
  pool: Emily
state: ONLINE
  scan: resilvered 1.43T in 3h14m with 0 errors on Sat Dec  3 19:09:24 2016
config:

		NAME											STATE	READ WRITE CKSUM
		Emily										  ONLINE	  0	 0   0
		  raidz2-0									ONLINE	   0	 0   0
			gptid/1f0d2f4b-b9a3-11e6-bc6d-002590aecc79  ONLINE	 0	 0   0
			gptid/740e4aee-52db-11e6-ae65-002590aecc79  ONLINE	 0	 0   0
			gptid/d7c0c9b8-af48-11e6-af9b-002590aecc79  ONLINE	 0	 0   0
			gptid/a24b0d9e-af31-11e6-af9b-002590aecc79  ONLINE	 0	 0   0
			gptid/bff1b9fb-b977-11e6-bc6d-002590aecc79  ONLINE	 0	 0   0
			gptid/b6fbb2e2-5c52-11e6-8395-002590aecc79  ONLINE	 0	 0   0
		  raidz2-1									ONLINE	   0	 0   0
			gptid/1cfedba6-b9a2-11e6-bc6d-002590aecc79  ONLINE	 0	 0   0
			gptid/7e7d3908-52db-11e6-ae65-002590aecc79  ONLINE	 0	 0   0
			gptid/7f3df14b-52db-11e6-ae65-002590aecc79  ONLINE	 0	 0   0
			gptid/b1b4f417-b978-11e6-bc6d-002590aecc79  ONLINE	 0	 0   0
			gptid/efe7fa77-5c6c-11e6-8395-002590aecc79  ONLINE	 0	 0   0
			gptid/a338376b-7898-11e6-95e0-002590aecc79  ONLINE	 0	 0   0

errors: No known data errors

The chassis I have is like this:
http://www.ebay.com/itm/SUPERMICRO-3U-CSE-836TQ-R800B-CHASSIS-16x-TRAYS-2x-800W-POWER-SUPPLIES-W-RAILS/172236203793?_trksid=p2047675.c100005.m1851&_trkparms=aid=222007&algo=SIC.MBE&ao=2&asc=38530&meid=c7011e4aa56c4fc787472b5d08553df7&pid=100005&rk=2&rkt=6&sd=152329483625
The backplane that the drives slot into direct connects the drive to a channel on the SAS controller, so I have to use two controllers to run all 16 drives. Not all my bays are populated, but I have a mirror of 1TB Seagate Constellation drives that I use to run my Jails for PLEX and virtual machines using VirtualBox. Then 12 more bays are dedicated to storage. I arrange the drives so that half of the drives are on each controller. It works really well for me and the chassis wasn't very expensive when you compare it against buying a full tower chassis and the hot-swap bays you have purchased. The big advantage is the redundant power supplies and I can still stand it on edge like a tower system.

Naturally, you will do your thing, but I think you could save significant cash and have a system that would do all that you want to do by picking up a second-hand system board, processor and memory from eBay.
I have three like the ones in my NAS systems. I use one for pfSense, and the other two for FreeNAS and they all came to me from eBay and have been totally reliable.
You don't need more than 32MB of RAM unless you are going to have more than 32TB of share space, so unless you go wild with your usage, the board I have is adequate.
I am just not a fan of spending money that doesn't need spending. Have a look at this:
http://www.ebay.ca/itm/Supermicro-X...889442?hash=item2824e834a2:g:Dm0AAOSwcUBYNHEb - Motherboard -C$ 106 + shipping
http://www.ebay.ca/itm/Intel-Xeon-E...813881?hash=item58e87fd579:g:66EAAOSwHMJYNK9- - Processor - C$ 159 + shipping
http://www.ebay.ca/itm/New-samsung-...563638?hash=item2a70691e36:g:gTwAAOSwMgdX0mOu - Memory - C$ 140 + shipping
http://www.ebay.ca/itm/4-Heat-Pipe-Quiet-CPU-Cooler-Heatsink-for-Intel-LGA1150-1151-1155-775-1156-AMD/131981017758 - CPU Cooler - C$ 26 (free epacket from China)

That fan needs 124 mm of clearance, so be sure to check it.

With the change from US $ to C $ and the cost of shipping, maybe it isn't a deal... I don't know. Around $450 plus shipping.

Yep, typed it all on the fly, sorry if there are any errors or omissions.

 

[artlessknave]

531$ CAD for those listed parts off of ebay with shipping included, which would be ~350$ less than my original plan but for older/used parts, and I just dont want to do the cheap option (thats whats being replaced basically)
those videos are good, however, they highlight even better how nothing i have is even remotely close to 10G ready...the nearest workstation to the server location is ~200m of cabling distance (and has no free PCIe), and nothing that is near the nas location can even take a 10g card. that could change in the future, so i think I will probably go with the extra PCIe board.
while I much prefer new for the main system, those parts look like they could be great for a backup/replication system, but at this time that is all future upgrades.
as an example, for that particular chassis they specifically ship only to the US. shipping large things like that are typically VERY expensive (I wouldnt be at all surprised if the shipping was significantly more than the chassis)
I very much like the idea of redundant power supplies in a good chassis but i have the case and cages literally all ready to be used thus why that would be a future upgrade as well

 

[Stux]

the flash drive was only there to limit the boot pool to that size (otherwise any replacements would HAVE to be 300GB, excluding using a smaller SSD)

FWIW, you can turn off the auto-expand property on the pool as well.

 

[artlessknave]

FWIW, you can turn off the auto-expand property on the pool as well.

auto-expand defaults to off on the boot pool, however, I don't know how to force the freenas installer to a specific size since it uses the entire drive.
I realize its an ugly hack, and getting actual SSDs would be less noobish

 

[Stux]

auto-expand defaults to off on the boot pool, however, I don't know how to force the FreeNAS installer to a specific size since it uses the entire drive.
I realize its an ugly hack, and getting actual SSDs would be less noobish

Good point. Maybe that should actually be a feature request for installing on say >32GB disks

 

[artlessknave]

https://forums.FreeNAS.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/
http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/
https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

these are part of why i chose mirrors instead of raidz