BUILD My Socket 2011v3 Build

[ian351c]

For anyone who wants to follow along, I know Socket 2011v3 is considered trailblazing here in the forums...;)

The purpose of this build is to replace my existing FreeNAS server with something that reflects the lessons learned with that build and will allow me to accomplish the following goals:

• Home File Server
  • Storage of Family Pics and Home Movies (with backup to Crashplan)
  • Storage of the Family Media Library (a few TB of music, movies and TV shows)
• Home Media Server - Plex
• Personal Cloud Server - Owncloud
• Log Analysis - Splunk
• Backup - Second FreeNAS box that is semi-portable
• Time Machine
• Future Projects
  • Home Automation
  • Home Security
  • 10GbE when it gets a bit cheaper
  • Central Authentication (Samba as Domain Controller)
  • etc.

Current Environment:

• FreeNAS: Core i3-2100T, 16GB RAM (non-ECC), ASRock H67M-ITX, 3Ware 9650-SE, 6x 3TB RAIDZ2
  • Note that the RAM is non-ECC (lesson learned)
  • The 3Ware RAID controller is running each disk in Single Mode, which gets me write caching with no actual RAID functionality, but since the controller is ~6 years old at this point and only has 256MB RAM, it's not buying me much in the way of performance and I still have to deal with the fact that if the controller dies it will take all the disks with it (i.e. they can't just be plugged into motherboard SATA ports and be recognized by FreeNAS as the same disks)
• ESXi: Core i5-2400S, 32GB RAM, Local SSD for VMs
  • Linux VM running Plex, Owncloud, Splunk, MythTV, cacti, Pocketmine, Crashplan...
  • Another (isolated) Linux VM for DMZ type stuff
  • Windows VM for Windows Stuff
  • pfSense VM for playing with IPv6 (my firewall doesn't support GIF tunnels)
  • Infoblox DDI VM
  • Juniper SSL VPN VM
  • Quantum vmPro for VM Backup to NAS

New Environment (v0.2):

• FreeNAS: Xeon E5-1620v3, 32GB ECC DDR4, SuperMicro X10SRL-F, IBM M1015, 6x 4TB RAIDZ2
  • Note: At first I was just going to use the built in SATA Ports, but the nerd in me really wanted the M1015 and for ~$100 I couldn't say no.
  • The idea here is to run "production" services, especially those that would benefit from direct access to the FreeNAS storage in a jail and leave the ESXi server for ESXi specific and/or lab stuff.
  • Also, Plex is getting to the point where it needs more horsepower than I have in the ESXi box, but running the NAS box without ECC RAM is a Bad Idea(TM), and I don't have the budget to redo both boxes at once.

• ESXi: Same as Current, but with some of the services migrated to a jail on FreeNAS

• Backup: The old FreeNAS guts, transplanted into a BitFenix Prodigy case so it's semi-portable

Build Notes:
I Bought a Corsair H60 Liquid Cooler for the Xeon. I've been liking these things since I bought one about 4 years ago. They're quiet, self contained, and not attractive to dust bunnies. All my research said that it should work, and even the box said it was compatible with Socket 2011. It turns out that Socket 2011 can be different from Socket 2011... As the nice support person from Corsair said: "Our Hydro coolers are compatible with the LGA 2011-3 (square mounting style). Most server-type boards use a rectangular mounting style which we unfortunately do not support." :( I mangled the mounting bracket to make it fit (mostly) and am getting good temps, but I may end up getting a dust bunny magnet after all.

Server class motherboards are so worth it! :D IPMI is the best thing since sliced bread (and much, much cheaper than a KVM over IP box that wouldn't work when the system was off anyways).

Since I have the opportunity, I'm playing around with some benchmarks. I'll post more about those in a follow up.

Relevant Threads (though I'm sure there are others I should mention but have forgotten):
Cyberjock's Hardware Recommendations
Jgreco's SuperMicro X10 FAQ
Jgreco's Burn In Thread (Though I'm not as paranoid, er I mean thorough as he is...;))
https://forums.freenas.org/index.php?threads/benchmarking-zfs.7928/
https://forums.freenas.org/index.php?threads/checking-new-hdds-in-raid.12082/#post-85381

 

[mjws00]

Love it. Looking forward to seeing you put it through its paces.

 

[Mlovelace]

Watching this one :D

 

[ian351c]

Some performance stats as I go through the build. Right now, I'm using 4 of the 6 disks I purchased for this build (the other two are holding a backup of the existing NAS for the moment, but that should change soon).

These are the stats from iozone with a 42GB data set size (24GB for the first test on my old as since it only has 16GB RAM). I noticed that even with this large data set I was still getting lots of cache hits on the ARC, so I'm using it just for comparing performance on the old vs. new rather than for an absolute performance number. In any case, I'm more than capable of saturating my current network in any of these configurations.

Also note that the first test on my old NAS was probably skewed because it's always in use, at least a little.

Command line:

Code:

iozone -a -s 42g -r 4096

Edited to add: the first table of results is all with the built in SATA controllers since the M1015 hasn't come in yet.
EDIT 2: I was able to start using all 6 drives for testing. I've added those number into the bottom of the table.
EDIT 3: Added in results using the M1015 HBA
EDIT 4: A drive died, so here's what it looks like with a degraded pool...
Output (in KB/sec):

Code:

                        Dataset Size  reclen   write      rewrite    read       reread     rnd read   rnd write  bk read    rec rewrite   str read   fwrite     frewrite   fread      freread
Old NAS RaidZ2 24GB     25,165,824    4,096    128,770    331,418    439,603    410,563    26,930     315,247    93,804     4,294,201     96,355     323,450    338,511    390,499    368,182
RaidZ2 42GB 4 Drives    44,040,192    4,096    135,701    124,396    398,914    288,796    106,035    130,186    174,511    10,855,085    257,199    146,332    131,531    289,458    344,169
RaidZ 42 GB 4 Drives    44,040,192    4,096    257,407    235,743    539,094    403,582    126,624    263,375    246,394    11,056,810    333,380    276,657    209,197    433,729    432,961
Stripe 42GB 4 Drives    44,040,192    4,096    505,546    405,288    672,036    508,979    112,699    512,938    248,720    10,863,003    323,693    521,778    462,311    571,311    619,557
RaidZ 42 GB 6 Drives    44,040,192    4,096    372,065    339,236    484,987    492,428    177,775    297,261    279,517    11,083,228    354,139    352,766    324,638    591,328    539,071
Stripe 42GB 6 Drives    44,040,192    4,096    760,373    528,534    703,384    718,426    140,113    719,639    421,453    11,222,395    380,178    747,448    498,753    614,540    737,881
RAIDZ2 6 Drives M1015   44,040,192    4,096    417,146    324,520    520,275    624,620    209,778    416,612    298,640    11,033,743    347,269    380,850    418,978    607,585    604,325
RAIDZ2 Degraded M1015   44,040,192    4,096    397,897    311,410    494,840    550,418    211,415    370,757    301,008    11,382,074    324,752    376,339    325,469    499,571    518,219

 

[ian351c]

(Note: after this post, we take a little detour through replacing a defective disk and helping a fellow user flash their HBA card. If you wan to skip past the detour, click here.)


Well, the M-1015 came in yesterday. The good news is I was able to reflash it to 9211-8i IT firmware (I skipped the boot rom). I first read through jgreco's thread here, which leads to (somewhat outdated) instructions on reflashing the M-1015 here. I ran into an issue where the boot thumb drive I was using (Ultimate Boot CD, available here, "burned" to a thumb drive, which BTW the latest version comes with wdidle3 built in, so you don't have to worry about a "custom" UBCD anymore). Basically, the USB drivers on the UBCD break keyboard access to the system and instead of crafting my own UBCD with the LSI Flash tools on it, I discovered that you can do all this via UEFI boot shell as documented here. There are a few differences though. So here is my recipe:

• Step 1 - Download version 16 of both the UEFI Flashing tool and the DOS/Windows Package containing the flash ROM image from LSI Support.
  • Make sure it's v16 because that's the version of the driver FreeNAS 9.2/9.3 use and the firmware should match the driver.
• Step 2 - Decompress them onto a USB Stick
• Step 3 - Boot to UEFI shell on my SuperMicro motherboard
• Step 4 - Navigate to the folder with the flashing tools and ROM like so:
  • fso:
  • cd foldername
• Step 5 - Erase and Flash the M-1015
  • sas2flash.efi -o -e 7
  • sas2flash.efi -o -f 2118it.bin (Append -b mptsas2.rom if you want a boot ROM, but you don't need it)
• Step 6 - Reboot and profit!

The bad news is the SAS cables I ordered for the card are defective. Arg... So now I wait another week for another set of cables.

 

[Ericloewe]

Well, the M-1015 came in yesterday. The good news is I was able to reflash it to 9211-8i IT firmware (I skipped the boot rom). I first read through jgreco's thread here, which leads to (somewhat outdated) instructions on reflashing the M-1015 here. I ran into an issue where the boot thumb drive I was using (Ultimate Boot CD, available here, "burned" to a thumb drive, which BTW the latest version comes with wdidle3 built in, so you don't have to worry about a "custom" UBCD anymore). Basically, the USB drivers on the UBCD break keyboard access to the system and instead of crafting my own UBCD with the LSI Flash tools on it, I discovered that you can do all this via UEFI boot shell as documented here. There are a few differences though. So here is my recipe:

• Step 1 - Download version 16 of both the UEFI Flashing tool and the DOS/Windows Package containing the flash ROM image from LSI Support.
  • Make sure it's v16 because that's the version of the driver FreeNAS 9.2/9.3 use and the firmware should match the driver.
• Step 2 - Decompress them onto a USB Stick
• Step 3 - Boot to UEFI shell on my SuperMicro motherboard
• Step 4 - Navigate to the folder with the flashing tools and ROM like so:
  • fso:
  • cd foldername
• Step 5 - Erase and Flash the M-1015
  • sas2flash.efi -o -e 7
  • sas2flash.efi -o -f 2118it.bin (Append -b mptsas2.rom if you want a boot ROM, but you don't need it)
• Step 6 - Reboot and profit!

The bad news is the SAS cables I ordered for the card are defective. Arg... So now I wait another week for another set of cables.

I'd add that, for FreeNAS, the boot ROM seems to do more harm than good, so staying away from it is a good idea.

 

[ian351c]

Finally got the new cables in... These ones appear to work. Huzzah! I've added the iozone test results into my original post. I've also moved the hardware off my test bench an into its permanent home. Something wet a little wonky with all that hardware moving around though and one of my drives dropped out of the array with too many write errors, which is odd after sitting on my test bench for a month. I wanted to replace it with itself (which the GUI is no good at, probably on purpose) but I could figure this out (anyone know how to do this? the Googles and searching the forums came up with nothing that worked), so I ended up trashing the volume and recreating it. Now for some more testing...

iozone -a -s 1024g -r 4096 muahaha

But seriously, does anyone know how to replace a disk with itself?

 

[Ericloewe]

But seriously, does anyone know how to replace a disk with itself?

The only way I know that doesn't bring nasty side-effects is to wipe it and then resilver.

 

[ian351c]

I tried wiping it with:

Code:

zpool labelclear /dev/da3

But when I tried the "replace" operation, the GUI complained that the disk had the same GUID UUID as the disk I was replacing so no resilver for you! :-/

 

[Glorious1]

Is that the disk's UUID? There are some utilities that can change a disk's UUID, such as this one for Mac:
https://bombich.com/software/files/tools/reset_volume_uuid.zip

I appreciate your updated steps for flashing the M1015 to IT mode. I'll be getting that board soon. But I looked on the LSI support site and couldn't find the files you mentioned, unless they go by different names. I first tried searching by Storage > Raid Controllers > MegaRAID SAS-9240-8i for all assets. I found some firmware but doesn't look like what you mentioned. Then I searched for a UEFI flasher tool and only found Installer_P20_for_UEFI, which again doesn't sound right. Can you tell me how you found those?

 

[BigDave]

I tried wiping it with:

Code:

zpool labelclear /dev/da3

But when I tried the "replace" operation, the GUI complained that the disk had the same GUID as the disk I was replacing so no resilver for you! :-/

Just curious but wondering if it would make a difference if you zeroed out the drive using another machine?

 

[mjws00]

That is the wipe Eric meant. You need to kill the legit and backup partition tables. Often the quick wipe in the disk gui works fine. Full zero is the overkill I'm sick of fighting it option.

 

[BigDave]

Is that the disk's UUID? There are some utilities that can change a disk's UUID, such as this one for Mac:
https://bombich.com/software/files/tools/reset_volume_uuid.zip

I appreciate your updated steps for flashing the M1015 to IT mode. I'll be getting that board soon. But I looked on the LSI support site and couldn't find the files you mentioned, unless they go by different names. I first tried searching by Storage > Raid Controllers > MegaRAID SAS-9240-8i for all assets. I found some firmware but doesn't look like what you mentioned. Then I searched for a UEFI flasher tool and only found Installer_P20_for_UEFI, which again doesn't sound right. Can you tell me how you found those?

Try this link...
http://www.files.laptopvideo2go.com/hdd/sas2008.zip

 

[ian351c]

Thanks for the links guys. And yes, I meant UUID.

As for the LSI stuff, I have a sas2008.zip file (not sure if it's the same as BigDave's, but I got it from one of the links in my original posts and mine has all the P15 firmware on it, which is out of date). The files I ended up using were found by going to lsi.com, going to their support page from the main menu and selecting Storage > Raid Controllers > LSA SAS 9211-8i (which is what the M1015 becomes if you flash it). I then expanded out the "Firmware" section (85 files at this point), clicked on "Archived" and downloaded both the "9211_8i_Package_P16_IR_IT_Firmware_BIOS_for_MSDOS_Windows" (which has the firmware on it) and "Installer_P16_for_UEFI" (which is the EFI flash utility). The P16 is important because the firmware version has to match the FreeBSD driver version. Hope this helps...

 

[essg88]

Great build and thread. Lots of good info, hoping to achieve similar goals. Keep us updated

 

[ian351c]

So, I spent all afternoon (well, off and on) troubleshooting the disk that dropped out of the pool when I moved all the hardware to its permanent home. After deleting and recreating the pool, I couldn't make the drive drop out of the pool again. However, I got a ton of bus errors that followed the disk around between different cables, drive cages, etc. So, having determined that it wasn't anything outside the drive, I've decided to RMA it. Why is it that now that I've decided to do everything "right" I get all these problems?

The silver lining is that I have an excuse to run iozone on the degraded pool. And guess what? Performance is pretty much the same as a completely healthy pool. I've posted it up above in case anyone is interested.

 

[Glorious1]

Many thanks for the link BigDave and the explicit instructions on finding the files on the LSI site ian351c. Man this stuff is obtuse. Anyway, I eventually found those P16 files at Storage > Host Bus Adapters > LSI SAS 9211-8i > Firmware > Archived.

 

[ian351c]

Jail Builds

Since I sent in the bad disk for RMA, I'm trying to resist the urge to copy all my data from my old NAS and resilver it all when the new disk arrives. But since the jails involve relatively little data, I've decided to move forward with building them. I've decided against using any plugin jails, since I'm comfortable enough being my own FreeBSD admin and like the ability to customize the jails to my heart's content. That, and jail snapshots make it easy to unshoot myself in the foot...

Notes on Jails for 9.3 Beta:
For some reason, the build I am running (11/27 and 12/3 so far) don't look for jail templates in the right place. I set the URL to: http://download.freenas.org/jails/9.3/x64/ which has a standard and a plugin template in it.
Feature request: set up the templates to have the same CLI environment as FreeNAS itself. Things like /etc/csh.cshrc and friends.

Jail 1: Plex
Easy peasy. Standard template install, pkg update, pkg install plexmediaserver-plexpass. Set up storage to map folders in the jail to my media on the NAS. Set up Plex as normal via its Web GUI.

Jail 2: Splunk
Also pretty easy. There's no port for Splunk, but it's super simple to install. Just follow the instructions on splunk.com. The one issue I ran into here is that jails are configured such that syslogd binds to udp/514 even though it's not configured to accept or send any data. I wanted to use udp/514 for splunk since I have at least one device (curse you Aruba Networks!) that you cannot confgure which port to use for sending syslog. Adding syslogd_flags="-s -s" to /etc/rc.conf fixed the port binding issue. Syslogd still runs, but doesn't bind to any TCP or UDP ports.
On the Splunk side, the only real issue I ran into was the Splunk for Palo Alto Networks app. It has a couple bugs in it and insists on using its own index. That took some time to fix, but is working as desired now.

Jail 3: Owncloud
Super simple. I used DrKK's Definitive Guide to Installing OwnCloud in FreeNAS (or FreeBSD). I like the idea of using lighttpd especially since each jail is dedicated to one application and there's no reason to use something as heavy duty as Apache for this.

Jail 4: Cacti
Again, very easy. There's a port for Cacti, which makes the install pretty easy. I cribbed from DrKK above again to get it set up with lighttpd rather than Apache. I'm thinking about migrating my data over from my old Cacti install, but I'm not sure I'm masochistic enough to go through with it...

So Splunk, Owncloud and Cacti are in production now, and I'll be turning down their old instances. Plex is waiting for me to migrate the data off the old NAS, which is dependent on the new drive arriving from WD.

I've been thinking about a Virtualbox jail for a Windows box and a Linux jail for Crashplan. From what I've read, neither of those is either easy or stable at the moment. Anyone care to comment on that?

 

[marbus90]

My experiences with 2012 R2 via virtualbox were outrageously bad. You need to be into SM to find that appealing.

 

[SweetAndLow]

I got crasgplan running in a BSD jail but haven't found a clean way to load the Linux module (kldload linux) in FreeNAS. If you have any ideas of how to do that and make it survive an upgrade I'd be interested.