New large server build suggestions

[leonroy]

We've rolled our own Debian+Samba+Active Directory NAS boxes for a while now but the man power required to configure and maintain them is becoming a little too much.

We've tried a few NAS distros in vmware and FreeNAS seems the best. That said before proceeding with wiping one of our NAS boxes and putting FreeNAS on it I wanted to run the setup by the folks here:

• Chassis: Supermicro SC836
• Motherboard: Supermicro X7DBE-X
• CPU: 2x Intel Xeon 5150
• RAM: 8GB
• RAID: Adaptec 31605 RAID Controller
• HD: 16x 1TB Samsung F3

Our old Debian setup has been rock solid, but apart from being tricky to build and manage its CIFS performance has been pitiful (40MB/s up/down). We use CIFS and NFS mainly with a few clients reliant on AFP. Our old setup has two RAID partitions, one a 4x disk RAID 10 array for vmware hosts over NFS, and the other a large user data volume for Windows roaming profiles.

With FreeNAS I know a simpler storage controller is better but we'd prefer to keep the Adaptec we have unless there's a big advantage to be had?
In fact would we be better off using the hardware RAID controller's own RAID or should we use it as JBOD with ZFS across it?
If so what sort of volumes structure would be best suited with ZFS and a RAID 10 type level of redundancy?

Edit: to add, we have a daily backup executing to an identical server so whilst a high level of redundancy is good, it's not absolutely paramount.

 

[cyberjock]

With FreeNAS I know a simpler storage controller is better but we'd prefer to keep the Adaptec we have unless there's a big advantage to be had?
In fact would we be better off using the hardware RAID controller's own RAID or should we use it as JBOD with ZFS across it?
If so what sort of volumes structure would be best suited with ZFS and a RAID 10 type level of redundancy?

If you consult the manual there is a NOTE in the FreeNAS manual that answers all of your questions very bluntly.

 

[ProtoSD]

I think this is the link that @noobsauce80 is referring to:

http://doc.freenas.org/index.php/Hardware_Requirements

 

[survive]

Hi leonroy,

Lose the Adaptec & buy more RAM (FB-Dimms are kind of a bummer, but max it out on RAM).

Which Supermicro chassis is it (TQ, A E1 or E2....how do you connect the drives)?

What are you looking for out of your disks? Max capacity? Performance? Are you going to use the box with ESXi?

-Will

 

[leonroy]

Thanks for the replies. I'm very tempted to go for Synology all the way, but a part of me doesn't want to give up the flexibility of rolling our own NAS boxes. I think ditching the existing hardware and going for something lower power and current gen would be better. Does a spec like this seem sensible?

Supermicro X9SCL+-F Motherboard
Intel Xeon E3-1220LV2 CPU
2x4GB ECC Memory

We're using SAS expanders (836E1-R800B which uses the LSI SASX28) on some chassis and some chassis have 16x (836TQ-R800B). Would a card like the IBM M1015 work with SAS expanders?

 

[survive]

Hi leonroy,

If you are going to replace the hardware with that Supermicro board don't skimp on the RAM! The best thing you can do with FreeNAS is to give it a boatload of memory.

I like this vendor:

http://www.superbiiz.com/query.php?...rom+current+results&ob=r&myanchor=#displaytop

Sorry about the absurd url, but that's sorted on the 8GB unbuffered ECC memory you need for that Supermicro board. Check Supermicro's supported memory list and be sure the Samsung modules match up between the 2, then get 32GB.

You will probably want to look into adding some SLC SSD's if your NFS performance isn't all that great. ESX really wants sync writes which can drag down your performance.

-Will

 

[jgreco]

We've got some of the X9SCL+-F's running as ESXi nodes and we're very happy with them. We've also got some X9SCi-LN4F for FreeNAS use in 1U storage servers, similar board, just happens to have 4x gigE and less expansion.

For the Xeon, make sure you scope out Supermicro's CPU compatibility matrix to make sure that new part is compatible with their boards. We have some of the first gen E3-1230's and they are fine, nice, fast CPU's. And here's something you should know: despite what an E3-1230 is rated at (80 watts), we've bench tested an X9SCL+-F with E3-1230 and 16GB of RAM at 45 watts idle, about 65 watts under moderate load, and 95 watts full tilt... so you may wish to ponder the question of would it make more sense to buy a beefier CPU that can kick it up a notch under stress. On the other hand, the 1230's proven to be extremely competent, so it's quite possible a lesser dual core variant will still be adequate.

As for memory, do not do 2x4GB. The 4GB sticks are dirt cheap. Load up. More memory is better. 16GB is cheap and easy. You can go all the way to 32GB, for about $400, using Kingston KVR1333D3E9SK2/16G or similar, but the 8GB sticks do come at a price premium.

The M1015's are sufficiently cheap that you might want to consider using a pair of them, but do be aware that they burn something like 10-15 watts apiece. Performance is generally better with dedicated channels. We haven't used them with SAS expanders, sorry.

 

[leonroy]

Thanks, great info there, and thanks jgreco for the power statistics. Glad to see Intel's new stuff is so much more energy efficient!

This is what I've ordered, just want to run the setup by you guys in case I'm being dense and missing something:

Hardware:
Motherboard: Supermicro X9SCL+-F
CPU: Intel Xeon E3-1220L V2
RAM: 16GB ECC
HBA: 2x IBM M1015 in IT mode
Case: 16 bay SATA/SAS hotswap chassis
HD (Data): RAID 0 across 2x 8 (1TB) disk RAID-Z2 pools

Users will be approximately maximum of 3-4 concurrent and in total there's about 6TB of storage required for mainly photos, videos and possibly 6-8 vmware vmdks (mainly idling) served over NFS.

 

[paleoN]

Looks good. Do note the admonition about connecting with dedicated channels.

HD (Data): RAID 0 across 2x 8 (1TB) disk RAID-Z2 pools

I would actually put the vmware hosts on a separate pool. Besides separating vmware for the user data, it will allow you to play around with the zpool config if needed. E.G. if performance is too low you could try running 4 striped mirrors. If it's still too low you would likely need a pair of SSDs for the ZIL.

 

[pointman05]

Hello everyone,

I have speced the following server out with care but wanted to run it by some experts before I go purchase it. I have set up FreeNAS, openFiler, iSCSI, Isilon, XSAN and such so I am familiar with storage.

This system would be for a 3D and Graphics Design studio. Around 5-20 Users.

My 2 big questions are:
- With the Seagate Barracuda ST3000DM001 3TB drives, Will 3TB drives be an issue, Will I take a significant performance hit with the SATA disks in a RAIDZ2 config with 2 x 8 disk pools
- How does freeNAS do with 12 Core Sandy Xeons? Would I be better off with 8 Cores for stability?

Thanks ahead of time.

SuperMicro SuperStorage Server 6037R-E1R16N
◦ QTY = 1
◦ Main Chassis with Dual power, 4 port GBe NIC, Dual SAS Expansion
Seagate Momentus ST906403N1A1AS-RK 640GB 5400 RPM 8MB Cache SATA 3.0Gb/s 2.5"
Internal Hard Drive
◦ QTY = 2
◦ OS Hard drives
Seagate Barracuda ST3000DM001 3TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5" Internal
Hard Drive -Bare Drive
◦ QTY = 16
◦ Main Storage
Crucial 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1333 (PC3 10600) Model
CT102472BB1339
◦ QTY = 8
◦ Server memory
Intel Xeon E5-2620 Sandy Bridge-EP 2.0GHz (2.5GHz Turbo Boost) 15MB L3 Cache LGA
2011 95W Six-Core Server Processor BX80621E52620
◦ QTY = 2
◦ Server CPUs

 

[leonroy]

Seagate Momentus ST906403N1A1AS-RK 640GB 5400 RPM 8MB Cache SATA 3.0Gb/s 2.5"
Internal Hard Drive
◦ QTY = 2
◦ OS Hard drives
Seagate Barracuda ST3000DM001 3TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5" Internal
Hard Drive -Bare Drive
◦ QTY = 16
◦ Main Storage

Notebook drives aren't ideal for the OS, SSDs might be better. Although that said FreeNAS is designed to run fine off a USB flash drive.

I'd also recommend with that number of 3TB disks you don't want all of the disks to be failing around the same time, so mix vendors or at the very least batches.

 

[jgreco]

SSD's suffer failures around the same way hard drives do; a mirrored pair of notebook drives will be similar in reliability to a mirrored pair of SSD's. However, in both cases, you may be better off just finding a few USB sticks that are small, keeping one for the running image, one as backup, and one for upgrades, and that may well be nearly as good.

Definitely good advice to mix drives from different vendors or batches if you can. Try to pick similar performance characteristics if possible.