Production SAN build

[secret]

Hi,

I've recently been tasked to build a SAN for our still young and thriving company after we've started outgrowing our current equipment that was hosted and provided by a 3rd party. Still being new on the block a big budget is not available and unfortunately a Dell Equallogic is out of the question :(

Having used FreeNAS and ZFS before it was my first choice to build our SAN with a limited budget on. So I would like some input where this build can be improved and also I have some questions about a horse that's already been beaten to death : SLOG drives and L2ARC drives. Before I get into specs I should explain our current needs.

We host a couple of services in the cloud, 3x 3CX (VoIP) servers, 2 Crashplan (Client Backup) servers and a plethora of servers (8x) that we use for internal use (AD, WSUS, SQL, Vyatta and some monitoring application we use to monitor our servers). These are all VMs within 3 ESXi hosts connected to the storage via iSCSI. With all these services running I think we need a storage solution that doesn't have to be excellent, just good at being well-rounded but with depandable redundancy, because I like spending my weekends at home and not in a datacentre.

This is the build that I've scrapped together in the last 48hr with some research :

• CPU - Intel Xeon E3-1220 – LGA 1155 – 8M Cache, 3.1 GHz
• Motherboard - Tyan S5512 – S5512GM4NR
• Controller - LSI SAS2008 Controller
• RAM - Transcend 4Gb DDR3-1333 ECC Module x 4 (16Gb in total; I would like 32Gb but I'm still waiting on Tyan to confirm which 8Gb sticks are supported)
• Storage Drives - WD RED NAS series 3Tb x 8 (raidz2)

All of this is going to run in a chassis with 3 redundant 80+ Gold PSU and I've made sure that all components are supported by the motherboard and manufacturers. I'll still be using iSCSI to connect my ESXi hosts to the storage.

Now, onto SLOG drives!

• For my SLOG SSD(s) I've chosen the Intel 330 60GB. My question is would it be advisable to run two Intel 330 60GBs in RAID0 to improve throughput on the SLOG drive?
• What 10GbE NIC would you recommend?
• For my L2ARC SSD I've chosen the Intel 520 120GB, is this a good choice?
• And lastly, I'm not too comfortable with running FreeNAS on a USB stick in a production environment, so would running it on 2x 250Gb 7200RPM desktop grade drives (new, that I pulled out of a couple HP microservers) in RAID0 with a 32GB USB stick to backup the config to be any good?

Any assistance would be greatly appreciated.

I apologize if somewhere in this wall of text I wasn't completely coherent, been awake 28+ hrs.. I think I might need 3 redundant power supplies myself.

Thank you!

 

[jgreco]

Scrap the Tyan unless your company is already married to them.

Netgear, TrueNAS, etc., all use the Supermicro platforms because they're well-built and totally awesome.

For 24TB worth of disks, the Hardware Recommendations make it clear that you ought to have 24GB of RAM, but for SAN use this is more complicated. Do you know the working set size of your stored data? You are on the edge of outgrowing Xeon E3 and needing to move up to E5-1600.

So here's the thing. I'm just going to throw out what *I* would do and briefly maybe why:

1) Get a Supermicro SC826BA-R920LPB chassis. Redundant power. 12 3.5" bays. 2U. 2 2.5" bays too.

2) Take ten (not eight) 3TB RED drives and stick them in bays 1-10. Use RAIDZ3 on 9 of them, hold one back as a warm spare. You are much less likely to encounter a catastrophe with RAIDZ3. Actually 11 drives would be the ideal number for best performance, and if you wanted to go that route, you'd get an even dozen drives so you had a warm spare. But my plan involves using two bays for SSD's, so not an option.

The spare drive means you can log in from remote and start a drive replacement operation with no one on site.

3) Take your SLOG and L2ARC devices, stick them in a 2.5-to-3.5" converter, and stick them in the two front bays not yet allocated.

4) Take the cheapest supported RAID controller you can find (or maybe an onboard one) and two cheap/small SSD's in the rear drive bays for the OS. Set them up in RAID1.

So, me, wanting 10GbE, I would notice that it is cheaper to buy it on a board than to buy a PCIe for it. And that pushes you up to Xeon E5 anyways, so totally awesome.

5) Buy an X9SRH-7TF. Onboard dual 10GbE. Also onboard RAID for the OS as mentioned in point 4. You might even be able to get away with using the onboard controller for the backplane, but only if you can actually get them to pass through (don't know the answer here).

6) Get a really fast quad core E5-1620. Relatively inexpensive (E5 is usually $$$$$$$$). Insanely fast, ideal for a NAS.

I've been writing a document on SLOG devices over in the performance forum so I expect you can go read my thoughts on that there.