BUILD Building a Fiber SAN

[Darren Myers]

Hello everyone! I recently found out through some googling of how to make my own iSCSI SAN, that FreeNAS supports it and after trying it in a small virtual environment and it working im considering a large storage using it, with that in mind im gonna go over what I'm thinking....and then ill take any constructive criticism and help in making my decisions :)

Here is what im thinking right now:
x16 2TB harddrives
Adaptec RAID 51645 (i looked and this card is supported) + SAS to SATA break out cables
More then likely a i5 CPU with 32GB of ECC RAM

-----BUT------

Here is where i am stuck, i received x2 Netxen Qlogic NX3-20G 10GB PCI-E FC Sever cards.
I am not sure if these are supported or if support can be added, i only wanted to utilize it because i came into them + there GBICS and fiber cables. My plan if i can use the fiber cards is add one to the SAN and one to my server and then fiber patch it into my switches SFP's

If my fiber cards aren't an option then i'll get x2 quad NICS and move on with my life.

Other question is, what suggestions can you make to improve on this? I only planned to use the 2TB harddrives because i can get them for roughly 50 dollars each, I'd rather not spend a whole lot for right now i already dropped $830 on x4 4TB harddrives for my server itself and am quickly running out of storage.

Other questions include:
1. What raid configuration would i want for 16 hdds to give me a good amount of space, i saw a post that RAID6 would give me 28TB (roughly usable) or if i did RAID10 it would give me 16TB (roughly usable).

2. Is the ZFS file system expandable? I.E: If i purchase 8 harddrives when building this SAN, can i add another 8 without issue down the road or reformating my data and all?

3. I am also open to a better/more preferred raid card

Sorry if this is a long post, im sorry if any of this information is incorrect I am doing my best to work with FreeNAS, thank you for taking the time to read my post :)

 

[Robert Smith]

i5 does not support ECC

 

[Ericloewe]

You have a bit of reading to do on ZFS:

http://forums.freenas.org/index.php...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

 

[Darren Myers]

Looks like i'll be getting another Xeon to go with my other servers :) no problem there, and thank you ill do some reading this evening. Thank you guys

 

[gpsguy]

As Ericloewe said, you need to do some more reading. In addition to the URL he provided, search the forum for iSCSI.

Don't buy a RAID card - you want a HBA instead. The IBM M1015 is popular. To expand beyond 8 drives, some users add another M1015 or a SAS expander. If the latter, try a forum search for the model number.

For best performance with iSCSI, used striped mirrors and plenty of RAM. Don't use more than 50% of your storage, else you'll run into fragmentation/performance issues.

 

[Darren Myers]

Alright now they my behind is getting straightened out, I still have reading to do...but if I get two of the M1015s and flash them over, fill both with 8 drives each how will freeNAS handle that? Will it allow me to make a raid out of all the drives together, or separate per card? Sorry if this is in the reading, and last I searched for iscsi but didn't see a whole lot in reference to what I am trying to accomplish, any links or reference?

 

[gpsguy]

Yes, if you get two M1015s you can create a volume using drives connected to both HBAs. You can also combine them with SATA connections on the motherboard. Consider buying a motherboard/CPU that can accommodate more than 32Gb of RAM.

Here's the latest sticky on hardware recommendations: https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/

 

[Darren Myers]

I will put together a different set for the motherboard and RAM, more then likely i will snatch up 2 M1015's. Doe anyone have any input on QLOGIC QLE3142 functioning with FreeNAS, i only managed to dig up a single post from a user who said he got his to work in 9.2.1...but no confirmation any that post https://forums.freenas.org/index.php?threads/ns522sfp-aka-qle3142-cu.18518/

Any advise?

 

[Ericloewe]

If you don't have a motherboard yet, save some cash by going with one that has an LSI 2308 SAS controller, like the X10SL7-F. It'll spare you from buying one of the two HBAs and is cheaper than separate components.

 

[Darren Myers]

Thanks for the replies everyone, i think i deceded on a motherboard, and have found some scattered information on its compatibility with FreeNAS. I read the hardware recommendations sticky and choose this motherboard as it will allow me to use my 2 spare 1366 xeons and a spare kit of 32GB of ram and its abundance of PCI-E slots. Any feedback on this board?

View: http://www.ebay.com/itm/171268449929?_trksid=p2060778.m1438.l2649&ssPageName=STRK%3AMEBIDX%3AIT

 

[cyberjock]

1366 CPUs aren't exactly power efficient. Other than that I don't see any problem with it.

 

[Darren Myers]

I'm rocking 2 servers atm with 1366's and havent had any REAL power draw. but we will see :)

 

[Darren Myers]

Updated parts list, opinions please :D

Motherboard: SUPERMICRO MBD-X8DTE-F-O
CPU's: x2 Intel X5560 (1366LGA @ 2.80Ghz)
CPU Coolers: x2 NH-U9DX
RAM: 24GB DDR3 ECC 1333Mhz
Case: x2 Rosewill RSV-L4500
Caches: x2 SSDSA2M160G2GC INTEL X25-M 160GB
RAID/HBA Cards: x3 M1015 Crossflashed P16 IT Mode
SAS to SATA: x6 HighPoint Int-MS-1M4S SFF-8087

+ my x2 10GB Fiber cards that need testing tomorrow.

Best way to utilize the SSD's as caches? RAID1 for better data writing protection?

 

[HoneyBadger]

Best way to utilize the SSD's as caches? RAID1 for better data writing protection?

You do want to mirror SLOG, but X25-Ms aren't terribly good at that job; they're old MLC with low write throughput and no power loss prevention.

Suggested new drives would be the Intel DC series (DC S3700, DC S3500) - other options would be the Seagate Pro 600. If you're stretched for budget, Intel 320. I've seen some folks using the Crucial MX100 since it's cheap as hell and has power loss prevention, but I'm wary of using "value SSDs" for SLOG.

Your X25-Ms would be great for L2ARC if your workload can benefit, but you'd get better results from more RAM first.

 

[Darren Myers]

Thanks honeybadger i have scrapped those SSD's off my list and will look into those other models. I got my x2 10GB NIC cards, plug and play for my windows server.....Obviously not so much for FreeNAS. Heres the question the card is supported on Red hat and SUSE SLES both x86 and x64.....is there a way to compile drivers for FreeNAS at all, or am i just asking for a world of pain and arent going to get very far?

 

[HoneyBadger]

Unfortunately driver support under Linux means basically zilch for FreeBSD.

10Gbps support at the moment is a bit spotty, the only fully supported cards are the ChelsIO Terminator series. The Intel X520/X540 are having driver issues at the moment (they'll just quit responding, apparently) so they're not recommended.

Your only other option, assuming you've got the rest of the use-case for it, is a quad-port Intel NIC and iSCSI MPIO.

 

[Dennis.kulmosen]

The X520/X540 should have been solved under 9.2.1.8, but i havent had the chance to test that yet. Plus as I can figure out from the bug report its mostly when running NFS and it got tricked by using MTU 9000.
The Chelsio would be the safest to go with until X520/X540 has been reported ok in real life tests.