Questions about first build

[Jkuz]

Hello all! I am thoroughly a newbie at this so please forgive me if I make a really stupid mistake. I am looking to put together a FreeNAS build for my company and would love to help and guidance.



Backstory: I joined a small development company and 2 months in the IT Admin decided to up and leave the company. Before he left us, he blessed us with 2 QNAP NAS boxes which he moved all of storage from our server’s internal drives to these NAS boxes. The QNAP boxes each are 4 bays and the one is running all the servers and the other box is a replication of the first box. We have 17 VMs running over 4 physical servers and the datastores are all on this one NAS box with 4 7200 RPM SATA drives going over a 1Gbps connection. (See NASCrap.jpg for a visual representation.) I don’t know if you guessed it but the whole system is HORRIFICLY SLOW! An octogenarian snail moves quicker than our servers now. As well, it’s not terribly secure since we can only really have one drive die for each server at a time. We do have a backup system set up that gets copied to an external drive but overall the system is slow and unsafe.



So since I have the most IT experience at the company, (I worked in the IT department at my university where I just graduated from) I got “promoted” to IT admin. Now, I am trying to fix all this crap we have going on here.



Just before the previous guy left he installed a Dell PowerEdge R320 and retired a Dell PowerEdge 2900. Trying to figure out how to speed everything up I started investigating FreeNAS. I want to put FreeNAS on the 2900. Just to make sure I’m clear. FreeNAS would be installed on the bare metal and not as a VM. I would be setting up the system to share the volumes with other ESXi hosts to run the VMs on.



Here is my planned configuration:

• Dell PowerEdge 2900
• CPU: 2 Intel Xeon 5000x
• RAM: 16GB DDR2 Kingston Dual Channel ECC (KTD-WS667/4G)
• NIC: Two Broadcom NetXtreme II 5708 Gigabit
• OS: USB FreeNAS install
• Storage: 10 SATA Bays

My thought is to have 2 very small SSDs installed and have one act as the ZIL and one act as the L2ARC. Then I will have 8 bays with 500GB 7.2k SATA HDDs. I would like some input on whether I should run RAIDZ3 or RAID1+0. I have found online that RAIDZ3 is not necessary for small drives like our 500GB but the allowance for 3 drives to fail is appealing. We are using Oracle as a development DB which does have more small reads and writes than the average media server or personal use server. I would like to put more RAM into the machine but that is currently still being decided. Would this setup really suffer with only 16GB?



I have read CyberJock’s powerpoint and studied the section about ZIL and L2ARC. Because we are hosting multiple VMs through ESXi I am assuming that we will want/need both a ZIL and a L2ARC device. Maybe someone can fill me in about that more?





Overall, is this a good setup?



Again sorry if I am asking dumb questions, I read through a bunch of posts and such before posting.

 

[Jkuz]

A response would help. Even what I'm doing wrong....

 

[krikboh]

There are multiple issues with using that Dell server with FreeNAS.

1. The FSB architecture is very slow with any ZFS filesystem. The high amount of data passing through the RAM gets very slow with FSB.
2. Most PERC controllers in Dells do not play well with FreeNAS.
3. You will need more RAM. You should have a minimum of 64 GB of RAM if you plan to run multiple VM's with ZIL and L2ARC.

 

[Jkuz]

1) Ok. Well there's not too much about that I can change :/
2) I have the Hardware RAID disabled but I guess even just the PERC controller interfacing with the motherboard is still a problem?
3) The highest this board supports is 48GB so I guess I'm stuck on that front as well. And getting RAM for this old computer is a little difficult.

Well, that's not great news for this project.

 

[krikboh]

Yeah, even if FreeNAS detected the PERC controller and the attached drives there is a good chance SMART info would not be passed to the OS. You would be at risk of silent drive failures.

The experts on the forums probably saw where this was going and that is why you didn't get a previous response. FreeNAS could still be a good solution for you if you can purchase new hardware or maybe some newer used hardware. Take some time and read the hardware stickies.

https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/

 

[Jkuz]

Ok thanks for the info.

I didn't think I egregiously broke any of the forum rules and read a lot before posting, kind of frustrated a the lack of response.

This is a little disheartening. We don't have a ton of money to throw at new systems since we installed this horrible NAS system... Rock and a hard place. I'll keep reading through stuff here and see if anything works for me.

 

[DrKK]

The setup/use-case you describe is something you can make happen with FreeNAS (or, especially, the commercial solutions iXsystems sells), but you'll need to spend some cabbage on it, no matter how you slice it. With all these VM's and whatnot, any budget-constrained NAS build will produce subpar performance. You really need a well-provisioned system for a project like this...not "have-to-make-do" hardware.

 

[DrKK]

Ok thanks for the info.

This is a little disheartening. I didn't think I egregiously broke any of the forum rules and read a lot before posting, kind of frustrated a the lack of response.

We don't have a ton of money to throw at new systems since we installed this horrible NAS system... Rock and a hard place. I'll keep reading through stuff here and see if anything works for me.

Sir, you are "disheartened" that you didn't get a response since dinnertime last night?

Come on.

 

[Jkuz]

Sir, you are "disheartened" that you didn't get a response since dinnertime last night?

Come on.

Calm down there DrKK. I put the disheartened bit in the wrong place in my post. I am very upset that the previous guy did this crap and made it all so terrible and now I'm left picking up the pieces. Is it unrealistic to expect someone perfectly explain how to set up my system? Yes. But even something as simple as what Kirkboh put would have been and is very helpful.

 

[mjws00]

Have you tested the speeds the qnap boxes are giving you? Are you sure that you aren't bound by 1Gb ethernet vs the storage? Assuming some VM's are more IO bound than others. Why not use hardware raid and some local ssds for those? A xeon with 8 drives and 48GB should outperform any 4 bay qnap. But your diagrams show very little actual volume necessary. Make certain you address the actual bottlenecks and don't just fire up a new FreeNAS toy. Don't even consider anything but striped mirrors with zfs. A small L2Arc may still be worthwhile, it is about balance. Not rules of thumb.

Fire it up and TEST. Obviously better hardware would be faster, and even a fast E3 might smoke that old girl... but in the end the wire is likely your bottleneck. MPIO, local drives, DAS, 10Gbe all address that. It is possible the Qnap is the issue... but there is likely more going on.

 

[Jkuz]

Have you tested the speeds the qnap boxes are giving you? Are you sure that you aren't bound by 1Gb ethernet vs the storage? Assuming some VM's are more IO bound than others. Why not use hardware raid and some local ssds for those? A xeon with 8 drives and 48GB should outperform any 4 bay qnap. But your diagrams show very little actual volume necessary. Make certain you address the actual bottlenecks and don't just fire up a new FreeNAS toy. Don't even consider anything but striped mirrors with zfs. A small L2Arc may still be worthwhile, it is about balance. Not rules of thumb.

Fire it up and TEST. Obviously better hardware would be faster, and even a fast E3 might smoke that old girl... but in the end the wire is likely your bottleneck. MPIO, local drives, DAS, 10Gbe all address that. It is possible the Qnap is the issue... but there is likely more going on.

Yeah, I'm pretty sure that the NIC is the bottleneck with those QNAP boxes. They do have dual 1Gbps connections but I'm worried the IO bottleneck stems all the way from the spindles to the NIC. Also these aren't terribly great QNAP boxes to begin with. You're right that we're not pushing very much data through but the big thing I'm seeing is that Oracle isn't very happy being delegated to a NAS and is killing read and write speeds due to the frequent small reads and writes. Our biggest problem in in our loading from the database it is very slow.

I think I have enough information on this setup at this point. I'm going to keep playing with all the different points in my network and FreeNAS was never the whole solution to the problem but I see I need to probably explore different avenues, at least different hardware. You can let this thread die unless someone has a new revelation for me. Thanks for all your help!