Archive - Recommendations

[stephenb]

Hello,

As the titles states I am building a Archive NAS for important video footage.
The hardware used to run Open-E (500GBx16 use 3ware RAID10).

I've upgraded it in regards to drives and memory to the following specs,
Supermicro X7DBN ( http://www.supermicro.com/products/motherboard/Xeon1333/5000P/X7DBN.cfm )
Two Dual-Core Xeon's (5150 I think http://ark.intel.com/products/27218/ )
32GB RAM (FB-DIMM)
16x3TB WD Red Drives
3Ware 9650se-16ML (latest LSI/3Ware firmware)
2GB USB Stick (Its what ran the Open-E software)

I am unfortunately in an environment that will be entirely Windows so CIFS/SAMBA poor performance somewhat expected.
I've started out with 4, Four Drive RAIDZ1's in my ZFS pool. ~30TB storage.
I am looking for decent write speed and reliability is a must.
To overcome the issue of drive failure not being seen immediately (because of the 3ware controller) I did setup SMART scans (Offline test) each hour.

So where I sit, I've seen some transfers go well, upwards of 70MBs and others go very poorly, start at several KB and climbing to less than 10MBs at best. This is between my FreeNAS server with the two Gig NICs teamed (LACP) and another windows 2008R2 server also two NICs bonded (LACP). iperf give me ~1.00GByte results between these machines. I've read about a ZIL but was hoping after maxing out the expensive RAM I wouldn't need the extra expense of a SSD.

Suggestions, Ideas?
Thanks ahead of time.
Flame on.

 

[cyberjock]

Flaming commencing..err.. uhh... my suggestions:

1. Get rid of your vdevs and do 2 vdevs of RAIDZ2. RAIDZ1 is great in theory, but if 1 disk fails and the other had bad sectors you have no other redundancy. Instead opt for a RAIDZ2. Then if you have 1 disk fail and another has bad sectors you are still okay. Technically RAID5(aka RAIDZ1) died back in 2009. If you don't understand what I mean feel free to google it. It's about the probability that if 1 disk is bad another will have an error. It's virtually guaranteed with today's drives. Sucks, I know, but that's life. Plus with this combination you won't lose any extra disk space. If you really feel like taking chances and plan to stay on top of things with daily SMART emails, regular scrubs, etc you could do a 16 drive RAIDZ3.
2. Get rid of LACP. 99% of users don't know how it works, don't know that it doesn't work like they think(They're hoping for 2Gb/sec bandwidth, and then set it up improperly resulting in performance issues. If your a business user with more than 10 users then great and keep it, but if not then get rid of it.
3. I have the 24 port version of that card in a FreeNAS server. I recommend you do scrubs bi-weekly and setup the SMART emails like http://forums.freenas.org/showthread.php?6211-Setup-SMART-Reporting-via-email but obtain data from your 3ware drives. Shouldn't require much work to make the script I wrote work with 3ware. I could probably find the script that that server uses if you really want it.
4. I'm not sure which type of SMART test you are running, but if you do the smart poll every 30 mins when it fails you'll get a warning email, so no need to do tests. Additionally if parameters start changing you'll get warning emails. My recommendation would be to do long tests once a month. My schedule is scrubs on 1st and 15th of every month and SMART long tests on the 25th. Then you never have a scrub and long test running at the same time.
5. Do JBOD mode and do NOT do a bunch of single disk RAID0s. Its undocumented as far as I know, but I think CTRL+J allows you to do JBOD mode. Those keyboard buttons may be wrong, so don't quote me on that. Google may help you find the correct key combo if I have it wrong.
6. I'd swap that 2GB stick for a 4GB+ stick. Some 2GB sticks aren't really 2GB and 9.1 will require a 4GB stick anyway, so why not fix it now while you're still in the setting things up stage.
7. Keep a spare drive on your shelf just in case a drive fails. When you get it first do a long SMART test and if there's no problems stick it in a corner and never touch it again until you'll need it. You can thank me for this tip later. ;)