How many people here are taking SSD and FreeNAS to the max?

[elementalwindx]

I am thinking of doing a $1,500-2,000~ freenas box.

The hardware I'd like to use is for storage 10 Intel 120GB SSD drives, and for the FreeNas OS a 120GB SSD for itself, 24GB of DDR3, and achieve 4-8 nics teamed together.

Idea is to get as much throughput out to as many computers as possible while providing the most storage possible with some redundancy. I have an all SSD network and I love being able to get 130MB/sec to every single computer at any given time, even if almost all of them are demanding at the same time.


How many people have done a setup similar to this? Is the 24GB of memory over kill? What processor would you commend? Is an i5 or even i7 over kill? Is the 120GB SSD for FreeNAS over kill?


I'm very new to freenas, but when I like to do it, I like to do it big :)

 

[Joshua Parker Ruehlig]

I am thinking of doing a $1,500-2,000~ freenas box.

The hardware I'd like to use is for storage 10 Intel 120GB SSD drives, and for the FreeNas OS a 120GB SSD for itself, 24GB of DDR3, and achieve 4-8 nics teamed together.

Idea is to get as much throughput out to as many computers as possible while providing the most storage possible with some redundancy. I have an all SSD network and I love being able to get 130MB/sec to every single computer at any given time, even if almost all of them are demanding at the same time.


How many people have done a setup similar to this? Is the 24GB of memory over kill? What processor would you commend? Is an i5 or even i7 over kill? Is the 120GB SSD for FreeNAS over kill?


I'm very new to freenas, but when I like to do it, I like to do it big :)

INFO

If your gonna team NICs for performance make sure you have a switch that supports LACP.

24GB Ram isn't overkill. The more the better. You might as well put 8GB sticks in every slot cause they're fairly inexpensive.

What file protocol are you planning to use? CIFS is single threaded and wouldn't perform better on the hyperthreaded i7 compared to an i5, though other protocols might take advantage of the extra cores.

The SSD for the OS is a complete waste. FreeNAS is read from it's boot drive once, then put into ram. The boot drive is then made mostly read only, except for config changes, and possibly logs. FreeNAS is a 2GB image that is designed to be put onto a USB drive.

OPINIONS

10 SSDs is a bit overkill, I believe 10*fast harddrives in RAIDZ2 would be able to keep up with your network. It would be about the speed of 8 harddrives so you can do the math if that's fast enough. I guess I assume this because network bottlenecks is usually what people hit first but I'm not an expert on surpassing this barrier.

An expert you might want to look into is netflix, there open connect boxes use freebsd 9, and 36*3TB harddrive to deliver content. They use a dual 10Gb/s NIC but said they can only use 1 of the 2 ports currently for some reason, but that is enough for them. They use nginx (a webserver) to serve content cause http protocol is faster then networked file systems, and it makes sense for their implementation.
http://www.zdnet.com/blog/datacente...modity-hardware-and-open-source-software/1409

RECOMMENDATIONS - if I wanted to spend that much
* get a motherboard with enough PCIe lanes for your needs, more ram slots the better
* max out your ram for ARC caching (reads from ram instead of much slower hdd / ssd)
* get an i5
* get 2 dual port intel nics (get the ones that use 4 PCIe lanes) - I usually find these for about $40 used on ebay
* get a sata card with plenty of bandwith, I have the rocketraid2720 (I got for $90 on ebay) which is supported in FreeNAS 8.3, but alot of people have been getting the new IBM ones lately which I think has slightly better drivers in FreeNAS 8.3 (though I don't know)
This is a good resource - http://blog.zorinaq.com/?e=10
* get fast harddrives instead of SSDs. More storage for your money and shouldn't be a bottleneck
* get a switch that supports LACP (this might cost ya, I guess craigslist is your best bet, though I don't know)
* get an intel 313 SSD for ZIL, this basically helps ZFS perform better on writes because ZFS doesn't need to store it's temporary writes on disk
* get a 4GB usb stick for the OS install, install FreeNAS 8.3 64bit on there.

I use RAIDZ2 but you might want to use pooled mirrors if you want more speed+redundancy at the cost of some storage space.
I have a NORCO-470 which are under $100 and can have large fans directly on the harddrives

---

If you end up doing this, please show off your setup. I'd be really interested in seeing the performance results.

 

[elementalwindx]

I benchmarked that rocketraid vs the lsi megaraid 9265 on 4 intel 120gb ssd's, and the rocketraid was a horrid disappointment. 1.5Gbps on the rocketraid vs almost 4Gbps on the megaraid. I just recently found nas4free and I was thinking of trying this mb: http://www.newegg.com/Product/Product.aspx?Item=N82E16813131714 with nas4free.

I will definitely do this. 32GB DDR3, i5 quad core cpu, 10 data 120gb ssd drives, 2 smaller zil ssds (30~gb), 6 gigabit nics, 2 lsi 9211-8i's, 1,000watt platinum plus psu, cooler master case with scythe external 5.25 bays to house the drives. That is the plan so far.

I will come back with benchmarks when I do. I love benchmarking hardware that most people won't invest in. :)

 

[Joshua Parker Ruehlig]

I benchmarked that rocketraid vs the lsi megaraid 9265 on 4 intel 120gb ssd's, and the rocketraid was a horrid disappointment. 1.5Gbps on the rocketraid vs almost 4Gbps on the megaraid. I just recently found nas4free and I was thinking of trying this mb: http://www.newegg.com/Product/Product.aspx?Item=N82E16813131714 with nas4free.

I will definitely do this. 32GB DDR3, i5 quad core cpu, 10 data 120gb ssd drives, 2 smaller zil ssds (30~gb), 6 gigabit nics, 2 lsi 9211-8i's, 1,000watt platinum plus psu, cooler master case with scythe external 5.25 bays to house the drives. That is the plan so far.

I will come back with benchmarks when I do. I love benchmarking hardware that most people won't invest in. :)

What OS did you benchmark the RocketRAID 2720?

Any reason your getting 2 ZIL ssd's? If your using FreeBSD 8.3 or above (FreeNAS 8.3 or nas4free) mirrors aren't of use for ZIL because the ZIL is removable and a faulted ZIL won't cause any data loss.

What kind of switch do you have?

 

[elementalwindx]

I must have read wrong somewhere that if you lose your ZIL you lose the whole zpool and all your data. That was the reason for getting 2 drives to mirror the zil.

The OS was Windows SBS 2011 STD
Switch: http://www.newegg.com/Product/Product.aspx?Item=N82E16833156315

 

[Joshua Parker Ruehlig]

I must have read wrong somewhere that if you lose your ZIL you lose the whole zpool and all your data. That was the reason for getting 2 drives to mirror the zil.

The OS was Windows SBS 2011 STD
Switch: http://www.newegg.com/Product/Product.aspx?Item=N82E16833156315

Yeah it's true for zpool version 18 and below. Verson 19 allows a removable ZIL, so if the ZIL goes offline you just have a drop in performance until you online a new one instead of losing any data. Mirrors were suggested for zpool versions before 19 cause if you lost the ZIL you lost everything.

 

[elementalwindx]

awesome!! :)

 

[paleoN]

Verson 19 allows a removable ZIL, so if the ZIL goes offline you just have a drop in performance until you online a new one instead of losing any data. Mirrors were suggested for zpool versions before 19 cause if you lost the ZIL you lost everything.

Actually, you misunderstood this. Even in Version 19 losing the ZIL will still result in data loss. You would lose the data that's in the ZIL itself which is a marked improvement over losing the whole pool.

For the sake of argument let's say the ZIL contains the last 10 seconds of writes. If you were writing files to the NAS over NFS, e.g. the end of a 2GB file is lost along with a bunch of smaller files. The 2GB file would be corrupt and the small files wouldn't be there at all. This all becomes more serious if you are using a database, iSCSI or using it to back VMs. Which can result in corrupted database, corrupted filesystems and inconsistent VMs.

The recommendation still is to mirror the ZIL.

I just recently found nas4free and I was thinking of trying this mb: http://www.newegg.com/Product/Product.aspx?Item=N82E16813131714 with nas4free.

Then you are in the wrong place, http://forums.nas4free.org/. You do know that nas4free is based on a different version of FreeBSD which means different hardware/driver support.