My FreeNAS HW build vol.2

[HolyK]

EDIT1: Build done, details bellow
EDIT2: Replaced single 1TB green by two REDs. Specs bellow updated.
EDIT3: Replaced MoBo, RAM and CPU for ECC/AES-NI support

Hi all,

this is second attempt to build my home NAS. First one was destroyed by Thailand flood where the disk prices gets sky-high

Requirements:
- Data storage with resistance to disk failure => RAIDZ2
- Scheduled data backup
- Rsync
- CIFS, NFS
- Transmision
- Maybe MySQL & Apache (or something else for web services, just for own usage, nothing big)

Network - Home network is managed by Routerboard RB450G.

MoBo - SuperMicro X10SL7-F- ECC up to 32GB, 6x SATA (2+4) + 8x SAS2, 2x Intel NIC, IPMI, 2x PCIe
CPU - Intel Core i3-4340 - 3.6GHz, two cores, ECC, AES-NI, max TDP 54W
RAM - Kingston 16GB (2 x 8GB), ECC - 16GB ECC RAM ... supported by Supermicro above (another 16GB later, if needed ^^)
Disks - 6x Western Digital WD Green WD20EZRX 2TB in RAID-Z2 (encrypted) + 2x Western Digital WD Red WD20EFRX 2TB as Mirror for Plugins (encrypted)
PSU - Enermax ErPRO80+ 350W => 350W is enough
Case - Fractal Design Define R4 => Like a castle
UPS - Eaton Ellipse ECO 800USB FR, 800VA => Full support by NUT
OS Flash - Pretec i-Disk Elite E01 8GB
FANs - 5x Silent Series R2 140mm
Thermal paste - ARCTIC MX2 Thermal Paste

I will appreciate any usefully comments, warnings, replies ...

Thanks a lot :]

Holy

 

[jgreco]

I was thinking the power supply looked a little shy (12V @ 17A) when you figure 2A to spin 7 drives, 14A, plus all the rest of the system. But it has two 17A rails, I'm just blind this morning. My suggestion would be to see if you can segregate some of the disks onto each 12V rail though.

You've obviously been hanging around, reading, and paying attention, so there's a great chance of success here. I mean, you seem to know it's a good idea to start at 16GB and to do it with two higher density modules, leaving room for expansion. And not using the onboard ethernet, getting an Intel instead. Good. Board looks like it'd be nice, MSI's been around awhile and AFAIK has a good rep, nice featureset on the board. The only problem with the G630T ... and I'm guessing you know this full well, based on your past build attempt ... you're locked in at that cruddy 2.3GHz, meaning lower speeds if you're doing Samba based transfers.

But I'm going to make one more point, here. You've WAITED a long while to do this, presumably due to disk costs. No need to explain that. But why are you doing 2TB drives?

Here's the basic math of it all. 6 2TB drives in RAIDZ2 works out to about 8TB usable space. Your average drive takes about 8 watts, so let's say that this takes 48 watts for the disks. It also costs $109 per disk.

But your motherboard, CPU, RAM, PSU, and other parts probably total up more than $400. So here's the thing. Your total storage system, seven disks plus base system, is around $1200, or, put differently, 8TB for $1200.

Now put together that same system with 6 3TB drives in RAIDZ2, $139 per disk, for around $1380, or put differently, 12TB for $1380.

Your 2TB box? $150 per TB. My suggested modification? $115 per TB.

Of course, that's not the only potential change, but my point is, look at the options through this lens. You could also go to 5 3TB drives in RAIDZ2, cost of $1241, end up with an extra TB, also end up with fewer watts.

Either way, you either save watts and gain a little more space for a trivial extra few dollars, or you get FIFTY PERCENT more space for less than $200 more.

 

[HolyK]

Hi and thank you for the reply ...

about CPU - Yea, but since i will not use Samba that often, i guess i can live with that ... but yea, i am still thinking about G860 and then underclock it a bit to get that 40W at least :]

about DISC - Right now i am a bit limited by my budget, so getting 6x3TB is way over that, but yea, you are not the first one who suggested 3TB drives and i have this on mi mind. About that "5 drives" ... RAIDZ2 should have (2^n)+1, so 4,6,10,..., or did i missed something ?

About the drive spinning ... most energy is need for spinning disks up from sleep, but i guess "stagged spin up" should be well known function for any controller, so i think that PSU will handle it just fine.

Holy

 

[jgreco]

The good news about the CPU is that generally most modern CPU's have worked pretty well anyways, so even the G630T is probably just fine.

RAIDZ2 should have 4, 6, or 10 drives for optimum performance. However, you're already doing one less-than-optimal thing in terms of CPU. It isn't a "you must use 4, 6, or 10." It's 4, 6, or 10 for optimum performance. If you're going the G630T route, you might deem a savings of watts from fewer spindles to make sense.

I presented the economic argument in favor of 3TB drives because it is so often that people forget to factor in the total expense of the system into the equation. It's a high fixed cost item. For our 1U Xeon boxes here, the cost of the server actually overpowers the equation; the chassis is around $700 IIRC, so if you have 4x$109+$700=$1136 for 4TB RAIDZ2 ($314/TB), or 4x$139+$700=$1256 for 6TB RAIDZ2 ($209/TB), or 4x$229+$700=$1616 for 8TB RAIDZ2 ($202/TB), the interesting bit is that the hellishly more expensive 4TB drives are STILL CHEAPER on a per-terabyte basis. Also fewer spinny drives for a given number of TB means fewer watts.

As for the drive start current, just be *aware* of it and do what you can to not heap everything on one rail and you'll be just fine.

Anyways, you asked for input, that's mine.

 

[cyberjock]

I thought I had written something but now I see no post. I'm not a fan of Pentiums or Celerons. The important thing to keep in mind is that the wattage given is the maximum wattage at full load. And while Intel may rate your Pentium for 35watts, they actually are providing a value that they promise the CPU won't exceed. That may change from batch to batch too! Also, your Pentium may draw 10w idle but if an i3 draws 12w idle which would you go with? When heavy load time comes(like a scrub) one of those will smoke the other performance wise. Those wattages are just made up to prove my point. You'd have to look around to see if anyone actually provides a meaningful comparison of idle load between CPUs to see how much the increase is.

I'd just rather have some reserve CPU capacity that can kick in on request than buy some underpowered CPU that works but won't let you expand later without an upgrade. Who knows what plugins may be available in 2 years that you'll wish you could use. I know that if Plex Media Server comes out for FreeBSD I'll have a jail setup and installed running the alpha releases.

 

[HolyK]

2 jgreco: G860 is cheaper than G630T, at least here ^^ So yea, i will probably take it. My guess is that i3 is overkill for my need. I will be using mostly the iSCSI, UFS will be probably used by my GF, but she sits on Wireless, so there is no need to get max form Samba since the bottleneck will be the "air" :D

2 noobsauce80: Yea, you right about the min/max power consume. I'll try to search some comparison or something like that :]

 

[paleoN]

- Maybe MySQL & Apache (guide)

Do you need Apache? I believe there are a couple how-tos floating around on setting up MySQL & NGINX in the jail.

RAM - CORSAIR XMS3 16GB (2 x 8GB) => another 16GB later, if needed ^^

Have you considered ECC RAM? It's the one thing, well it changes a few others, I would do differently if I had to do it over again. Read this [post=30179]post by survive[/post].

Disks -
6x Western Digital WD Green WD20EZRX 2TB ==> RAID-Z2
1x Western Digital WD Green WD10EARS 1TB ==> UFS for Transmission

I would buy an additional e.g. 2TB disk for a cold spare. You could even use it for the 1TB if it starts acting up.

OS Flash - Kingston DataTraveler Micro 8GB

Pick up a few.

will be using mostly the iSCSI,

In what capacity?

 

[HolyK]

Do you need Apache? I believe there are a couple how-tos floating around on setting up MySQL & NGINX in the jail.

Not necessarily apache, so thanks for the NGINX idea. ill take a look on it.

Have you considered ECC RAM? It's the one thing, well it changes a few others, I would do differently if I had to do it over again. Read this [post=30179]post by survive[/post].

I will read your post, for now i can just say that i don't think that there will be MB for $100 which will support ECC :(, also price for the ECC memory and i3 cpu :/

Pick up a few.

:D :D , yea, why not ... i have like three flashes here bud none of them is "lowprofile" :/

In what capacity?

Don't know exactly now, but like 70% of the raidz i guess... maybe more

 

[cyberjock]

When paleoN was asking in what capacity I think he meant "run a VM for developer programs", "run a MYSQl database for customer information for 10k+ customers" etc.

I think what he was leading into is that iscsi doesn't always go really well with ZFS. Jgreco has a support ticket discussing this and how(with alot of experimenting and testing with tweaks) you can solve a lot of those drawbacks. If you don't want to do the tweaking and experimenting you should consider UFS. Those drawbacks mostly go away.

 

[HolyK]

Oh, i see this post and i guess this is what you mentioned. I understand what Jgreco was talking about in that topic and it sounds logical that the block-level protocol on copy-on-write system is not a good combo, Well, let say CIFS on ZFS and iSCSI on CFS are the right combinations.
That NAS will be mostly only for me (backups, snapshots, rsync, archives of RAW files (video/photo), etc... combined data storage), then some kind of web service (nothing huge), above mentioned transmission, etc....

I did not had a time yet to think more about "how much" "how" "for what" and "why". I'm still trying to get as much information as i can, so thank you for pointing on this problem.

Now is the time to reconsider that CPU once again. It's look that i will need to stick with CIFS mostly, because i really would like to keep ZFS, at least for some data which are important for me.

Srry for typos, 2:20AM here :(

Holy

 

[paleoN]

When paleoN was asking in what capacity I think he meant "run a VM for developer programs", "run a MYSQl database for customer information for 10k+ customers" etc.

Entirely.

However:

Don't know exactly now, but like 70% of the raidz i guess... maybe more

It's not recommended to fill the pool more than 80% including any iSCSI volume. 70% to the iSCSI would only give you 10% for everything else. Not to mention depending on how you are using iSCSI, jgreco has recommended keeping even greater amount of free space, around 60% I believe, to allow iSCSI to actual be performant.

 

[HolyK]

Sorry, typo, i meant to say 40%, i have something around 2.4TB for backup, which mean 30%. This must go on RAIDZ2 to make sure that i will not lose this.
I understand the reason for that free space.

BTW: My prev reply was "approved" after your post, please see this :]

 

[Tysken]

I just wanted to recommend this motherboard.
ASRock B75 Pro3
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157314
It has 8 sata ports that are supported by freenas and is cheap. I have been using that board a couple of month now and it works great. The only drawback is that the onboard nic is supported but is a bit slow so i use a intel nic instead.

 

[HolyK]

2 Tysken: Interesting , but the additional two SATAIII ports are handled by ASMedia chipset, which i don't like :/

- 2 x SATA3 6.0 Gb/s connectors by ASMedia ASM1061, support NCQ, AHCI and Hot Plug functions

 

[HolyK]

Hmmm ... once again about the CPU ... Intel released new low-cost Dual-Core Ivy Bridge ...

Intel® Core™ i3-3210 Processor (3M Cache, 3.20 GHz)
vs.
Intel® Pentium® Processor G2020 (3M Cache, 2.90 GHz)

http://ark.intel.com/compare/71053,71070


I guess that the G2020 is best buy. 300Mhz difference does not matter i think, also graphical core does not give a sh!t. What about HT? Will HT improve performance significantly or not? And the rest (Anti-Theft Technology and My WiFi Technology .... i did not even know that these things exits :D )

Most wonderful is the PRICE ... its almost half of the i3 !!!

 

[HolyK]

Following review is for the OLD system specs liste bellow !!!

Ok, Baby is home ^^ ... Here is my review :]

Hardware:
MoBo - MSI ZH77A-G43 LGA 1155 Intel H77
SATA Controller - PESA230 (Marvell 88SE9128) => For that 1TB disc
CPU - Intel® Pentium® Processor G2020 (3M Cache, 2.90 GHz) LGA 1155 55W
RAM - CORSAIR XMS3 16GB (2 x 8GB)
Disks -
6x Western Digital WD Green WD30EZRX-00DC0B 3TB in RAID-Z2
1x Western Digital WD Green WD10EARS 1TB as single UFS
PSU - Enermax ErPRO80+ 350W
NIC - Intel EXPI9301CT
Case - Fractal Design Define R4 => Like a castle
UPS - Eaton Ellipse ECO 800USB FR, 800VA => Full support by NUT
OS Flash - Pretec i-Disk Elite E01 8GB
FANs - 5x Silent Series R2 140mm
Thermal paste - ARCTIC MX2 Thermal Paste

Sw: FreeNAS 8.3.0 p1

RAID-Z2 performance:

dd if=/dev/zero of=/mnt/st0rage/test.tmp bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 290.185959 secs (370018531 bytes/sec)

dd if=/mnt/st0rage/test.tmp of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 259.444886 secs (413861241 bytes/sec)

Network transfer over CIFS (Samba)
This is NOT burst speed on the begining of the transfer, i have 100MB/s during whole transfer.

I am still playing with parameters, so maybe i will be able to squeeze it a bit more to achieve 110 ++ :] Anyway the speed is amazing!

Mapped as network storage ... love the numbers :D

"Before" and "After" ^^

Note: I did not moved yet all of the things from that 1TB HDD which sits in my PC, so that's the reason why there is only 6 disks and Kouwell is not occupied.

Network is managed by RB450G where the NAS and workstation port are on HW switch chipset, so there is no impact on RB CPU.

Yesterday i was moving like 500GB to test speeds. Here are some graphs from webgui. Transfer was started at 19:30 and ended few min after 21:00. So half of the TB was moved in 90min, which is goood i guess :]

Samba is nasty, but G2020 is powerful enough ^^

ZFS memory utilization ... yea, like that !

SWAP was never touched ^^

Network traffic ... i was working on my PC, so the average speed was 88MB/s since i was searching something on the net and downloading few things. So the reason why not 100MB/s is not the NAS

Still lot of things to do, but now its nice and i like it !