Preliminary Hardware Outline..thoughts?

[trscho]

i there, I've been spending some time coming up with a future FreeNAS build.
I've been trying to balance these 3 key fundamental features..
-Low Power Consumption
-Long Term Expansion Viability (Optimum Possible HDD/RAM Expansion)
-Ram Speed Optimization (2133mhz, and 1GB Ram : 1TB HDD Rule)

This box will be designed for home use, and will not be accessed by more than 2-4 devices (including tablets/phones)
If possible, I hope to use this in the following ways:
-file storage
-audio/video streaming
-to act as an install pathway for games on my desktop PC that will provide minimal loading delays (some is acceptable)....graphical processing will be done by the desktop pc

Here is the initial structure I was thinking of using:
-------------------
Athlon II X2 270u -25watt (before underclocking)
MSI 970A-G46 -Regulated Power Usage http://www.msi.com/product/mb/970A-G46.html
1x 8GB 2133mhz Ram
Antec 1200 v3
Any PSU -I have many spares
--- --- ---
Total Build: 400$~

My planned expansion would go up to the following specs:
32GB of Ram
27TB-32TBGB of HDD Memory (respecting the 1:1 Ram:Memory ratio rule, using a mix of 3 and 4tb hard drives)

Can you think of any problems I might encounter with this build?
Does anyone know a better/more power efficient A3+ socket motherboard (without sacrificing ram performance)?
Is ram speed important for freenas, or would I be able to acheive my functional goals with lower end RAM?
Will this processor be sufficient for what I'm hoping to use it for?
Are A3 processors compatible with A3+ Sockets?

Things to bare in mind:
-I will eventually ultimately use a SATA controller card (which means any motherboard choice must be able to support this)
-Motherboard must be able to support at least 32gb of ram (i will be aiming to use 4x2TB, and 8x3TB = 32TB) minimum; I couldn't find a motherboard that supported A3 processors and higher RAM.

 

[DrZed]

Hi there, I am also interested in some hardware selection directions.

I would like to do the same as you, but filled in a mini-ITX board in order to put 6 disks max in the Fractal Design Array R2 300W SFX PSU Case like done here:
http://forums.freenas.org/showthread.php?1310-My-new-NAS-box-running-on-8-0-2&highlight=torrent

I did some searches and found this motherboard that seems right:
http://www.anandtech.com/show/4595/zotac-z68itxae-wifi-review-miniitx-meets-z68/1

Plan is to remove the Wifi adapter and plug an SSD for system caching in place.
Then use 2x SATA 6GBs on-board for 2 disks, and plug a SATA 6Gbps extension card on the PCIE port for the other 4 disks.
Do you know which SATA extension card I could use (cheap and efficient ^^) ?
EDIT: I have found this one that seems nice, what do you think? http://www.lsi.com/channel/france/products/storagecomponents/Pages/MegaRAIDSAS9240-4i.aspx

Do you think this build would do the job?

Thank you in advance for any direction ;)

 

[aaronb]

Hardware thoughts...

I can answer a few questions, but some I leave to you to search on-line.

After playing with a few different builds and reading a lot of places on the forums:

1. CPU Speed is critical for CIFS (Windows) shares. SAMBA is single threaded, and you will get performance with a faster processor as opposed to multiple cores. I confirmed this for myself when changing processors. My quad-core 2.2Ghz processor dropped my throughput to about 45 Mbps as opposed to my dual-core 3.0 Ghz processor running at about 70 Mbps. There are other factors involved - but processor speed is important here.

2. As for HDD controllers - if you browse the forums you get a couple of answers. The 'inexpensive' solutions are to use SYBA controllers based on the SIL3124. Search newegg for parts: SY-PEX40008 (PCI Express) or SD-SATA2-4IR (PCI). If you are looking for high-performance, I believe someone in the forums suggested finding HP SAS controllers on eBay for cheap, plus requires fan-out cables.
- As an additional note for drive numbers, you can use port multipliers with the SYBA cards (and possibly others). Again, search NewEgg for them. The 5-port expander allows you to use one SATA port on your card to connect to 4 drives. Thus, you can make one of the 4-port SYBA cards support 16 drives with 4 port multiplier cards. For a performance factor, you would rather use controller cards over port multipliers.

3. Speed of RAM: You really need to ask how you are connecting to this machine. If it is over a 1 Gbps Ethernet link, that will be your limiting factor for most operations. Having plenty of RAM means better caching of the drives, thus better performance. That being said, DDR2-400 (or PC3200 as it is also known) has a throughput of 3200 MBps. That is 25600 Mbps (8*3200) or ~ 25 Gbps. Thus, DDR2-400 exceeds your 1 Gbps Ethernet link by a decent margin. I wouldn't worry too much about RAM speed for your NAS. Somewhere in the calculation it will help, but not enough for you to notice.

4. Hard drives: Remember that RAID or ZVOLs really only work properly with same model / size of drive. You will lose performance or space if you different size drives. Thus, if you are going to use different size drives, then best practice says you create volumes with only the same drive sizes. For maximum space and performance, all drives should be the same size / manufacture.

5. If you want to support more ram, look at server motherboards. You can find boards with 8 RAM sockets there, but it will cost a bit more.

 

[palmboy5]

<snip>
2. As for HDD controllers - if you browse the forums you get a couple of answers. The 'inexpensive' solutions are to use SYBA controllers based on the SIL3124. Search newegg for parts: SY-PEX40008 (PCI Express) or SD-SATA2-4IR (PCI). If you are looking for high-performance, I believe someone in the forums suggested finding HP SAS controllers on eBay for cheap, plus requires fan-out cables.
- As an additional note for drive numbers, you can use port multipliers with the SYBA cards (and possibly others). Again, search NewEgg for them. The 5-port expander allows you to use one SATA port on your card to connect to 4 drives. Thus, you can make one of the 4-port SYBA cards support 16 drives with 4 port multiplier cards. For a performance factor, you would rather use controller cards over port multipliers.
<snip>

I was running 4x HD201UI in RAIDZ1 on a PCI SiI3124 card and got horrible performance; no more than 30MB/s write. This is compared to 131MB/s off of my motherboard ports.
To me, the SiI3124 is much too slow to even consider. So, can you specify what you mean by "high-performance" versus the SiI (the last letter is an I btw, not L) 3124? What would be the threshold speed to count something as high-performance?

 

[aaronb]

I was running 4x HD201UI in RAIDZ1 on a PCI SiI3124 card and got horrible performance; no more than 30MB/s write. This is compared to 131MB/s off of my motherboard ports.
To me, the SiI3124 is much too slow to even consider. So, can you specify what you mean by "high-performance" versus the SiI (the last letter is an I btw, not L) 3124? What would be the threshold speed to count something as high-performance?

Well, I was reading off other's info in this forum:
http://forums.freenas.org/showthread.php?2177-PCI-e-SATA-expanders&highlight=SIL3124
- Look for the Post by cypher1024

As for the Silicon Image chip... Well, I took the part off of Newegg where they have it listed as a SIL3124.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816124027&Tpk=SY-PEX40008

Common question on this forums is 'performance'. Look at the path for performance and figure out what is holding you up.
One question I have is how did you measure your write performance? CIFS? NFS? iSCSI? Direct write from OS?
I wouldn't be too surprised that you would see faster writes with the mainboard ports. Especially if you didn't make sure your PCI card was not conflicting with another device. This is very dependent on the mainboard and which slot the card it in.

I have one of my friends talking about needing a high-performance NAS for a project he is working on. I had to look at him funny when they won't upgrade their primary network from the 100Mbps switches. Even 30MBps writes would exceed their LAN speed. Performance is often relative.

As for the Dell SAS controllers from Cyper1024's post - well, it is a PCIex4 slot card - thus it has a potential higher bandwidth than a PCIex1 slot or PCI card. Still depends on drives, configuration and protocols used.

If you need it to be faster yet, add a SSD drive to the zpool as it's cache drive. Let the ZFS subsystem decide what should be cached for you and see how your performance goes.

 

[palmboy5]

I was testing writing 100GB of 0's using dd with a 2048k block size. The main reason why I cannot stand using the card is because once the cache fills up (I'm guessing is what's happening), it spends the next 30+ seconds dedicated to writing what it has to the disks (as seen with zpool iostat -v 1). During that situation, NO reads or writes happen, not even on the main pool I have normally connected to the motherboard ports. So, any copy progress window related to my NAS freezes up and does not have its transfer speed updated to account for the 30+ seconds of 0 Bytes/s. I'd love to get a good CIFS speed value but I can't. Also, adding an SSD for a larger cache is only going to make the freezing period even longer as it has to write even more to the disks. Anyway, enough of me hijacking this thread... maybe I'll make a thread about my issue some time since I doubt the behavior is normal, but I did just order an M1015 card and cables so I'll likely be good to go soon regardless.

+1 on getting a CPU with good single-threaded performance. I was cursed with lousy 15-25MB/s speeds on both my Atom 330 1.6GHz and Athlon X2 2.2GHz. I kept blaming other things like the 512 byte vs 4k sector issue for my performance woes until, on a whim, I tried it on a Q6600 at 3GHz and was blown away by the fact that I was instantly getting over 110MB/s over CIFS. I immediately got an Athlon II X2 3.2GHz and accompanying motherboard and have been happy ever since. New issue being that my 4 additional drives on the SiI3124 are too slow for me to make use of yet.

 

[aaronb]

On the add-in card - try changing slots.

Not sure if you have any configuration for that card, be it BIOS or jumper...
Make sure BUS Master is on if an option - should be, since it is PCI.

How do you have the card / disks configured. JBODs, straight controller, .... ?
Also, not sure if it exists, but check to see if there is a BIOS update for the card. May require putting it in a windows box if there is one.

It looks like there may also be a firmware for the HD201UI drives - you might make sure they are up to date.
http://forums.whirlpool.net.au/archive/1597038

I've had it be just the little tweaks to get things to work. Hard to say if any of this will help.
I can say that going to the SAS controller I mentioned should help a lot - just the fact that it is PCIe x4, but couple that with SAS being mainly server oriented will help.

 

[palmboy5]

My motherboard is a GA-78LMT-S2P and only has one PCI slot to begin with, the other card I have is an Intel Gigabit PCI-e x1 card installed right above the PCI slot.

That card had bigger problems, it was causing read/write errors on all drives and caused ZFS to deem the pool degraded enough to say that "Applications may be affected". I have returned it. Went back to this card now which I have had for a couple years, its BIOS is up to date as a straight controller for ZFS. It has a SiI3114 where basically the difference is that it is SATA I and doesn't support port multiplication (still PCI). The only jumpers the card has is to choose between having two of the ports be internal or external. I get fantastic (ha. ha. ha.) speed with this card:

$ dd if=/dev/zero of=tmp.dat bs=2048k count=50k

Code:

----------  -----  -----  -----  -----  -----  -----
tank2       39.7G  7.21T      0    124      0  15.6M
  raidz1    39.7G  7.21T      0    124      0  15.6M
    ada6p2      -      -      0     62      0  5.41M
    ada7p2      -      -      0    138      0  5.43M
    ada5p2      -      -      0    122      0  10.6M
    ada4p2      -      -      0    124      0  10.7M
----------  -----  -----  -----  -----  -----  -----

107374182400 bytes transferred in 4033.099758 secs (26623240 bytes/sec)

Don't know about the BUS master thing...

The HD204UI drives are very new, less than a month old. The fixed firmware was supposed to be shipped out with any new drive since about a year ago, but I have patched three of them tonight anyway... One of the drives is the Seagate ST2000DL004 version of the HD204UI and the patch program refused to patch it. I think their patch is pretty dumb, the version number stays the same so there is no way to know if the drive is patched or not, and you can continue to patch it over and over again as a result.

Anyway, my PCI-e x8 M1015 SAS card is coming in a couple days so... whatever.

 

[palmboy5]

Ahh much better :P

$ dd if=/dev/zero of=tmp.dat bs=2048k count=50k
$ dd if=tmp.dat of=/dev/zero bs=2047k count=50k (I was paranoid about over-reading lol)

Rosewill RC-209-EX - 4x HD204UI RAIDZ1
98.0 MB/s read, 26.6 MB/s write

IBM M1015 IT-mode - 4x HD204UI RAIDZ1
355.0 MB/s read, 314.4 MB/s write

Strange that my other RAIDZ1 doesn't even hit 200MB/s...

IBM M1015 IT-mode - 2x WD20EARS, 2x WD20EADS RAIDZ1
196.3 MB/s read, 143.1 MB/s write

Is it because the WD array has less than 1TB free?

 

[cyberjock]

Well, the Rosewill FC-209-EX is a PCI card. The PCI bus(not any individual device) is limited to a theoretical maximum of 133MB/sec. I'd say your reads are about what you should expect. There's no way you will hit 133MB/sec, plus any other PCI devices will lower the total available bandwidth. Some devices built into the motherboard are also PCI devices and will lower that total.

I'm not sure why you chose a block size of 2047k. Since it's not a even number that "could" affect your values alot. You really are better off lowering the count and not the block size if you were concerned about over-reading(whatever that's supposed to mean..)

As for write speed, that l can't say. I will say I try not to buy Rosewill because they aren't exactly "high quality" components.

 

[palmboy5]

What happens if someFile is 100KB but you do the following?
$ dd if=someFile of=/dev/zero bs=101k count=1

You prompted me to actually try it now, and it looks like dd just stops at the end of the file regardless of how much larger the copy operation is compared to the file size. I was worried that dd would read past the file and start reading whatever junk exists beyond. I was also worried that the actual copied size may vary by a byte or so for any given bs and count value combination. (Did I mention I was being paranoid? ;) ) Fortunately, dd isn't that stupid.

I assume by "even" you mean divisible by the hard drive's sector size (4k in my case, I forced it upon pool creation). True that 2047k would technically mean there is one non-4k write every 512 4k writes, but that isn't even 0.2% of the time and will be way overshadowed by the margin of error between tests due to other factors. ZFS also hides this issue since it buffers the writes into transaction groups in the first place. EDIT: Actually, given the fact that ZFS writes transaction groups every 5 seconds rather than when a certain buffer size is reached, and that it adds its own data (such as checksum values), the actual transaction group write is not guaranteed to be divisible by 4k anyway. Due to this nature, ZFS might not be writing at 100% efficiency even if a proper block size was specified by dd.

Anyway, the M1015 is a bit weird with its drive numbering and LEDs... There is a row of 8 activity LEDs on the card, but from my observation, SAS port 1's 4 SATA ports correspond to the two LEDs on each end of the row. SAS port 0 corresponds to the four LEDs at the center. Also, da0-da3 are on Port 1 while da4-da7 are on Port 0. Non-intuitive to say the least.

 

[cyberjock]

You are correct about the block size being devisible by your hard drive's sector size. If they don't match then your hard drive will do a read and write operation to change the sector's data, then do a second read and write when the next block starts writing.

I wasn't sure how much it would affect the DD performance, that's why I said could and put it in quotes. :P

My guess(and I think you've probably figured it out already) is that something weird is happening with the Rosewill controller. It may be the controller, or related to the PCI bus. It's hard to say. I'd never recommend you use a PCI card unless you really don't care about performance.

 

[palmboy5]

I originally got the card for an Atom 330 board that had a single PCI slot, it wasn't meant for this NAS. But when I, once again, found the need for more SATA ports, my first card to try was what I had lying around. After that, a SATA II version of basically the same card, and then I decided to just go with a legitimate card.

 

[cyberjock]

But when I, once again, found the need for more SATA ports, my first card to try was what I had lying around.

How dare you do what's economical! Honestly, I'd be interested to know why the writes are so slow. That's just so odd! Was the performance with a 2048k block size significantly different?

 

[palmboy5]

I always used 2048k for the write tests, and now I see that I got confused in an earlier post - I was talking about doing 2047k for the reads due to fear of over-reading and immediately started talking about performance issues with writing at 2047k even though I always wrote at 2048k. Oops...

I'll see if I will bother trying the four HD204UI drives with the Rosewill card in my desktop system to rule out the NAS's motherboard, but no promises!

 

[palmboy5]

Oh man. Don't know what happened but, when trying the PCI SATA card in my desktop with the HD204UI drives connected (and desktop's drives disconnected), I killed both my main FreeNAS flash drive and a second flash drive. Like killed, killed. The FreeNAS drive doesn't respond anymore on any computer and the second drive was very hot to touch and the case is warped/melted. I took the warped drive and grabbed my IR thermometer, the drive's side was 164F/73C. Thankfully the desktop's parts all seem to still be working, including other USB devices...

 

[cyberjock]

Oh my! Sorry to hear that palmboy5. Did you build that machine? I had a friend that connected the internal USB cable wires incorrectly to the motherboard and anything he plugged into those ports got fried. Unfortunately, he burned up a keyboard, mouse, and 3 USB drives before he figured it out.

If you remember what ports you used, I highly recommend you figure out what went wrong before you use them again.