My new NAS box, running on 8.0.2

[Daisuke]

Update: I upgraded certain components of my NAS (December 1, 2011), in order to improve the overall performance.

Original Build Parts
1 x Fractal Design Array R2 300W SFX PSU Case
1 x Supermicro X7SPA-HF-D525-O Intel Atom D525 1.8GHz Motherboard
6 x Western Digital WD20EARS-00MVWB0 3-Platter 2TB Drive
1 x Patriot Xporter XT Boost PEF4GUSB 4GB USB 2.0 Flash Drive (FreeNAS)

Upgrade Parts
2 x Corsair CMSO4GX3M1A1333C9 4GB DDR3-1333 CL9-9-9-24 204PIN SODIMM Memory
1 x HighPoint Rocket 620 PCI-Express 2.0 x1 SATA III (6.0Gb/s) Controller Card
1 x Crucial M4 CT064M4SSD2 2.5" 64GB SATA III SSD
6 x Xion 18" SATA II with Metal Latch Cable
1 x Rosewill 10" SATA III with Metal Latch Cable

Hardware Details
I picked Supermicro because their well-known product reliability, as well because they are the only ones who offer dual Intel NIC's on an Atom based motherboard.
I decided to use the HighPoint Rocket 620 x1 controller because it has a Marvell 88SE9128 chip, supported out of the box by FreeBSD. Be aware that this is not a RAID controller. If you plan to use 2 disks, the controller will generate HUGE bottlenecks. However, when the controller is connected to a single drive there are no issues. In my tests, it allowed me to read data locally at 400MB/sec. I could not believe it.

Even if we are dealing with a x1 controller, FreeNAS detects the SSD disk as SATA III:

Code:

ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)

HighPoint added something in their new firmware, compared to previous version tested by FreeBSD developers. From their instructions manual (see page 5), they display instructions for 1.0.0.1003 BIOS version which matches their description how the controller works (2 AHCI channels). However, when I hit CTRL+M in my box the BIOS version shows as being 1.0.0.1012 and the firmware as 2.1.0.1404, which might explain the new "feature":

Code:

pass1 at ahcich7 bus 0 scbus7 target 0 lun 0
pass1: <Marvell 91xx Config 1.01> Removable Processor SCSI-0 device 
pass1: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)

If you have any insides related to this matter, please let us know. I attempted to contact Highpoint support without success, they never replied to my emails neither responded to my technical support ticket.

I recently found out that the X7SPA-HF-D525 supports up to 8GB of RAM, even if Supermicro advertise it at 4GB. The manual also says that 1666 MHz and 1333 MHz memory may be used, but will operate at 800 MHz. With the memory upgraded, FreeNAS is reporting 8177MB of RAM:



I also think there is a myth related to the fact that Atom processors cannot handle more than 4GB of RAM. I'm not sure if all Atom processors support the larger memory setup, or just the Atom D525 installed on my Supermicro motherboard. My system uses on a regular basis 6GB of wired RAM, which is technically impossible based on Intel specs:



Hardware Setup
The OKGear 20cm SATA 6Gbps cable I initially used to connect the SSD disk was not "OK" at all. The sleeve snapped right away when I tried to bend a little the cable and hook it into Rocket controller. I ended up using a Rosewill 10" SATA III cable instead.

The disks cage, ready for RaidZ2:



The "skinned down" USB stick that holds the OS:



I had a hell of a time to get the cables organized inside, in order to achieve a clean wiring:



The Rocket controller that adds a SATA3 port for Crucial SSD:



Disk Specs
The dmesg info related to disks (da0 USB disk, ada0 SSD disk, ada1 regular disk):

Code:

da0: < Patriot Memory PMAP> Removable Direct Access SCSI-0 device 
da0: 40.000MB/s transfers
da0: 3824MB (7831552 512 byte sectors: 255H 63S/T 487C)

ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich8 bus 0 scbus8 target 0 lun 0
ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich9 bus 0 scbus9 target 0 lun 0
ada2: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich10 bus 0 scbus10 target 0 lun 0
ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich11 bus 0 scbus11 target 0 lun 0
ada4: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich12 bus 0 scbus12 target 0 lun 0
ada5: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada6 at ahcich13 bus 0 scbus13 target 0 lun 0
ada6: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

Disk Read/Write Stats
My setup has all FreeNAS settings set to default, no special tweaks or adjustments.

Hyper-threading enabled:

Code:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 602.091800 secs (178335235 bytes/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 363.791025 secs (295153467 bytes/sec)

Hyper-threading disabled:

Code:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 685.600273 secs (156613389 bytes/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 401.614066 secs (267356628 bytes/sec)

I get better results with hyper-threading enabled, anything else related to BIOS configuration has stock settings.

Data Transfer Speed
Based on my comparison tests done before and after upgrade, the extra memory and Rocket + Crucial combo set as cache drive boosted the data speed by over 500Mbit/sec. Previously, I was reading data at 400Mbit/sec and writing it at 300Mbit/sec. With the SSD cache enabled and the memory upgrade, I can read data at 900Mbit/sec and write it at 600Mbit/sec.

CIFS reads from Network Drive:



CIFS writes to Network Drive:

 

[ProtoSD]

Looks sweet, I like the drive cage design with the drives verticle... unlike my Lian case where the drives are stacked horizontally so the heat accumulates on up the stack.

I don't even see where your USB stick is... Anyway, nice clean build. I might only recommend SATA cables with clips so they don't wiggle loose somehow, or get accidently yanked while your digging around for some reason.

 

[Daisuke]

Thanks. I looked at your case also but decided to go with Fractal Design because it was smaller and did not have a cut for DVD drive. The case is ultra quiet with all 6 drives humming inside, very happy with the end results. I plan to use this box for storing all my development work (Linux) and media files (Windows).



Wiring "fun", playing with the Xion SATA cables:




Part of my working desk:

 

[ProtoSD]

Now I think I see your USB stick, no wonder I didn't recognize it, it's all naked! You shouldn't need to worry about it getting too hot, they're very low power devices for one and you've got everything so clean that there shouldn't be any heat build up significant enough to cause problems.

I like my case even though the drive cage is poorly designed because with my PicoPS I have a whole lot more room to add more drives if I install another controller. I also plan to mod my drive cage so the drives are vertical like yours at some point.

To be honest, I never had any problems with permissions on 8.0 because I pretty much opened them wide. My network is pretty secure, so I'm not worried. Supposedly the permission problems were fixed, but I'm waiting to hear from someone that could be still having problems, we'll see.

I use CIFS and NFS because I have a MythTV system and mount my NAS to it. I never tested NFS with Windows, so you could be right.

 

[Bohs Hansen]

Impressed with how good you got your cables aligned. Didn't manage that myself with the same case :) Well I didn't really try to hard either :)

 

[Darkaine]

I... Just might have to get that case. Very clean and efficient. Nice work.

 

[Daisuke]

Thanks guys for the nice comments. :)

I use CIFS and NFS because I have a MythTV system and mount my NAS to it. I never tested NFS with Windows, so you could be right.

I see. Well, apparently in 8.0.1 you only need to enable the NFS shares and that will do it in both Linux and Windows. I find it redundant to use CIFS/SMB for Windows and NFS for Linux. When I mean "permissions", I refer to the current 8.0 issue where in Windows you can create directories but not files on a NFS share. I opened tot eh max the perms, I could not do anything about it. I think is best to open a separate thread for that.

P.S. I'm glad you used a Supermicro Atom mobo, they rock the dual NIC's. :)

 

[Daisuke]

Upgraded my box, see the OP for hardware changes. :)

Based on my comparison tests done before and after upgrade, the extra memory and Rocket + Crucial combo set as cache drive boosted the data speed by over 500Mbit/sec. Previously, I was reading data at 400Mbit/sec and writing it at 300Mbit/sec. With the SSD cache enabled and the memory upgrade, I can read data at 900Mbit/sec and write it at 600Mbit/sec.

8GB of memory on the Supermicro X7SPA-HF-D525-O Intel Atom D525 1.8GHz mobo:



Network Disk to Local Disk (almost tapping the max network speed):



Local Disk to Network Disk:

 

[Letni]

Very Nice Rig!

I'm still a n00b here trying to discover the flaws of this product before I trust my data on it (I currently build CentOS boxes to run as custom NAS servers, so I'm not a n00b in that sense)... But couple of questions.

1. Do you have spindown successfully functional with your rig on FreeNAS 8.X?
2. Do you Bond the dual NIC on the supermicro board ? If so, do you notice any performance benefit
3. Do you know if the SSD drive gave you more benefit or the additional 4 GB Ram?
4. How did you enable the "console" output within the web GUI at the bottom? (its in your screenshot above)


Thanks

LetnI

 

[Daisuke]

1. I don't want my disks to spindown, the NAS is used mostly for development files shared across several CentOS boxes. However, the motherboard has ACPI/APM power management but the disks show no APM support. Here it is the info you need:

Code:

# ataidle ada1
Model:                  WDC WD20EARS-00MVWB0                    
Serial:                 WD-WBDXU6435219
Firmware Rev:           51.0AB51
ATA revision:           ATA-8
LBA 48:                 yes
Geometry:               16383 cyls, 16 heads, 63 spt
Capacity:               1863GB
SMART Supported:        yes
SMART Enabled:          yes
Write Cache Supported:  yes
Write Cache Enabled:    yes
APM Supported:          no
AAM Supported:          no

2. I use a Cisco SA520W-K9 mixed with a SG 300-10 switch, so I can use LACP without issues. Reading data at 115MB/sec is more than enough for my needs, so I use one NIC now. However, I know for a fact that bonding the 2 NIC's will increase the speed to over 150-160MB/sec. The problem is none of current drives I use in other boxes are worth the trouble, they would not handle properly the amount of data thrown at.

3. The added RAM is mostly the big gainer. The SSD cache helps if your working dataset is larger than system RAM, but small enough that a significant percentage of it will fit on the SSD. In my case, I have a dataset with several MySQL databases that is 30GB in size. So that set will get cached all the time on a fast drive, serving data at same speed but avoiding the ZFS hits. Again, this is is really beneficial if you run your connections through LACP... Since I like technology, I wanted to take advantage of all features FreeNAS offers related to performance. :)

4. There is an option in Settings > Advanced > Show console messages in the footer

 

[t3h0th3r]

I recently found out that the X7SPA-HF-D525 supports up to 8GB of RAM, even if Supermicro advertise it at 4GB. The manual also says that 1666 MHz and 1333 MHz memory may be used, but will operate at 800 MHz.

That's great news! this info should be posted everywhere! (at least on the product page at supermicro :/ ).

 

[topsub]

how much does the ssd caching help? I have on order 6x 3TB drives with a 8gb patroit for the OS freenas. I was looking for somekind of either raid card or another card so i can hook up 2 more drives ( max my case will allow total of 8 drives ). Instead of trying to slam another 2x 3TB drives should i look at SSD Caching? The mobo i have is all sata3 connections so if i did just put another 2 drives in there i was trying to find a sata3 raid card that is supported by freenas.

 

[Daisuke]

how much does the ssd caching help?

That is what I'm trying to determine also, there is no easy way to find out. If anyone knows, please post the procedure. I requested a graph for ZIL/Cache.

Freenas documentation says that SSD cache devices only help if your dataset is larger than system RAM, but small enough that a significant percentage of it will fit on the SSD. In other words, if you have a 64GB disk and your dataset has 30GB used out of 5TB, it will help a lot. Please correct me if anyone thinks I got it wrong.

Personally, I have 1 dataset (mnt/nas) with 2 sub-datasets (/mnt/nas/unix and /mnt/nas/media). I store development data files like MySQL databases where the cache speed is beneficiary, into /unix dataset. That dataset is now 50GB, so it does fit into SSD without issues. I do not have a defined quota or reserved space for that dataset.

 

[SoftDux-Rudi]

Freenas documentation says that SSD cache devices only help if your dataset is larger than system RAM, but small enough that a significant percentage of it will fit on the SSD. In other words, if you have a 64GB disk and your dataset has 30GB used out of 5TB, it will help a lot. Please correct me if anyone thinks I got it wrong.

So this is really only very useful for smaller datasets. If 10 people work on 10 different datasets of 32GB each, then you'll probably at least need a 400GB SSD otherwise everything will grind to a halt. Our quickbooks books set is 39GB right now and the accounts dept. does a lot of work on it throughout the day.

I suppose, ideally one should try and cut down on the larger datasets or try and set certain times for certain people to work on certain datasets. OR, alternatively have another NAS with it's own dataset and split the workload among the 2 NAS's if productivity is important.

 

[jonnyfive]

Upgraded my box, see the OP for hardware changes. :)

8GB of memory on the Supermicro X7SPA-HF-D525-O Intel Atom D525 1.8GHz mobo...

...Network Disk to Local Disk (almost tapping the max network speed)

Very Cool, I am looking at a similar build (same case 5x2TB Samsung F4 drives in RAIDz2) I was trying to decide between the ASUS E35M1-I board, and the 510 version of your board with the intel graphics: MBD-X7SPA-H-O (could also get the MBD-X7SPA-HF-O with Matrox graphics) mainly to be able to use more RAM but after reading this I was looking around for the MBD-X7SPA-HF-D525-O as the dual Intel NICs and internal usb socket are preferred

I have a few questions if you have a moment:

1. Where in the world did you even get the MBD-X7SPA-HF-D525-O I can find it from a retailer I am comfortable with and doees the MAtrox GPU give the system back some memory vs the Intel?
2. What were the transfer speeds like with 4GB RAM
3. Do perhaps you know if the MBD-X7SPA-H-O will recognize 8GB DDR2 did some searches no love?
4. In case of the AMD platform Is the trade off for the extra RAM vs the single Realtek NIC going to even improve real-usage-case speeds?
5. For my purposes as a home user (data backup, media storage, torrent box) is the jump from 4 to 8GB to enable pre-fetching even worth it since I'm not hosting DBs and will only have 2 possibly 3 local users ever connected and even then probably only streaming video or using it as a Time Machine with a Mac?
6. Finally, Will the pre-fetching increase drive life if the the zfs array is hosting active torrents? (I'm considering having a single drive hosting active torrents)

Thanks!
j5

 

[ProtoSD]

Jonnyfive,

I have the 510 / MBD-X7SPA-H-O and am planning to order 8GB of RAM and see if it'll work, but not for another 3 weeks. I completely overlooked the documented 4GB limit when I got mine, so I'm hoping it's incorrect.

-- Proto

 

[jonnyfive]

Cool, thanks for the reply and good luck Proto I am curious to see! I had read your performance reports on what you have currently and that seemed fine for me, but then I read TECKs post and figured why not go for the possible bump in performance with the recommend memory config for the size of my array...

Actually if DDR2 was cheaper for 4GB 200-pinn DIMMs I would have just gone for it and lived with 4GB of RAM if it didn't work. Anyway I hope you get the 8GB to be recognized.

I went ahead and gambled on a unfamiliar retailer for the X7SPA-HF-D525-O since I bought the Drives already (Newegg had a 24/hour $40 discount on the Samsung 2TB drives, and that was best price I've seen in months of weekly checking) and would like to build this and run it for a while so if I have any drive problems I can deal with Newegg RMA instead of Samsung.

I may end up with your setup and be out the price of the MB if I have issues...

Thanks and goodluck man! Y'all are very helpful over here!
j5

 

[ssingleton]

Can you please advise your BIOS settings. Reason i ask is that i have the same board and it is taken minutes to detect all drive in boot up and slow to boot freenas

 

[Daisuke]

Can you please advise your BIOS settings. Reason i ask is that i have the same board and it is taken minutes to detect all drive in boot up and slow to boot freenas

The only changes I made are related to AHCI and headless mode.

 

[Daisuke]

Very Cool, I am looking at a similar build (same case 5x2TB Samsung F4 drives in RAIDz2) I was trying to decide between the ASUS E35M1-I board, and the 510 version of your board with the intel graphics: MBD-X7SPA-H-O (could also get the MBD-X7SPA-HF-O with Matrox graphics) mainly to be able to use more RAM but after reading this I was looking around for the MBD-X7SPA-HF-D525-O as the dual Intel NICs and internal usb socket are preferred

I have a few questions if you have a moment:

1. Where in the world did you even get the MBD-X7SPA-HF-D525-O I can find it from a retailer I am comfortable with and doees the MAtrox GPU give the system back some memory vs the Intel?
2. What were the transfer speeds like with 4GB RAM
3. Do perhaps you know if the MBD-X7SPA-H-O will recognize 8GB DDR2 did some searches no love?
4. In case of the AMD platform Is the trade off for the extra RAM vs the single Realtek NIC going to even improve real-usage-case speeds?
5. For my purposes as a home user (data backup, media storage, torrent box) is the jump from 4 to 8GB to enable pre-fetching even worth it since I'm not hosting DBs and will only have 2 possibly 3 local users ever connected and even then probably only streaming video or using it as a Time Machine with a Mac?
6. Finally, Will the pre-fetching increase drive life if the the zfs array is hosting active torrents? (I'm considering having a single drive hosting active torrents)

Thanks!
j5

1) I got the mobo from NCIX, special order.
2) With 4GB and no SSD, the transfers on CIFS were around 40MB/sec.
3) No idea, from what I read on hard forums only my model will take 8GB.
4) No idea, sorry.
5) Yes.
6) No idea, I rely on the SSD to do most of caching.

I've been running 24/7 my box for over 8 months never had a single issue. Knock on wood. :)
So far, I have 3TB full out of 7... which makes me slap my face for not getting the 3TB disks when they were in special.