Hardware: SATA Controller Selection, or Motherboard

[9C1 Newbee]

I have been running freenas for a total of 5 months for my home media server. I LOVE the rock solid reliability 8.3 has offered with various plugins. I have done a lot of reading. I have read probably 100+ threads and I still have some noob questions. I ordered an Intel/LSI SASUC8I (recommended here http://forums.freenas.org/showthread.php?1568-FAQ-What-hardware-should-I-Buy-Get-Use-or-is-recommended&p=6757&viewfull=1#post6757 thinking I could flash the IT and be good to go. What I just read was this card does NOT support >2TB drives. most of these threads offer hardware ideas that are outdated or no longer available.This is my current hardware:

Dell Power Edge 105T
Dual-Core AMD Opteron(tm) Processor 1216
8 Gigs Memory (maxed out)
Three 3TB WD Green Drives (Yeah I know, lack of research. However it has ran flawlessly thus far <knock knock knock>)
ZFS Raidz1

My concerns are:
1) I need more SATA ports. I am outgrowing my 6TB storage capacity. My plans are to purchase 3 additional 3TB WD sata (red, not green this time) and run the 6 in raidz2.

2) From what I have read, my limitation of 8Gigs of memory is becoming a factor. From what I have read, I should be able to reliably squeeze by but take a performance hit.

So, should I just build a new i3 machine that can accept 32 gigs of memory and has ample SATA's? Or, is there a controller out there with my name on it? Please chime in with any other issue I should be concerned with.

 

[jgreco]

IBM ServeRAID M1015 crossflashed to IT mode. Cheap on eBay. 8 ports of ZFS awesomeness.

 

[ProtoSD]

I ordered an Intel/LSI SASUC8I (recommended here http://forums.freenas.org/showthread.php?1568-FAQ-What-hardware-should-I-Buy-Get-Use-or-is-recommended&p=6757&viewfull=1#post6757 thinking I could flash the IT and be good to go. What I just read was this card does NOT support >2TB drives.

Sorry about that, it's a tough job trying to keep up with changes. I updated that post and noted that controller does NOT support >2TB drives.

 

[9C1 Newbee]

IBM ServeRAID M1015 crossflashed to IT mode. Cheap on eBay. 8 ports of ZFS awesomeness.

Funny, after I contacted the guy who sold me the SASUC8I and told him I couldn't use it, he recommended the M1015. Can you point me toward a good crossflash tutorial?

Sorry about that, it's a tough job trying to keep up with changes. I updated that post and noted that controller does NOT support >2TB drives.

Information about the 2TB cap was hard to find. I happened to stumble into a thread where a guy was having issues. I had looked at several spec sheets from multiple locations and never was the cap mentioned. Not an obvious thing to find.

I am sure it is a huge job keeping things current. It is hard work from people like you that makes it possible for people like me to have a clue on how to tackle issues like this. If it was not for that list I would have been blindly swapping in cards.

Thank you guys! :D

 

[ProtoSD]

Funny, after I contacted the guy who sold me the SASUC8I and told him I couldn't use it, he recommended the M1015. Can you point me toward a good crossflash tutorial?

Jgreco's post here has a link about how to crossflash.

http://forums.freenas.org/showthread.php?11901-Confused-about-that-LSI-card-Join-the-crowd

Information about the 2TB cap was hard to find. I happened to stumble into a thread where a guy was having issues. I had looked at several spec sheets from multiple locations and never was the cap mentioned. Not an obvious thing to find.

I am sure it is a huge job keeping things current. It is hard work from people like you that makes it possible for people like me to have a clue on how to tackle issues like this. If it was not for that list I would have been blindly swapping in cards.

Thank you guys! :D

You're welcome! Its feedback from resourceful people like yourself that helps keep things updated.

 

[jgreco]

There's actually already a warning in the LSI sticky about the 2GB limit, but it is admittedly kind of vague. Around here, we kind of have a tendency to evaluate hardware and then settle on something solid and stick with it awhile, so despite the fact that the LSI 3Gb/s controllers were probably fine for ESXi host datastores (local datastores are typ. 1TB or less), I think we eliminated all the 3Gb/s controllers as an option once it was clear that we were within a few years of hitting their max cap. Let me think about what to add to that.

 

[9C1 Newbee]

Just in case somebody else runs across this thread...

If you have an older controller of some sort, one that doesn't do SATA-III, you may need some other drivers and you will also be limited to no more than 2TB drives. If you are considering buying one, don't, unless you're getting it for $10 or free or something like that... such controllers include the IBM ServeRAID BR10i and Intel RAID Controller SASUC8I (LSI SAS3082E-R/LSI SAS1068E chipset) and are driven in FreeBSD by the LSI Logic Fusion-MPT SCSI driver ("mpt"). It is possible to crossflash these cards to be a generic LSI SAS1068E card, but the silicon is still capped at 2TB drives.

That is pretty clear.

A favored card in the FreeNAS community is the IBM ServeRAID M1015...............If you get an M1015 and crossflash it to IT mode, you end up with one of the best HBA controllers available for FreeNAS, in my opinion.

So at this point in time, this is the way to go. I have an M1015 en route. I bought my card for $125. I shopped around for about 30 minutes. Looks like the going rate climbed a tad bit. I just created a bootable thumb drive with the flash utility for when the card arrives. I will report back with any issues and thoughts from a noobs point of view.

A few IBM ServeRAID M1015 facts:

SAS 6.0gbps/ SATA III
8 internal ports (SAS2/SATA3 6Gbps)
Uses a PCIe 8x v2.0 slot
Not limited by the 2TB cap (not sure what the cap is)


All quotes pulled from http://forums.freenas.org/showthread...Join-the-crowd

 

[9C1 Newbee]

The brown truck delivered my M1015 a few hours ago. I went ahead and popped it in my Dell T105 server and proceeded to flash it into IT mode per this thread.

I received some sort of error and it froze. When I reboot the server, it reported a pci conflict with the mass storage controller or something to that effect. I tried disabling everything embedded in the motherboard and nothing worked. Long story short, there are numerous accounts of the same issue with raid cards in Dell T105 computers. It appears Dell purposefully restricts the bios to not work with anything other than Dell cards. %#$k you Dell!

So, I will be in the market for a good motherboard combo that can support plenty of ram.

Cliff Notes: IBM ServeRAID M1015 (or any other non dell raid card) in a Dell T105 does not work!

 

[jgreco]

It appears Dell purposefully restricts the bios to not work with anything other than Dell cards. %#$k you Dell!

Wow, that sucks. Very sorry to hear it.

Not limited by the 2TB cap (not sure what the cap is)

I can't find one. And I doubt there's a meaningful one. The 2TB cap is a 32-bit limit. You can address 4,294,967,296 (0-4,294,967,295) in a 32-bit value. With 512-byte sectors, that gives you 2,199,023,255,040 bytes of capacity before you can't address more.

So at a minimum they would have added a single byte, which would allow for a 500TB hard drive. That's ~100x the capacity of today's drives. Looking back to the days of 40GB drives (4TB / 100), that's like 10-to-15 years in the past, so even adding a single byte future-proofs the controller for the likely lifetime of the technology.

If you are open to suggestions, I am very pleased with our Supermicro 1155 boards, a mid-range Xeon E3-1230 CPU, and 32GB of ECC RAM. I've seen lots of people building systems with random consumer-grade 1155 boards and various i3/i5 CPU's, and you'd think that ought to work, and it even does, but the Supermicro stuff is designed for servers and it just works. You lose some things (probably no USB3) but you gain rock solid support for ECC, extra cores for compression, support for AES-NI, and two Intel ethernets onboard. It will almost certainly cost more, but perhaps not as much as you would expect (especially since a lot of the best prosumer 1155 boards can be ~$200, and the X9SCL+-F is ~$180). Just something to think about.

 

[9C1 Newbee]

Wow, that sucks. Very sorry to hear it.

Thank you. Although I was on the fence about replacing the motherboard, I really wanted to continue using it if I could because it seemed fast enough. Adding more storage I felt the 8 gigs was questionable. That was strike one. But you guys did not mention it was an issue so I felt I might scrape by. Dell uses a BTX (read not ATX) configuration on their boards. I was going to get really crafty and in the end probably not worth it to make it work in an ATX case I have. That was strike two, but I was willing to try and see. The next problem is Dell uses a proprietary motherboard power plug pin out. I broke out the multimeter and discovered I was lucky and had a standardized connector. The M1015 not working was an absolute show stopper.

I can't find one. And I doubt there's a meaningful one. The 2TB cap is a 32-bit limit. You can address 4,294,967,296 (0-4,294,967,295) in a 32-bit value. With 512-byte sectors, that gives you 2,199,023,255,040 bytes of capacity before you can't address more.

Alright. That makes perfect sense. Kinda like the 3 gig ram cap. Nice to know I spent money on something I can use for a little while.

If you are open to suggestions...

Always! I am here to learn. However, I did pull the trigger on a few items based on a post recommending 3 different Supermicro boards. I can't find the post but I had researched. It might have even been your post. But here is what I ended up with. If you see any issues, please let me know.

SUPERMICRO MBD-X9SCM-F-O LGA 1155 Intel C204 Micro ATX Intel Xeon E3 Server Motherboard
x2 8GB DDR3 Memory
Intel Core i3-2130 Dual-Core Processor 3.4 GHz 3 MB Cache LGA 1155
Corsair Professional Series 760 Watt ATX/EPS Modular 80 PLUS Platinum Power Supply AX760
I hope these work with the M1015 I hope these work with the M1015: 3WARE Cable, 1 Unit Of 1 Meter Multi-lane Internal (SFF-8087) Serial Ata Breakout

 

[jgreco]

$90 a stick for that RAM seems pricey. You can get Kingston 8GB sticks for $66, or 32GB for $267. The X9SCM is a well-liked board. I'd ditch the e3-2130 because it doesn't have ECC, memory speed only goes to 1333MHz, it only has two cores, 3M of cache, and the speed is limited to 3.4GHz. The Xeon E3-1230V2 will do ECC, memory speeds of 1333 or 1600, four cores, 8M of cache, and while the base speed is 3.3GHz, it will turbo up to 3.7GHz, which is nice for Samba and anything else limited by the speed of a single core. Of course, it's more expensive too. The 3Ware cable should be fine.

 

[9C1 Newbee]

Something told me I should have asked before I ordered. I went ahead and ordered the Xeon E3-1230V2. The money I'll save in ram about paid for the CPU. Thank you!

 

[9C1 Newbee]

Alright, I thought I would report back. All the parts were delivered and I was able to spend a little quality time with my build last night.

1) The first issue I had was flashing the IBM ServeRAID M1015 controller to IT mode. Here is my post in jgreco's thread:

PROBLEM: While I was flashing the M1015 to IT mode, during the process I got an error message "Failed to initialize PAL".

SOLUTION: I used a different computer to perform the flash without issue. Once the M1015 was flashed to IT mode, I was able to place the card back into the intended machine and use it with out issue.

A quick search of the web turned up the same results for others during the flash process. The error seems to manifest on certain Intel chipsets. That made sense due to the fact I was attempting to flash the controller with Supermicro X9SCM-F with an Intel chipset, which resulted in the error. I ended up successfully flashing the M1015 in a Dell inspiron. I am surprised it worked because getting %#ck^d over by Dell machines seems to be the status quo for me lately.

2) The next issue I had was, freenas was not using the NIC cards. I couldn't log onto the GUI. After a search of this forum, I found that sometimes Freenas does strange things when you transfer over to new hardware. I even reinstalled Freenas via the cdrom method. I had the same issue. The solution was to reset Freenas to factory settings. That resolved my problem. Any changes from stock (default) need to be made by hand. If you import your saved profile, it will cause the issue(s) to come back. Now I need to research all those changes I made (as in about 5 changes cause that is all I really understand at this point). Oh and by the way, I am running the latest and greatest FreeNAS-8.3.1-RELEASE-p2-x64 at this point in time.

Right now she is chugging away as I am loading stuff back on. The reason I had to reload the data was due to migrating from a 3 disk raid z1 vdev to a 6 disk raid z2 vedev. The only way to do this is to destroy the existing pool if you intend to reuse the current disks. I am glad I am making the move now while I don't have a obscene amount of data to transfer.

Once again, THANK YOU jgreco for helping me with this awesome build! 10TB useable for about a grand.

 

[9C1 Newbee]

I have been rocking this server for a little over a year now. Over all I must say I am in love with it. There is only one stupid decision I made.

The only thing I would have changed is the use of Western Digital Green 3 Terabyte drives. These drives absolutely suck. I have a failure about once a month on average. Each drive has been replaced at least once. I thought I would save a bit of cash by going with the cheapest drive I could find. The warranty replacement through WD has been pretty good I must say. I can RMA like a champ. I ended up buying two extra drives as spares. So in the end, it cost me more to purchase the drives, and all the damn postage to Riverside, CA. But if you positively, absolutely, want to ask for trouble and use these drives like I did, set it up for raidz3.

 

[cyberjock]

I have 24 WD Green drives and they make excellent server disks. I bet you've done something wrong and that's why they are failing so often. I've had 3 failures in 3 years.