Backblaze Storage Pod 4.0

[cyberjock]

Depends on your use case. I wouldn't try to go with a system with less than 96GB and I'd recommend at *least* 128GB for most users. You may need even more though. :P

 

[Ericloewe]

Just a noob question , if 180TB of hds are installed, how much ram is needed? Around 180G of ram? Does the 1g of ram per Tb rule still applies?

The rule is a bit flexible, but you'll probably want at least 128GB, maybe some 192GB. It depends on your use case.

 

[pyrodex]

What is the feasibility of the Rocket card in a ZFS build? If people want to create dense mini-itx builds with decent ram amounts would this card help?

 

[cyberjock]

Umm. What the hell is a "Rocket" card?

 

[Ericloewe]

What is the feasibility of the Rocket card in a ZFS build? If people want to create dense mini-itx builds with decent ram amounts would this card help?

How the hell does the card impact RAM? Xeon E3s are limited to 32GB and Avotons are limited to 64GB (when / if the mythical 16GB DIMMs arrive - and they won't be cheap).

I have my doubts about their driver and the whole thing is a bit of a kludge. A controller and two expanders on that card?

 

[pyrodex]

Umm. What the hell is a "Rocket" card?

The Highpoint Rocket 750 card, that is what is being used in the build here and not sure if anyone is using it with performance reports on FreeNas yet.

 

[Ericloewe]

The Highpoint Rocket 750 card, that is what is being used in the build here and not sure if anyone is using it with performance reports on FreeNas yet.

Don't expect performance from that thing.

 

[cyberjock]

Highpoint controllers are a f'in nightmare and AFAIK never ever should be used with ZFS. I have stickies on other models if you want to read up.

Honestly, when I first saw you mention "rocket" cards I was going to say something like "If you are talking about Highpoint controllers and you are seriously asking about them then I need to stop replying". They are a nightmare and should be avoided at any and all costs.

 

[c32767a]

The Highpoint Rocket 750 card, that is what is being used in the build here and not sure if anyone is using it with performance reports on FreeNas yet.

We built a couple of the 4.0 storage pods for use with FreeNAS.. We did NOT use the highpoint Rocket 750 cards.. They're scary. :)

Instead, we used the LSI 9201-16i. It works great.. They're older (and getting harder to find), so you'll have to install the appropriate firmware to match FreeNAS's LSI driver, but they work just fine.

 

[KevinM]

We built a couple of the 4.0 storage pods for use with FreeNAS.. We did NOT use the highpoint Rocket 750 cards.. They're scary. :)

Instead, we used the LSI 9201-16i. It works great.. They're older (and getting harder to find), so you'll have to install the appropriate firmware to match FreeNAS's LSI driver, but they work just fine.

IMO the best part of the Backblaze solution was the publicity it generated. We had just spent $$$ to upgrade our NetApp infrastructure and still didn't didn't have enough space to consolidate the many terabytes of archival storage we had laying around on several smaller end of life SANs. Post-Backblaze, it was much easier to have a discussion about building lower cost storage that would finally allow us to get rid of all the old junk.

I looked at the storage pod for FreeNAS pretty closely last year and even went so far as to get a quote from Protocase, Backblaze's chassis supplier. However by the time you factor in the bump in cost for redundant power supplies it did seem like Supermicro's dual-socket E5 systems were a better solution for ZFS. The E1R36L costs about $2600 and comes with an lsi 2308 hba in IT mode, has 16 dimm sockets so it will support 256 gb memory using cheap 16 gb dimms, supports e5 CPUs, has hot-swappable drives, etc.

We've had 2 Supermicro FreeNAS boxes chugging along for about a year now. No real issues other than a batch of bad memory at the beginning, and surprisingly good performance given that the primary design criteria were reliability and capacity.

 

[c32767a]

I looked at the storage pod for FreeNAS pretty closely last year and even went so far as to get a quote from Protocase, Backblaze's chassis supplier. However by the time you factor in the bump in cost for redundant power supplies it did seem like Supermicro's dual-socket E5 systems were a better solution for ZFS. The E1R36L costs about $2600 and comes with an lsi 2308 hba in IT mode, has 16 dimm sockets so it will support 256 gb memory using cheap 16 gb dimms, supports e5 CPUs, has hot-swappable drives, etc.

We've had 2 Supermicro FreeNAS boxes chugging along for about a year now. No real issues other than a batch of bad memory at the beginning, and surprisingly good performance given that the primary design criteria were reliability and capacity.

IMHO the 4.0 pod is a lot better than the previous versions. We use it over the Supermicro chassis for a couple reasons. The largest is acoustic. We place these units in remote offices. They have HVAC, but are typically not full datacenter environments. In a lot of cases there is not much isolation between the server room and the local humans. The storage pod design is remarkably quiet when compared to the super micro servers. We also use the pod because some of our sites actually have more than 32 drives, and the incremental costs are lower than buying 2 super micro chassis.

We use the X9SRL-F-O with 64G of ECC and things work quite well for us. One socket with a 2.4g processor is sufficient for our workload.. Also, since it's tertiary storage, we also only use the single PSU version and stock spare PSUs on site.

The only downside (and it's not a small one) is that you get an erector set. When they say you have to build it, they mean it. the SATA cables come pre attached to the bases, but everything else is loose and needs to be assembled. Once you do a couple it gets easier.. :)

 

[marbus90]

I think with the recent findings it's time to bump that thread back up.

@cyberjock: You might find that the Highpoint Rocket 750 is a SATA-only card without any RAID functionality. I had a look at the FreeBSD 9.3 driver and was astonished by the mentioning of specific FreeNAS 9.3 GUI errata, which have been fixed troughout the driver revisions.
Of course one can't expect massive 40x 6Gbps bandwidth from the card - it's limited by the PCIe 2.0 x8 interface. But! this would be the case with any Expander-based chassis as well, squeezing many HDDs to few lanes.

Especially interesting does this card get because the newer Pods are available in 30 and 60 bays as well.
4x LSI 9201-16i -> 64W TDP total -> ~1500EUR give or take -> 4x PCIe x8 slots for best performance
2x Rocket 750 -> 22W TDP total -> ~1500EUR give or take -> only 2x PCIe x8 slots needed.

Many users are relying on SATA drives only, so that's less of an issue. If one were to stuff it full with 6TB Reds, that's 105A on 12V for spinup current. The Corsair AX1500i seems the only off-the-shelf model which features such a 125A @12V rail: http://www.corsair.com/en-us/ax1500i-digital-atx-power-supply-1500-watt-fully-modular-psu which currently runs for 400$ at newegg. Still, 2000$ for a 4U rackmount chassis with 60 SAS/SATA bays and a PSU is not too bad.

 

[cyberjock]

Yep, it just might be a good card. But Highpoint's reputation on this forum is earned, and earned very well because:

1. They refuse to support FreeBSD themselves.
2. I tried to work with them to help fix some of their driver/compatibility problems (they didn't want to improve it)
3. I tried to work with them to help make SMART work better with their cards (they refused to discuss the matter). In this case, we weren't even asking for code, someone was going to write the code FOR smartmontools, but Highpoint refused to provide any feedback.

So yeah, the 750 might work. But when you consider the history that Highpoint has had with FreeBSD, with FreeNAS, and with me trying to work with them, it does NOT paint a good relationship where a customer can buy a 750 and expect some kind of problem to be fixed if one came up. It also shouldn't give you any kind of expectation that they might not accidentally break compatibility someday and you end up with a card that no longer works with FreeBSD.

Highpoint is to LSI what Asrock Rack is to Supermicro. And when Asrock Rack came out with the C2750D4I board that everyone wants to love around here I commented that I wouldn't trust them until they've proven that they care about their customer and the community base that use their product. Well, Marvell controllers are having problems, and have had problems for almost a year. Still no ETA on when it will be fixed, and there is definitely no guarantee that it even WILL be fixed.

So keep this in mind when arguing for the card. Sure, the technology may work fine today. But I'm thinking about things more than "it works today" when I'm spending my money as well as when I give recommendations.

 

[marbus90]

1. They refuse to support FreeBSD themselves.
2. I tried to work with them to help fix some of their driver/compatibility problems (they didn't want to improve it)
3. I tried to work with them to help make SMART work better with their cards (they refused to discuss the matter). In this case, we weren't even asking for code, someone was going to write the code FOR smartmontools, but Highpoint refused to provide any feedback.

Download the Rocket 750 BSD driver package yourself and look trough that included changelog. Rocket is not RocketRAID. Seriously, do it.

 

[cyberjock]

Download the Rocket 750 BSD driver package yourself and look trough that included changelog. Rocket is not RocketRAID. Seriously, do it.

I know this. I've known this for quite a while. You totally missed my arguments, didn't you? My problem is NOT with the 750. My problem is with the COMPANY that is selling the product and how the COMPANY has handled problems in the past.

As I said above, and I will say again:

So yeah, the 750 might work. But when you consider the history that Highpoint has had with FreeBSD, with FreeNAS, and with me trying to work with them, it does NOT paint a good relationship where a customer can buy a 750 and expect some kind of problem to be fixed if one came up. It also shouldn't give you any kind of expectation that they might not accidentally break compatibility someday and you end up with a card that no longer works with FreeBSD.

 

[marbus90]

I do get the part that you had troubles with them /in the past/ concerning another product line. I don't get that you are clinging onto history while it is imho pretty obvious that Highpoint wants to sell the Rocket 750 card to FreeNAS datahoarders as well and even fixes problems specific to FreeNAS users.

Intel is going off your list because their ARK is errorneus. AMD never was on there. Are you building ARM servers in the future?

 

[cyberjock]

I do get the part that you had troubles with them /in the past/ concerning another product line. I don't get that you are clinging onto history while it is imho pretty obvious that Highpoint wants to sell the Rocket 750 card to FreeNAS datahoarders as well and even fixes problems specific to FreeNAS users.

Uh.. what!? By that logic we should have had the C2750D4I Marvell controller fixed over a year ago. It was intended for the FreeNAS community. Even their *own* documentation makes heavy mentions of FreeNAS. Notice that I've already said that the C2750D4I is broken and there is no ETA at present.

Let's look at the alleged WRT54G successor... the WRT1900AC. That was supposed to be all "community friendly", and it was no such thing. Even now, one of my friends that bought one of those gave it up and switched to pfsense because it never really became the product that Linksys wanted it to be.

Highpoint is in the same boat. They wanted it to be great for FreeNAS. In fact, they sent me a free controller (4520 I think it was) so I could use it, help them fix it up and such. It supposedly supports JBOD. So I got the controller and figured out I couldn't do SMART. So I went back to them so that I could get it added and such. Nope, no dice. What they want to do, and the effort they are willing to put into it, are not the same thing, at all.

I do NOT recommend hardware based on promises. I learned a decade ago that expecting companies to honor their promises after you've bought the product is really damn stupid on your part unless they have a history of actually honoring their promises. I just showed 3 examples from 3 different companies that those promises mean absolutely nothing. I also just showed that the *exact* company you are discussing has already gone back on their promises. So why the hell would I *ever* recommend them until they've actually proven themselves with more than some extremely cheap lipservice that they plan to support it into the future. They can put their damn money where their mouth is and prove it. Until then, their history with their own products with FreeNAS has been so poor as to be unreasonably unreliable, especially for something that we want... no, need... to be reliable.

If your expectations are low enough that you want to trust highpoint, go for it. Buy it, use it, recommend it to others. You'll be the guy that gets questioned by dozens of people when you tell someone to buy the 750 and 6 months later it's not working. And you'll be just as helpless as they are if Highpoint decides to abandon the 750, doesn't want to put devs where their mouth is, etc.

Intel is going off your list because their ARK is erroneus. AMD never was on there. Are you building ARM servers in the future?

You'd better be quoting me where I said Intel is going off my list. Because I've said no such thing, ever. The only thing I have said is that if you want to trust ECC, you'd better go with Xeons.

AMD has never on my list because I wrote them off a decade ago, and nobody here has ever offered any evidence that contradicts what I've seen. In fact, I'd argue that there's such an overwhelming quantity of easily observable evidence against AMD that I don't even need to discuss it further.

ARM... yeah, you're silly for even mentioning that in 2015. There is no ARM version of FreeNAS, so you're just making silly statements now.

 

[jgreco]

Notice that I've already said that the C2750D4I is broken and there is no ETA at present.

That may be because no one's beating on Marvell to fix it, and the only one with sufficient clout to do so might be AsRock.

You'd better be quoting me where I said Intel is going off my list.

Seems quite clear to me that you're unhappy with the Intel state of things.

ARM... yeah, you're silly for even mentioning that in 2015. There is no ARM version of FreeNAS, so you're just making silly statements now.

In 2015, I'd say that's actually an interesting threat. FreeBSD for ARM has been coming along quite nicely and some of us are looking at platforms to run it on. I'd love to see it on the Ubiquiti EdgeRouter X or Lite, for example. And there are rumblings of future product offerings, above and beyond Gigabyte's. With up to 128GB of RAM onboard, the R120-P30 is a poorly balanced platform because it only has 4x 3.5" bays, but I could see them making something with 2.5" bays, or someone else building it into a non-idiotic chassis for serving low power, high performance ZFS-on-SSD. Would probably take a company with some experience in building custom servers.

 

[cyberjock]

Yes, I am unhappy, but that is totally different from "Intel going off my list". I've been unhappy with the state of affairs for ECC since before I got involved with FreeNAS. Nothing has changed about that...