FreeNAS with Raspberry Pi. Is It Possible?

[jgreco]

Platform-hopping is not that unusual in the consumer space. Apples APs went through all sorts of platforms starting with the CISC 486 in the design they licensed form Avago to various flavors of RISC. I have no doubt that cost, heat, and performance were drivers in all this. Ditto re: switching from 680xx to PowerPC to Intel to Axx on the desktop side. High-volume OEMs can afford to change their minds re: processors / architectures.

I doubt this is the case with iXSystems. They don’t have the scale to justify significant processor departures, even if good compilers take care of the bulk of the work. There are so many dependencies to track down and adjust, it’s not just a brain transplant. Given Intel and AMDs adequate CPU and I/O performance, I’d suggest focusing the resources at iXSystems on fixing extant bugs and adding new features, not adding new platforms unless a really compelling reason presents itself.

I don't think it is likely that they're going to be releasing a general purpose ARM release or anything like that, but there are opportunities for licensing the core technology, or maybe something exciting like a FreeNAS Mini based on a much more compact ARM form factor.

A few people in the know around here are probably aware that I maintain a highly customized appliance version of FreeBSD that is designed for Internet service applications. I support both i386 and amd64, because the disk and memory footprints of i386 are substantially smaller than amd64 (by maybe a quarter to a third, generally speaking), and you usually don't want or need your DHCP server VM to be an amd64 based host.

I had attempted a few times to do an ARM build, but due to the special magic hacking most of the ARM platforms need to get a toehold, I didn't have that much success. Until VMware released its ESXi-on-ARM fling, that is. Then, it turned out to be about a morning's hacking plus a few tweaks over the next several days to make it usable and stable, because VMware worked correctly with straightforward EFI boot.

So my point is, that most of the stuff running ON a FreeBSD system is (or can be made to be) platform-agnostic, but the actual details of getting the bootloader/partitioning/etc set up, and other hardware specific details tend to be the vast majority of the work with these sorts of brain transplants.

 

[darkmode]

I was referring to the Pi having ECC or not, not the need for it to do ZFS right. (of course ECC should be used).
...

It's not a question of if the Pi (actually the CM4 that I cited) has ECC or not. The specifications state that it has ECC Ram.

 

[Ericloewe]

It's not a question of if the Pi (actually the CM4 that I cited) has ECC or not. The specifications state that it has ECC Ram.

That statement was so crazy that I went looking, fully expecting to refute it with the datasheet. And yet, the datasheet agrees with you!

Now, because crazy statements need more than just a single line in a datasheet to support them, I did some digging. No other documents seem to support the claim. So I went looking for one of the DRAM part numbers: Samsung K4F8E304HB-MGCJ, an 8 Gbit LPDDR4 IC. Samsung doesn't make it clear, but I think it's a 32-wide part, which makes sense for a typical embedded memory controller.
That means that the datasheet is most likely full of crap for three reasons:

• It's hard to imagine a heavily cost-constrained part like the BCM2711 employing ECC and then not advertising it widely
• 8 Gbit is 1 Gbyte, the absolute minimum configuration for the CM4. ECC is always going to require additional space - you might cheat to some extent with compression, but sooner or later incompressible data will bite you in the ass. So, you'd end up with less RAM than advertised. See the next point for just how much overhead that is (spoiler: 6.5 Gbit).
• With a width of 32 bits, to get traditional ECC behavior (correct single-bit errors, detect two-bit errors), you'd have 26 data bits and 6 parity bits, which would destroy performance of any memory-bound workloads (This thing is running four ARM cores and a GPU... Memory bandwidth is always in short supply.).

So yeah, I'm not believing that without some really solid evidence. Which is unfortunate, because this would be so cool.

 

[darkmode]

That statement was so crazy that I went looking, fully expecting to refute it with the datasheet. And yet, the datasheet agrees with you!

Now, because crazy statements need more than just a single line in a datasheet to support them, I did some digging. No other documents seem to support the claim. So I went looking for one of the DRAM part numbers: Samsung K4F8E304HB-MGCJ, an 8 Gbit LPDDR4 IC. Samsung doesn't make it clear, but I think it's a 32-wide part, which makes sense for a typical embedded memory controller.
That means that the datasheet is most likely full of crap for three reasons:

• It's hard to imagine a heavily cost-constrained part like the BCM2711 employing ECC and then not advertising it widely
• 8 Gbit is 1 Gbyte, the absolute minimum configuration for the CM4. ECC is always going to require additional space - you might cheat to some extent with compression, but sooner or later incompressible data will bite you in the ass. So, you'd end up with less RAM than advertised. See the next point for just how much overhead that is (spoiler: 6.5 Gbit).
• With a width of 32 bits, to get traditional ECC behavior (correct single-bit errors, detect two-bit errors), you'd have 26 data bits and 6 parity bits, which would destroy performance of any memory-bound workloads (This thing is running four ARM cores and a GPU... Memory bandwidth is always in short supply.).

So yeah, I'm not believing that without some really solid evidence. Which is unfortunate, because this would be so cool.

Perhaps it's one of those in-chip ECC types (like DDR5 memory). But what do I know. I take this stuff at the value it's printed. I would think it's kind of hard to accidentally type, "LPDDR4-3200 SDRAM with ECC" in official documentation.

 

[Ericloewe]

Perhaps it's one of those in-chip ECC types (like DDR5 memory)

Not inconceivable, but that's fairly expensive stuff for a jelly bean part.

I would think it's kind of hard to accidentally type, "LPDDR4-3200 SDRAM with ECC" in official documentation.

It's weird as hell, and I suspect there's an interesting story behind that...

 

[Constantin]

I too found the Product Selection Guide to be less than helpful. they label memory modules with ECC or parity memory but the memory component mentioned above features neither designation.

The SKU decoder in that selection guide doesn’t even contemplate DDR4 memory. So I wonder if there is a better data sheet to take a look at.

However, I doubt it’s a ECC component because samsung would presumably mention ECC prominently in the guide for that memory module section just like they did elsewhere.

 

[sretalla]

In making my original comment, I had read this:

Does the Raspberry Pi 4 have ECC LPDDR4 RAM? - Raspberry Pi Forums

www.raspberrypi.org

As far as I can see, it is "on-die ECC"... but I don't know that it practically equates to what we would call ECC... the end of that thread still leaves the question hanging for anyone who knows better to just say Yes (or no), but nobody has... for me the jury is out and it doesn't look good in real terms for the points made in that thread and for reasons mentioned above in this one.

 

[Ericloewe]

but I don't know that it practically equates to what we would call ECC

Easy answer is "no". At a minimum, you have an entire DRAM bus and memory controller that are not protected. Realistically, the on-die ECC is only there to get the error rates down below the specified threshold as DRAM cells shrink, probably correcting single-bit errors and ignoring two-bit errors to save a bit per line.
On top of that, if you can get Samsung ICs and if Samsung makes no claim of supporting this on their parts used on RPi units, you don't get even that much ECC.

 

[darkmode]

Well I'll take what I can get. Ultimately I just want a arm64 build of Core to tinker around with. Now I just have to find the time to install FreeBSD and perform a build.

My test hardware is a RockPi 4c with a sata hat and 2x5TB Seagate 2.5 drives. I may do a functional test as well with an nvme drive as a single disk pool.

 

[darkmode]

FYI I just discovered a FreeNAS build for ARM64.

PersonalBSD.org » FreeNAS Promo for AARCH64

FreeNAS Promo for AARCH64

personalbsd.org

Color me excited.

This is the million dollar answer to the thread topic.

 

[Constantin]

My test hardware is a RockPi 4c with a sata hat and 2x5TB Seagate 2.5 drives. I may do a functional test as well with an nvme drive as a single disk pool.

Should be interesting. Running at 1/2 the minimum RAM suggestion and on ARM. Good luck! Will be interested hear how far you get!

Also: those 5TB drives are almost certainly SMR. Beware of the performance impacts that may have.

 

[jgreco]

Should be interesting. Running at 1/2 the minimum RAM suggestion and on ARM. Good luck! Will be interested hear how far you get!

Also: those 5TB drives are almost certainly SMR. Beware of the performance impacts that may have.

Just to be clear, the words "minimum RAM suggestion" are incorrect. I raised the minimum REQUIREMENT to 8GB years ago. FreeNAS may seem to work on less than that, but there are a variety of problems you will face, and while it is probably okay for experimentation, I wouldn't trust it.

The SMR drives are fine for experimentation but will fall apart if placed to any serious use over time.

 

[darkmode]

Just to be clear, the words "minimum RAM suggestion" are incorrect. I raised the minimum REQUIREMENT to 8GB years ago. FreeNAS may seem to work on less than that, but there are a variety of problems you will face, and while it is probably okay for experimentation, I wouldn't trust it.

The SMR drives are fine for experimentation but will fall apart if placed to any serious use over time.

This ARM NAS is for non-trivial purposes. The TrueNAS instance that feeds my (production/homelab) infrastructure is filled with a bunch of 10TB and 6TB enterprise HGST drives.

 

[jgreco]

This ARM NAS is for non-trivial purposes. The TrueNAS instance that feeds my (production/homelab) infrastructure is filled with a bunch of 10TB and 6TB enterprise HGST drives.

Well, then please make sure it is placed in a role where failure or loss is not a problem, because this is not expected to end well with your existing ARM hardware manifest. I think the SMR drives are a bigger problem than the RAM, but the low RAM can lead to lockups or panics under load, which is admittedly highly dependent on the workload, but I also wouldn't be shocked to see swap ballooning and for you to run out of swap, which will tend to kill the NAS middleware.

 

[Constantin]

Just to be clear, the words "minimum RAM suggestion" are incorrect. I raised the minimum REQUIREMENT to 8GB years ago.

I am chastened and corrected!

Given the cost of a RockPi 4C, attendant hats, etc. I wonder if buying a old SM motherboard isn't a more viable / cost-effective solution for a non-trivial application. Or use some other flavor of ZFS already optimized for use on a Pi platform. Granted one loses the nice GUI, etc. but at least there will be some sort of knowledge-base on what does and does not work in a production environment.

 

[darkmode]

Well, then please make sure it is placed in a role where failure or loss is not a problem, because this is not expected to end well with your existing ARM hardware manifest. I think the SMR drives are a bigger problem than the RAM, but the low RAM can lead to lockups or panics under load, which is admittedly highly dependent on the workload, but I also wouldn't be shocked to see swap ballooning and for you to run out of swap, which will tend to kill the NAS middleware.

Thanks and noted. My current backup strategy is recurring ZFS snapshot puts to a Backblaze S3 bucket. I'm not worried at all about the data to be on this ARM build since it will be mostly resting and nothing wildly transient--that's what my TrueNAS instance is for.

Since I haven't heard back from the person who posted that ARM based FreeNAS video (had a few questions first), I'll very likely start out with a FreeBSD 13 instance on this RockPi box. I want FreeNAS/TrueNAS, but I really don't need it. All I want is another ZFS target to play around with.

 

[keithsplace]

I would like to run FreeNAS via Raspberry Pi for my website Silicophilic for a data backup project for the Website. Is it possible to run it from a Pi device? If yes how is the performance? Would there be any issue?

Maybe not with TrueNAS but this video shows a new device with sata drive support with a Pi that can be loaded with other OS’s To do a small NAS

[mod: removed YouTube link]

 

[jgreco]

Maybe not with TrueNAS but this video shows a new device with sata drive support with a Pi that can be loaded with other OS’s To do a small NAS

Please, no promotion of YouTube channels. There are hundreds of alternative choices for small footprint NAS. If this were 12 years ago and you were telling us about PogoPlug, that'd maybe be different.