anyone had gone down the FreeNAS route on home grade equipment
Everyone and their grandma's dog. And the experience is:
- It usually works
- When it doesn't, it's probably that Realtek Ethernet, which can be replaced with an Intel i210 or similar add-in board
-
If you have a memory failure, you won't know - lots of time spent chasing that down
- Ryzen in particular requires C6 states / CoolNQuiet be disabled, some other tweaks to make sure PSUs don't spin down
- Some REALLY old boards won't even boot the installer, though that's never been chased to root cause AFAIK
- Some REALLY crappy SATA on-boards are crappy SATA on-boards, though that shouldn't be too common any more. Do not ever trust SATA add-ins.
- If your on-board SATAs aren't reliable (rare but happens), you can sidestep the issue with an M1015 off eBay, and going through the learning curve of cross-flashing it to P20 IT mode, then using SAS to SATA breakout cables. With cables, maybe USD 50? Thereabouts.
- If you use USB sticks as boot media, you're likely going to regret it. If you use USB 3 sticks at USB 3 speeds, you will come to regret it.
- 9 out of 10 issues with FreeNAS not working right are hardware related
- IPMI and iKVM are god sends. If you've never used it, and experience it, you will not EVER want to build on a consumer board without it, ever again
- What you do depends on where you are in your socioeconomic journey. A lot of us are in the enviable state of being gainfully employed and having enough disposable that we can buy cheap server stuff off eBay. A day or two spent troubleshooting a bad RAM stick is more costly to us than buying DDR3 Registered ECC on eBay for 15-20 a stick. And, that might not be where you find yourself. Or, you just want to get a feel for FreeNAS first, and splurge for that 350-400 dollar "used server kit" build down the road. Those are all completely legit things to do. You're already doing the right thing by asking questions and thinking it through so that building on consumer kit is a conscious, deliberate choice, not just a "well it was there" default that might be regretted later.
For what it is worth, while I agree with the greybeards here that server grade kit is way nicer and saves you so much trouble, I don't have religion about it. I think you can absolutely build on consumer kit if that's what you already have. Know the potential trouble spots ahead of time and accept that you may need to deal with them.
When it comes to buying new or used: A server grade build won't be appreciably more expensive new, and is dirt cheap used. There, going with "server grade" as the default makes a lot of sense.