Just an update on my experience with this server in case someone else is inclined to go the same route. I have to say it was pretty painless, except for one issue (highlighted below).
I bought the server, as I described it in the first post, for $599. Subsequently I:
1) Replaced the supplied with a SAS 9207-8i, in IT mode, that I got on eBay for $59.
2) Removed the Intel quad port NIC - I'm just using the onboard 1GbE port until I upgrade to 10GbE someday soon.
3) Installed 2 120GB SanDisk SSD Plus drives (2x$20.99), connected to onboard SATA with spiffy bundled SATA cable ($8.99) (
https://smile.amazon.com/gp/product/B00VI3MFMO/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1 ), and the Supermicro tray ($11.39)
4) I bought 8 Seagate ES.3 3TB SAS drives on eBay for $312
5) Transferred about 15TB of my data from my Windows PC - that freed up 4 3TB SATA drives.
6) Installed another 64GB or ECC memory thanks to the find on eBay by "Death Dream" - $96
7) Bought a lot of 6 3TB IBM/Seagate SAS drives for $145.50, and installed them an 2 of my SATA drives as a second vdev (all raidz2).
8) Transferred the rest of my data, freeing up 4 4TB Seagate IronWolf drives.
9) Bought 5 4TB Seagate ES.3 SAS drives for $299.50, combined with 3 IronWolf SATA drives and installed as a 3rd vdev.
So, in summary, I spent $1,580.36, and now I have a commercial grade server with 32.29TB of total pool space configured as 3 8 drive raidz2 vdevs.
Some observations:
Running memtest on 128GB of memory and badblocks on all those drives took FOREVER. on the 5 new (to me) 4TB drives badblocks took 4 days to finish one pattern, so I decided that was good enough and stopped it. I have spare drives of both sizes, so I'll just swap anything if it fails.
Getting into the BIOS on the Supermicro motherboard is just about impossible. I got in once, and I don't even know exactly how it happened - i haven't been able to get back in again.
I ended up using both the "import disk" fjunction in the GUI and figuring out how to mount my NTFS disks with ntfs-3g - the GUI is pretty useless at providing status, and if you have a crash (I did - we had thunderstorms), it will not re-import the same disk...
Even with all these disks my new (to me) Tripp-Lite UPS says the system is drawing about 440W, so that's really not bad.
Now to my one issue...
When I installed Freenas I used the two SSDs in a mirrored config. Everything went fine and then a day (maybe) later I got an alert that the boot pool was degraded because of an error on the /dev/ada1 SSD. SanDisk doesn't have a diagnostic for BSD, so I pulled the two drives, installed them in a Windows box and ran the SanDisk test utility. I also ran a variety of Windows-based tools to try to stress test the drives. The utilities , and SMART, show no errors. I reinstalled in the Unix box, reinstalled Freenas, and within a day it dropped /dev/ada1 again. I have tried multiple SATA cables, no change. Any ideas?