Board recommendation?

[Chris Moore]

I'm using lz4 compression for the first time, and I'm getting a ratio of 1.00x - which to me reads that compression is doing exactly nothing. My thoughts are turn it off since there must be some hit somewhere on power consumption/ram/speed.

As you get more data in the pool, the ratio will increase. The algorithm for LZ4 quits trying if the data is not compressible, so it doesn't spend a lot of time working on it if it can't be compressed. It isn't going to help you with data that can't be compressed, but it does help with data that can be compressed.
One of the storage servers I am managing is reporting a ratio of 1.36x because the data is so compressible. It just depends on your data.
Here is an article about it: https://www.servethehome.com/the-case-for-using-zfs-compression/

 

[Chris Moore]

I did start out looking to replace 2 servers with a single server. And the gear for that was reasonably easy to find since I wanted to have a fairly powerful system to do two things VMs and NAS. I've been warned not to do this and that FreeNAS should have it's own box which has me looking for lower end hardware. Trying to find a lower end Supermicro board and CPU but also having it be able to take 32GB of RAM isn't so easy to find. Performance would likely be on par with the server as it is just now.

If this is for a home system, and you don't mind a little risk, there is no reason you can't use virtualization.
Did you read this very good write-up by @Stux :

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Also:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

 

[Halk]

The data in the pool was added by rsync so should have been compressed as it was added.

https://i.imgur.com/N2gk541.png

I had done a little reading and understand that lz4 is a low overhead compression technique for ZFS and it'll give up quickly. My understanding is that looking at how it's done with my data, it's done no good.

 

[Chris Moore]

Is it worth it? I understand ECC RAM is/could be beneficial but in my use case is it really that desirable? If the answer is yes then I will build the server...

I always use ECC, even when I built a desktop computer for my wife, it has Registered ECC memory. There is no reason not to.

 

[Chris Moore]

I had done a little reading and understand that lz4 is a low overhead compression technique for ZFS and it'll give up quickly. My understanding is that looking at how it's done with my data, it's done no good.

But is doesn't hurt to try. The CPU that is doing the compression (or not) is much faster than the disks.

 

[Halk]

If I didn't already have the server up and running on non-ECC hardware and I was to go out and buy it then I would buy ECC. But since I already have it and would essentially be doing a like for like replacement then that's when I wonder if it's worth ECC - and I don't know enough to answer that.

Edit : re-reading your comments you're saying I should move to ECC I think?

 

[Halk]

I've had another look - amazing what a few days off work can do to clear my head.

If I'm going for low end hardware, and fighting hard against my instincts to get a "bargain" of very powerful stuff that'd be redundant...

X9SCI-LN4F - This is a very basic supermicro board that takes Sandybridge and Ivy Bridge Xeons, (E3 [V2]) which are fairly basic Xeons. 4 cores and 3 Ghz roughly.

It takes a max of 4x8GB DIMMs for 32GB of memory.

I can get 32GB (8GBX4) SAMSUNG M393B1K70CHO-CH9 for £65 This isn't on the list of compatible/tested memory from Supermicro. They're looking for memory starting M391, my guess is it'll be fine.

I can find the board on eBay for £50.

https://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Xeon_E3-12xx_(uniprocessor)
https://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Xeon_E3-12xx_v2_(uniprocessor)

Thinking back I don't think Ivy Bridge really was earthbreaking over Sandybridge so as much as Ivybridge is preferable I don't think there's much in it.

The fastest ones are the most expensive. I don't think I'm ever going to feel the difference across that range really. I also think I'll need some kind of integrated graphics.. so a 1225 or 1245 V2. Or a 1225, 1235 or 1245 V1(they're not marked as V1) would probably be more reasonable for price.

Looking at eBay quickly for a 1245 shows £75 for a v1 (8048 passmarks) or £100 for a v2 (9122 passmarks)

It's also time to replace the 10 year old PSU. It's a good quality Corsair one but it's very old now and should be replaced. Do I need to be looking for anything specific connector wise when I'm buying it, or can I just get a high quality/efficiency at 600W and be fine?
Edit : Looks like it. The manual and board layout just look the like standard 24 pin motherboard and 8 pin CPU power connections, and any PSU I I look at has 20+4 and 4+4. Superflower Platinum King 550W. Plenty of power I think and I like that it's 80plus platinum and from a high end manufacturer.

EditEdit: Seems the board has onboard graphics... there's a blast from the past! And they don't recommend the xxx5 CPUs.
That means switching to a E3-1240V2 which is £80 and the same performance as the £100 1245V2.

Total £75 for PSU. £80 for CPU. £50 for board. £65 for memory = £270. This in my opinion is inexpensive.

There's an option to go for a 2011 socket board, but it adds £100 to the cost for the board right away, while opening the door for much more memory and more cores at broadly the same CPU prices. Certainly value for money but if I don't have a use for it I think I've found my level.

 

[jgreco]

I have an archive system (at work) that has 230 TB of disk capacity and using gzip-9 compression, we have over 280 TB of data in it with a reported free space of 100 TB. This system ran fine with 32 GB of RAM. I upgraded it to 128 GB of memory just to try and get a little more speed from it for those occasions when data needs to be accessed, but I can't see that it made a notable change.

Has much of it been rewritten? It would be pretty normal for it to be "fast" pretty much all the way through the first write cycle, as free space is contiguous in huge ranges. Where it typically falls apart is when space is freed and then reallocated. If you are freeing and reallocating ginormous amounts of space, that's probably the best case scenario. Basically what happens is that as the system has to work harder at hunting for free space, write speeds will fall precipitously. Read speeds are also impacted to a lesser extent.

https://forums.freenas.org/index.ph...tb-64mb-cache-sata-6-0gb-s.25810/#post-162517

It is important to note that for ZFS, the interesting crap rarely happens when things are "new". What you want to look for is performance down the road.

 

[Stux]

Not entirely. It's pretty valid at the smaller memory sizes, and back in the day when people would come here with their 4GB RAM systems, it was helpful to point to something that said "8GB minimum." But I've said for years that as you get out past 16GB that the rule softens a bit, and by 32GB you can easily do 2x the storage, probably 3x the storage, as long as you're not doing heavy workloads. This hasn't really changed. For an archival system, I'd expect you can do 100TB of disk on 32GB of RAM without issue. 16GB is definitely kinda tight, and the system will start to thrash if you get a lot of fragmentation.

With respect, I think what you’re saying is the rule is out of date.

With disk sizes where they are, and memory capacity, we’re beyond the 1:1 mapping the rule codes.

For all intents, since 8GB is the current minimum, common low/mid FreeNAS memory configurations are:

8GB, 16GB, 24GB, 32GB.

So, even at the next level (16GB) after minimum the rule is not correct any more. And I’m sure no one is saying that a min spec FreeNAS box is limited to 8TB right?

 

[Halk]

I'm going to order the stuff but I wouldn't mind if someone was just to reassure me tell me I'm not making a mistake with these parts.

Supermicro X9SCI-LN4F £50
(8GBX4) SAMSUNG M393B1K70CHO-CH9 £65
Xeon E3-1240V2 £80
Superflower Platinum King 550W £75 (new)

To go with an LSI 9207-8i (In IT mode correct firmware), 8x12TB Seagate Ironwolf Pro, 1x120GB Kingston SSD.

I don't think I've missed anything obvious....

 

[KrisBee]

Check the m/board memory type. You've picked registered ECC RAM, you need unbuffered ECC RAM. The giveaway is the price. The correct RAM will be more expensive even second hand if you can find 32GB.

 

[Halk]

Ah... well thanks for catching that. I'm used to non-ECC desktop stuff and didn't realise. Back to the drawing board!

So that rules out that board. The main problem with it is that it doesn't take RDIMMs and I'd have to use UDIMMs (I didn't realise there was such a thing as unbuffered ECC memory... shows what I know!)

The good news is that if I pay a bit more for a different board I can get one that does support RDIMMs.

Supermicro X9SRI-F - which has 8 memory slots as well as taking a faster processor.

 

[Chris Moore]

Supermicro X9SRI-F -

That board is crippled with missing PCIe slots. IF you want a board like that get this one:
Supermicro X9SRL-F
https://www.ebay.com/itm/132762499937

 

[Halk]

I've found 1 of each available, but there's a significant price difference. £163.85 for the SRI-F and £243 for the SRL-F. I can't see me using the PCIE slots - I'll put the LSI HBA in one of them, but beyond that I don't see what I'd use the other 2 for. The SRL-F has 7. Both of them seem to have enough SATA ports that if I wanted to reconnect the old 4x8GB drives I'd be able.

Even if for some reason I wanted a graphics card in there I could use the x16 slot for it and put the HBA in the x8 slot.

It may just be that there's far less stuff available in the EU than there is in the US, but I can't think of any possibility of using the extra PCIe slots and paying 50% extra for the board?

 

[Chris Moore]

If you wanted an NVMe SSD card like this:

https://www.ebay.com/itm/Intel-Opta...444074?hash=item23a9bac16a:g:-LQAAOSwQqJbkw5w

Then if you wanted to add a 10Gb Network card like this:

https://www.ebay.com/itm/MNPA19-XTR...X-2-10Gbe-NIC-10GBe-1m-SFP-Cable/132480615612

For a high speed connection to your main workstation.

There are things you might want to add later. That is why I went for the more expensive option.

 

[KrisBee]

It may just be that there's far less stuff available in the EU than there is in the US, but I can't think of any possibility of using the extra PCIe slots and paying 50% extra for the board?

+++++++++++++1

UK s/hand market cf to US is tiny and prices are (much) higher. RAM prices both s/hand and new have risen steeply over the last 18 months..

 

[Halk]

I see Chris.... not bad suggestions but if I follow that path I'll end up with a dual board 128GB and 20 cores at 3GHz just in case!

I won't go 10Gb ethernet in this property but when I move it's something I'll consider.. so it would be handy to have that as an option. For the server I'm very unlikely to add an m.2 drive or the equivalent. But as it stands that board will take 2 more cards so I think that gives me room should I change my mind even about both of those options.

If it was only an extra £10 or so then I'd do it but the board you linked won't ship to the UK and I don't think it's worth paying £80 for PCIe slots beyond 3. Again if the board only had 1 PCIe slot then I'd pay more for extra.

That leaves me with this in my box in eBay

Memory - 8x 8GB 64GB RDIMM ECC REG DDR3 1333 MHz Speicher f Supermicro X9SRi X9SRA X9SRL

There's a seller in Germany who sells the same stuff but lists it multiple times as all the different boards it's compatible with. I can find cheaper memory (slightly) but I'm keen to go for a similar model number to what's listed as compatible, and the ebay listing specifically guarantees compatibility.

Motherboard server Supermicro X9SRI-F LGA2011 for cpu E5-26xx/16xxv1 / v2

I'd like more PCIe slots. In fact the blanks on that board are an eyesore, but there's slim pickings in the EU so I'll just have to take what's there.

Intel Xeon E5 2690 V2 OEM 3.0GHz 10Core 25MB 130W 22nm 20Threads LGA2011 CPU

This is much more powerful than I need. I could go down to a cheaper CPU and I can't really justify why I need 10 cores at 3Ghz but that's probably why I spent more on my car than I needed... That CPU is £200 and I can find a E5-2620 V2 for £45. I realise I'm using the same justification not to get the other board... I may settle by trying to find the point at which the gains diminish on the CPU between the 2620 and the 2690.

£560 all in.

On top of that I'll buy the Superflower PSU, and as I said before it's probably more than I need to spend on a PSU but I'm not comfortable cutting back there.

That leaves me looking for a cooler - do Supermicro socket 2011 boards take standard socket 2011 coolers? If so it means I can get some kind of generic 120mm fan fitted aluminium tower cooler.

I think I may now be ready to go?

 

[Chris Moore]

It is unfortunate the supply problems and cost situation. Back in March I was able to buy two of the Supermicro X9SRL-F boards, NEW in sealed retail packaging, for $198 each and free shipping. The vendor had almost 200 available and a few days after I bought mine, someone else came along and bought 98 of them. They sold out within a few days.

 

[Chris Moore]

Intel Xeon E5 2690 V2 OEM 3.0GHz 10Core 25MB 130W 22nm 20Threads LGA2011 CPU

That is way more than you need for FreeNAS. What else are you planning to do with it?

 

[Chris Moore]

This is much more powerful than I need. I could go down to a cheaper CPU and I can't really justify why I need 10 cores at 3Ghz but that's probably why I spent more on my car than I needed... That CPU is £200 and I can find a E5-2620 V2 for £45. I realise I'm using the same justification not to get the other board... I may settle by trying to find the point at which the gains diminish on the CPU between the 2620 and the 2690.

There is a reason I went with the Xeon E5-2650 v2. It is more than enough, even for transcoding Plex video and the price (in the US) is only $85. I would spend the savings on a lower processor on the better board if it were me.