Buy used motherboard/CPU/RAM with ECC support

[idgar]

So after around 7 years of using FreeNAS/TrueNAS I've started to upgrade my hardware.
First, replaced hard drive with S.M.A.R.T errors, then tried to upgrade my CPU and unfortunately it fell on the motherboard socket and bent a pin. I was worried about it although tests went fine for a period of 1 day.
So next I took a better motherboard/CPU/RAM that I had aside and replaced it for the storage.
It has i7-4790 with 32GB 1600Mhz which is more than enough for my needs.

Now the more I read threads in this forum, the more I'm worried about the stability of the storage, so after I bought both LSI SAS controller + hotswap cage, I've started to run some more thorough testing - for CPU, RAM (Memtest86+) and dd repeatable read tests on large image files and verified sha256 output again and again (data is larger than RAM size so every cycle blocks are read from pools again).
dd went fine for about 2 days.
The problem is that memtest failed after 7 rounds in one test - one address that flipped 1 same bit 3 times.
Frustrated as I thought I'll be able to reuse old hardware and because what's written about the dangers in non-ECC systems I'm starting to think about ECC components.

I understand that what's needed is to replace CPU + Motherboard + RAM.

I need about the same spec as before (4 cores and 32GB). I'll have around 12TB usable pools (18TB raw), I guess recommended RAM is 8GB as base + 12GB for supporting the pools (please correct me if I'm wrong).

Would you suggest buying used ones? Is it safe enough based the assumption that ECC can detect errors and CPUs are mostly reliable and hard to break? (most important thing for me is to make sure data won't be corrupted)
I looked at ebay and found mostly supermicro X9SCL-F and X9SCM-M with a match E3-1230 v2 or E3-1265L v2 for lower watt consuption.
Not sure about which RAM brand and model is recommended.
Estimated price including shipping (I live in Israel and ebay shipping can be quite expensive for motherboards - around 60$-70$) is around 200$ (calculated one ECC 32GB UDIMM RAM I found).
If used are recommended, will be glad to hear if you have other suggestions.

And to make sure I'm going the right path - What is the advantage of registered / buffered ECC?

How much should cost new hardware with ECC support?

 

[BR14]

Nothing particularly wrong with SM X9 boards except the age. They should take e3-12* v3 with updated bios. E3-1220 should do fine. I understand that Xeon L-series are waste of money. So are e3-12*5 series as the board will not use the Intel graphics.

Most 8gb DDR3 ECC Udimms from Hynix, Samsung, and Micron (Crucial) should work. Kingston apparently could be problematic.

This is very well documented in older hardware guides in Resources. You should read these and check the specs on the Supermicro site.
Th SM X9 series used to be very popular, and a lot people on the forum have plenty of experience and expertise with these - perhaps they could provide better guidance if you have any specific questions after reading the guides.

 

[idgar]

Thanks @BR14 ,
I'll keep in mind about the CPUs. Does the motherboards without BMC/IPMI will work with CPU models discarding GPU? (thinking to buy with BMC unless recommended otherwise so just in case).

About used motherboard I'm curious how many bought ones (ebay or other recommended sites) and are they satisfied. Will be glad to hear about it.

There are several hardware guides in resources, is this the right link?
https://www.truenas.com/community/resources/hardware-recommendations-guide.12

 

[ChrisRJ]

I looked at ebay and found mostly supermicro X9SCL-F and X9SCM-M with a match E3-1230 v2 or E3-1265L v2 for lower watt consuption.

Forget about the L model. It has a lower TDP, but that does not equate to lower power consumption in a NAS scenario. It only means you can live with a smaller CPU cooler.

Not sure about which RAM brand and model is recommended.

You need to look those up on the Supermicro site. If you want to be sure that things work, you need to choose exactly(!) one of the listed modules. The slightest deviation in type may (or may not, but you never know upfront) cause the RAM to not work.

What is the advantage of registered / buffered ECC?

RDIMMs are way cheaper thn UDIMMs, because the former is in wide use in servers while the latter is not. Also, RDIMMs allow larger amounts of RAM to be used. UDIMM is often limited to 32 or 64 GB.

How much should cost new hardware with ECC support?

New server gear is really expensive. Adding a zero to the amount you quoted will likely not be an exaggeration. A used board like the X9 (not older!) is a good start. Although, today I would probably try to find an X10. But I run 3 X9 boards myself and I am really happy with the price-performance ratio

 

[BR14]

There are several hardware guides in resources, is this the right link?
https://www.truenas.com/community/resources/hardware-recommendations-guide.12

Yes, both JGreco and CyberJock are knowlegeable board's veterans.

I would try getting a MB+CPU+memory combo because it is very difficult to determine which of the componets is at fault without having spares to swap.
I think you get rudimentary VGA of the Matrox vintage regardless of IPMI. If you need better video you would need a separate video card.
SuperMicro boards are supposed to be very reliable, but remember that x9s series came out around 2011. I bought a used x9scl-f combo 4 years ago and it still runs.

 

[idgar]

You need to look those up on the Supermicro site. If you want to be sure that things work, you need to choose exactly(!) one of the listed modules. The slightest deviation in type may (or may not, but you never know upfront) cause the RAM to not work.

Looked at the site but it gives me like 1 or 2 DIMM models. For x10 it doesn't even print out the vendor name (https://www.supermicro.com/en/suppo...=83446&type=DDR3&ecc=1&reg=0&fbd=0&cpu=others). Are there other sources?

BTW, X10 allows also 1.35V DIMMS, is it something worth considering? on laptop I would seriously consider it but for server not sure what's the point, maybe less heating...

New server gear is really expensive. Adding a zero to the amount you quoted will likely not be an exaggeration. A used board like the X9 (not older!) is a good start. Although, today I would probably try to find an X10. But I run 3 X9 boards myself and I am really happy with the price-performance ratio

By this quote I assume you bought used one right?
Are there good sites other than ebay to find this hardware?
For now I'm looking at X9SCM-F and X10SLL-F.
Does X10 has real advantage over X9?
as I saw, it mainly provides the following:
- USB3 ports
- PCIE 3 x16 slot (but for x8 support).
- newer CPU with AVX2 (does it even being used for compression in TrueNAS?) but with similar specs.

Are there advantages for X9? Looks like the majority here use it. Not sure if due to price or other concerns (reliability?).

I would try getting a MB+CPU+memory combo because it is very difficult to determine which of the componets is at fault without having spares to swap.

The only combo I found so far is with memory or CPU, not both. Again, maybe I'm not searching in the right place.
The next problem is that by getting a combo, if the price pass 75$ boundry I need to pay extra taxes... but maybe it I can find something a bit above this price the taxes will worth it.

And just to make sure, are those motherboards work with any standard chassis and power supply? Does it consumes much more watts compares to desktop ones?
My chassis is an old silverstone one (desktop) and can hold up to ATX motherboards.
I have Thermaltake smart SE 530W semi modular, it has 87% efficiency - not a gold + standard but close to 80+ bronze.
Works for couple of years, hope it is a good candidate.

 

[Etorix]

By this quote I assume you bought used one right?

Yes, but as you may see in @ChrisRJ 's signature, his X9 are Xeon E5 systems, with RDIMM—lots of cheap RDIMM.

Are there good sites other than ebay to find this hardware?

Whatever small ads site is relevant to your place. There's some IT in Israël, so there must be a market for second-hand gear, and companies specialising on refurbishing and reselling discarded server hardware. You "only" need to find them…

And just to make sure, are those motherboards work with any standard chassis and power supply?

Server motherboards in standard mini-ITX, micro-ATX, ATX, CEB or EEB fit in consumer cases.
Motherboards in proprietary formats only fit in the corresponding proprietary chassis.

Does it consumes much more watts compares to desktop ones?

If power draw is a concern, get a (properly sized) Platinum or Titanium rated PSU. A NAS idles most of the time, so efficiency under low load is key (and the relevant CPU figure is power draw at idle, not TDP or power draw under maximal load).

 

[Redcoat]

Here's an X9SRiF in Israel fot $100 https://www.ebay.com/itm/1548008626...1Ck2BMZwS12LeUSVXnM7gFi6cfMYXEmRoCyasQAvD_BwE

 

[idgar]

OK, so after looking at the options, trying first to find combination of X10SLH-F / 32GB from QVL / E3-1271 v3 (although can go down up to E3-1230 v3).

I tried to find full combo of hardware from one seller that will both tested and RAM from QVL but couldn't find one yet.

The only full combo I found tested (said by technician) was X10SLL-F / 32 GB samsung / E3-1230 v3, but seller cannot supply details about RAM exact model.

On the other hand found (all tested only by plug it and power on to ensure basic operation, have 100% positive feedbacks and from Europe / US):
1. X10SLH-F with E3-1271 v3 from one seller (100% positive feedback)
2. 32GB from other sellers:
2.1. Hynix HMT41GU7BFR8A-PB
2.2. Samsung M391B1G73QH0-YK0 (also couple of options found from hong kong tested with memtest86 from top rated sellers)
2.3. Micron MT18KSF1G72AZ-1G6E1(also options from hong kong tested with memtest86 from top rated seller)

Are you usually expecting hardware to be tested more throughly (CPU stress, memtest) or is it uncommon?

So I need assistance with the following please:
1. Would you buy separately from 2 sellers assuming basic tests has passed? what about the option from hong-kong with memtest passed for RAM part?
2. Is there an advantage to buy full combo of all hardware from the same seller when only basic test has passed? and what about when not knowing the RAM exact model?

 

[ChrisRJ]

1. Would you buy separately from 2 sellers assuming basic tests has passed?

What is "basic tests" for you? In general buying from multiple reputable sellers has never been a problem for me.

what about the option from hong-kong with memtest passed for RAM part?

What is the background or critical point for this question? I am trying to find out what you are uncertain about.

2. Is there an advantage to buy full combo of all hardware from the same seller when only basic test has passed?

Again, what is a basic test? If it is a reputable seller, this has the advantage of things having been tested in combination, if (and only if) the seller says this was done. Don't assume the seller has tested things together when it is not explicitly mentioned.

and what about when not knowing the RAM exact model?

I don't understand that.

In general I would always try very hard to buy from someone in my country, because returning things is easier and, if necessary, the legislation is clear.

 

[idgar]

What is "basic tests" for you?

Attaching the required hardware, powering on and check that it turned on and optionally run a simple program, but not a through test.

I don't understand that.

For example when seller says the motherboard includes samsung DIMMs but I don't know if those are part of the QVL.

In general I would always try very hard to buy from someone in my country, because returning things is easier and, if necessary, the legislation is clear.

Unfortunately couldn't find in my country. tried to find an israeli website for used server hardware without success...

 

[Etorix]

I would not worry too much about the QVL for memory modules and assume that ECC DIMM with suitable characteristics from Micron/Samsung/SK Hynix will just work fine.