1st storage server - FreeNAS!

[Sir SSV]

If you're building a new system you should go with freenas 11.

Sent from my Nexus 5X using Tapatalk

I read that there are still some small bugs with FreeNAS 11 and for important/valuable data, should stay with 9.10.2?

 

[SweetAndLow]

I read that there are still some small bugs with FreeNAS 11 and for important/valuable data, should stay with 9.10.2?

Not true. Do you have links or posts? FreeNAS 11 has been very uneventful release and most people should update especially now that U1 has come out.

Sent from my Nexus 5X using Tapatalk

 

[nightshade00013]

Yep 11 is fine the only reason why some people are waiting for 11.1 is because of the jail manager changeover. But even if you are going to try 9.10 and then upgrade you will have the same things to deal with when you swap to 11.1 if you have created any jails. But you can also take a different view and have more opportunities to learn how to configure a jail, ya know practice makes perfect....

But I switched from 9.10 to 11 a couple weeks ago and have had zero issues with it and sent my step daughter home with a 1U with FreeNAS 11 installed fresh about a month ago.

 

[danb35]

I read that there are still some small bugs with FreeNAS 11 and for important/valuable data, should stay with 9.10.2?

I'm also not aware of any such bugs or issues. I expect that at least part of what you're seeing is (justified, IMO) caution after the FN10 fiasco.

 

[Sir SSV]

I'm also not aware of any such bugs or issues. I expect that at least part of what you're seeing is (justified, IMO) caution after the FN10 fiasco.

You are correct. I read different views about FreeNAS 11 and people were cautious transitioning after FreeNAS 10. I have since downloaded 11.0 -U2 and will be installing onto my server today.

I will have more updates sometime today. Completed plenty of fan/cable management so there is plenty of airflow in the case.

I've read different views in regards to connecting the SFF8087 cables to the backplane. Is it necessary to connect 2 or will 1 do the job? Some people are saying that 2 seems to increase the bandwidth however there is no evidence that this is true...

 

[danb35]

I've read different views in regards to connecting the SFF8087 cables to the backplane. Is it necessary to connect 2 or will 1 do the job?

I'd say follow the manual for the backplane. For my backplanes, there's no bandwidth aggregation, so I have one cable to the front backplane, one to the back.

 

[Sir SSV]

Can anyone give me any assistance, I have my Ethernet cable connected into the Intel 10gbe network card but cannot connect to the Supermicro ipmi interface unless the network cable is connected to one of the gigabit LAN ports on the mainboard. Is this normal?

 

[Stux]

Yes.

By default the IPMI is active in the solitary RealTek port with failover to the first LAN port. Which is gigabit.

You can change this behavior if you want in bios, but IPMI is not supported on any ports except the management or first gigabit.

In the enterprise there is separate management network, and thus separate cables.

 

[Sir SSV]

Big update time :)
Spent all day tidying up cabling and getting fan speeds in check. That dreaded oscillating fan speeds were driving me up the wall

So, here we have a nice, new bare Supermicro SC846 chassis

Time to rip everything out and keep only what is necessary :)
Install mainboard, CPU, cooler

For the CPU cooler to fit, make sure to change the mounting bars to the included NM-XFB5 90°. Should only take a couple minutes.
I used Thermal Grizzly Kryonaut thermal paste. Went on easy enough with the included syringe. I used the included nozzle and followed their guide of coating the whole CPU but when the Noctua fan was tightened down it sort of oozed the thermal paste. So, quick remove of cooler, clean up the thermal paste, add a nice drop to the middle and reseat. Much better :)

Swapped out the 1200w psu's for these 920sq psu's and damn they are whisper quiet. Unfortunately they arrived like this

Who on gods earth ships psu's without protecting in bubblewrap?!
Anyway, used long nose pliers to bend back in shape and a perfect fit

Now, some people over on servethehome reported fitment problems with the pdu in the sc846 chassis. Mine went in perfectly and maybe Supermicro did a revision to the pdu in later chassis. For older pdu's, you must use the 8284 version. My 8824 is on revision 6.3

Installed the Noctua fans and tidied up the cabling

I did order a ssd cage (Supermicro MCP-220-84603-0N) but it is yet to arrive so this will do for now :)

Now, spent most of today trying to diagnose the dreaded oscillating fan issue that plagues this mainboard/chassis when changing out fans. You must use ipmi tool which is included with FreeNAS. Install FreeNAS on your storage media of choice (mine is the Intel SSD). No need to complete any configurations just yet except enabling SSH and setting root password. Click on Shell and a new window opens.

For all fans, you must set both lower and upper thresholds using

• ipmitool sensor thresh "sensor name" lower *lnr* *lcr* *lnc*
• impitool sensor thresh "sensor name" *unr* *ucr* *unc*

Then make sure to reset ipmi via Supermicro ipmi website going to Maintenance -> Unit reset or power down server, disconnect power cables for a couple minutes and reconnect/start server.
see this thread for all information

https://forums.freenas.org/index.ph...nge-ipmi-sensor-thresholds-using-ipmitool.35/

Big thanks to Ericloewe for all the hard work :)

Here are my fans and subsequent sensor settings for a whisper quiet chassis


NF-A8
Lower: 200, 300, 400
Upper: 2200, 2300, 2400

NH-U9DX-i4
Lower: 100, 200, 300
Upper: 1600, 1700, 1800

NF-F12 PPC (3000pwm)
Lower: 200, 300, 400
Upper: 3100, 3200, 3300

I had a major problem with my fans continuing to oscillate even after the thresholds were set. The root cause was a 3-1 pwm fan splitter used on the FANA header. I ended up using the included Noctua fan splitters (2 splitters to connect 3 fans) and the problem was fixed. Shame it took most of the day to diagnose :(

Now loaded up Memtest86 and fingers crossed ram is ok! I did set the fan speed in the Supermicro ipmi to Heavy I/O to get the fans working a little harder.
So far, my temps are as follows

CPU: 46 degrees
RAM: 34 degrees
Quite happy so far and hoping things continue to progress well :)

 

[Sir SSV]

Small update: left Memtest86 running all night with no errors then realised I was not running Memtest86+ *slaps head*

Downloaded latest version and got it running on the server now.

Is it normal for the chipset and memory to read unknown?
Btw, iKVM Viewer in Supermicro IPMI is awesome :)

 

[nightshade00013]

More than likely because the chipset and memory info is newer than memtest version you are using (5.01) so it may not be able to correctly pull the information. The latest version of Memtest is 7.3

Regular memtest will work fine though it is a little slower for some stuff since it is single threaded only. The "pro" version is the only one that you can use to actually do ECC tests.

 

[Stux]

Your memory should be fine. I wouldn't use any more time on running the various forks and versions of memtest86 ;)

 

[Sir SSV]

So even though the "free" version of Memtest86+ doesn't do ECC tests, I should still be good running the current version?
Btw, how long do the tests need to run? I read Cyberjock say to run the test for 1-2 weeks?

 

[Stux]

So even though the "free" version of Memtest86+ doesn't do ECC tests, I should still be good running the current version?

Yes.

Btw, how long do the tests need to run? I read Cyberjock say to run the test for 1-2 weeks?

Typical overkill.

https://www.passmark.com/forum/memt...are-good-11-passes-10-hours?p=37961#post37961

Its ECC ram, after a few passes you've verified it, and chances are it's not going to change. And if it does ECC should catch it.

You need to check the IPMI event log for ECC errors after running the test.

 

[Sir SSV]

Yes.



Typical overkill.

https://www.passmark.com/forum/memt...are-good-11-passes-10-hours?p=37961#post37961

Its ECC ram, after a few passes you've verified it, and chances are it's not going to change. And if it does ECC should catch it.

You need to check the IPMI event log for ECC errors after running the test.

Awesome I'll give the ipmi log a check once completed.

I did notice this morning when I checked the log I was getting a lot of entries to do with FANA header hitting the lnr value or something to that effect. Do you know if this is normal considering I have changed the thresholds for the Noctua fans?

Maybe I've gone a little too low on the lower thresholds?

The fans attached to FANA header were operating at their normal range (1900rpm on Heavy I/O) whilst carrying out Memtest with no oscillating.

 

[Stux]

I set my lower thresholds to 0,0,200 for Noctua 300rpm min fans

When set up right, you shouldn't have fan errors.

 

[Sir SSV]

I set my lower thresholds to 0,0,200 for Noctua 300rpm min fans

When set up right, you shouldn't have fan errors.

Oh wow, is that for your NF-F12's? Do you get any log entries in the Supermicro ipmi log?

 

[Stux]

Oh wow, is that for your NF-F12's? Do you get any log entries in the Supermicro ipmi log?

Yes, no (not anymore)

But the SM Optimal setting can set fans below 30% and that will cause Noctuas to stall.

 

[Sir SSV]

Yes, no (not anymore)

But the SM Optimal setting can set fans below 30% and that will cause Noctuas to stall.

Thanks for the info, did not know that. Do you run your fans on Standard speed setting?

Once Memtest finishes, I will change the lower fan speed to your settings and see how I go :) The errors I am seeing in the log for FANA are:

• Lower Critical - Going low - Assertion
• Lower Non Recoverable - Going Low - Assertion
• Lower Non Recoverable - Going Low - De-assertion

I just checked the system log, no errors to report for the ram which is good. Test has been running for 5hrs 18mins and is currently on Test #10.

 

[Stux]

I run my fan controller script. It works by putting the fans in max mode and then controlling their duty cycles directly.

Standard and optimal modes control the fans based on CPU temperature. Heavy IO is the same as Standard, except FANA is set to 80% (iirc)

You will want to run a fan controller script if you want the drive fans to spin up/down to cool your HDs.

https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/