Large discrepency between my real and available memory. Where do I start trouble shooting?

[JTheNASBuilder]

My server has 24GB of ECC-RAM in it. The sticks are physically there, the sticks passed Memtest with flying colors, but FreeNAS doesn't use all the memory (even though it sees it all).

In the FreeNAS GUI it indicates that there is 16GB of system RAM. When I run the following command:

grep memory /var/run/dmesg.boot

I get the following output:

real memory = 25773998080 (24580 MB)
avail memory = 16505761792 (15741 MB)

So the RAM passed Memtest (and was accounted for at the time, all 24GB of it) and it's listed as "real memory" in the output.

How do I get the real and available memory into alignment with each other?

 

[cyberjock]

Hardware, FreeNAS version, all that good stuff we ask for in the forum rules would be really helpful.....

 

[JTheNASBuilder]

Whoops, alright well the information in the first post stands as my Title/Description bit and here's the additional useful information; no error information because it isn't throwing any errors related to memory (to the best of my knowledge, the log files for FreeBSD are... dense to say the least).

FreeNAS Version: 9.2.1.7-RELEASE-x64

Chassis: Supermicro SC836TQ-R800LPB

Motherboard: Supermicro X8DTH-iF

CPU: 2x Xeon E5540 2.53 GHz Quadcore

RAM: 6 x 4096MB DDR3-1333 ECC sticks / 4 are Kingston and 2 are Micron brand for what that's worth

RAID CARDS: 2x LSI 9240-8i cards flashed to 9211-8i IT passthrough mode for FreeNAS/ZFS use.

HDDS: 6x Western Digital Red 4TB (model: WD40EFRX)

RAID Configuration: 6 4TB drives arranged in RAIDZ2 on ports #0-#5 on LSI HBA 1

Extra Misc. Hardware: At the moment an external USB drive is mounted to perform a file dump (2TB WD Mybook)


So... as far as my hardware is concerned, I can't see an issue. The motherboard and CPU both support 100GB+ of RAM. The hardware, per the command line memory check, sees the RAM. But for whatever reason FreeNAS isn't using it. I noticed the issue earlier this week and initially assumed the perhaps FreeNAS was setting aside memory for the jails I'd set up (for Crashplan, etc.) but the value displayed in the GUI didn't change after I made another couple jails and I realized that my total install was running off 16GB and not 24GB.

Before I posted here for help I searched for quite awhile to find a solution but came up with nothing useful. Here are the semi-relevant but not problem-solving links I found yesterday; I couldn't glean anything useful from them but maybe someone with a better knowledge of FreeNAS might see something I didn't:

Server Fault Post: FreeNAS not using all installed RAM
--- Super old version of FreeNAS ; commentor says problem original poster is experiencing was fixed in v. 8*

FreeNAS Forum Post: Only half of installed RAM available
--- Relevant; my memtest and bios both show the full amount of RAM though. I'm tempted to recheck both after breakfast, however even though I happen to have taken a photo of the memtest screen and know my mind isn't playing tricks on me.

 

[SweetAndLow]

If you take out the micron and never the Kingston does it still display 16g? Then try taking it just 1 micron then try taking out just one Kingston. I have a feeling mixing memory is causing issues, even though that sounds strange.

 

[JTheNASBuilder]

If you take out the micron and never the Kingston does it still display 16g? Then try taking it just 1 micron then try taking out just one Kingston. I have a feeling mixing memory is causing issues, even though that sounds strange.

I took out the Micron memory and then the system memory (as displayed by the BIOS/MemTest) was 16GB and the FreeNAS memory was also 16/6 (instead of 16 used but 24 available). After that I ended up downloading the motherboard manual and studying the configuration models they have in the memory installation section. On a whim, I reversed the order of the RAM modules so that instead of Micron - Kingston - Kingston in the first three channels it was Kingston - Kingston - Micron.

For reasons unclear to me, reversing the DIMMs solved the problem. Now the output of the command:

grep memory /var/run/dmesg.boot

reads like this:

real memory = 25773998080 (24580 MB)
avail memory = 24826417152 (23676 MB)

Which fits perfectly with the information in the manual regarding how much of the system memory is reserved for the system (and accounts for the very minor discrepancy); the GUI in FreeNAS also reflects the new increased amount of RAM.

So, to leave a trail for those that come after me:

1. Carefully study your motherboard's manual. Although ultimately it wasn't studying the manual that saved the day in my case, it did reveal that there were very specific rules about RAM placement based on the RAM's voltage, the CPU configuration (and type) and other important details.

2. Even if you're following the manual's instructions to the letter sometimes it's something as quirky as simply trading the DIMMs into new channels (reversing the order of the DIMMs and putting the odd-man pair I had in the last channel instead of the first saved the day).

3. I have absolutely no insight into why this worked or into why the entire time my BIOS could see the RAM (and FreeNAS could see the RAM according to the "real memory" output) but FreeNAS only saw and used the full amount of RAM after I shuffled the sticks around.

 

[pschatz100]

It is because the timing of all the memory sticks are not identical, and the mother board is sensitive to which ones are first. You didn't give any information about your memory, so it is impossible to comment about the specific sticks. However, the more sticks you have installed, the greater the chance that a timing issue will show up. This will be most common when mixing brands from different manufacturers, but can also pop up mixing different batches from the same manufacturer.

All memory sticks are built to a certain tolerance - they may be a little "off" relative to their published performance characteristics. In fact, in the real world this is quite likely the case. One does not see the problem as much with DDR3 memory as was the case with older memory types, but the potential for problem is still there. Back in the old DDR days, manufacturers used to provide memory sticks in matched sets to minimize the problem.