Freenas don't use all the memory

[MoB]

Hi folks,

I am building a new FreeNas server based on a HP Proliant ml310e (Xeon E3, 2*500GB ZFS RAID 0, 8GB ECC RAM). The raid 0 choice is only for test purpose because it's getting the best of the drive actually.

I am performing test to fine tune my need. For the time being I am comparing my kind of high-end setup to my HP micro server Proliant N54L (N54L,8GB RAM 2*2TB+2*1.5Tb RAID1).

The issue is I have the same performance (sometimes less...) than on the N54L BUT only 2gig of ram is used by FreeNas with the Xeon setup, even when I am doing heavy operations on files.

I am quite surprised because same operations on the N54L draw 7Gig of rams (which seems normal behaviour with ZFS).

You can see the report, during files operations, below:

N54L (we can see the peek during files ops):

XEON E3 (maximum 3GB when same files are copied):

I can't explain the different behaviour , and I am hesitating to send back the 32Gb of RAM I have received...

So, any clue ?

 

[cyberjock]

Did you install the x86 version of FreeNAS on the machine that isn't using all of its RAM?

 

[MoB]

Nope sir, it was the 9.2.0-RELEASE-x64 !

 

[cyberjock]

Can you post the output of dmesg on the bad system?

 

[MoB]

Sorry for the big thing:

Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/home/jkh/checkout/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz (3093.04-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
Standard Extended Features=0x281<GSFSBASE,SMEP,ENHMOVSB>
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8218279936 (7837 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 8
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
acpi0: <HP ProLiant> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci5: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 1.1 on pci0
pci8: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci2: <ACPI PCI bus> on pcib3
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfacf0000-0xfacf03ff irq 21 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib4: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib5
bge0: <HP Ethernet 1Gb 2-port 330i Adapter, ASIC rev. 0x5717100> mem 0xfabf0000-0xfabfffff,0xfabe0000-0xfabeffff,0xfabd0000-0xfabdffff irq 16 at device 0.0 on pci3
bge0: APE FW version: NCSI v1.1.15.0
bge0: CHIP ID 0x05717100; ASIC REV 0x5717; CHIP REV 0x57171; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5717C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 9c:b6:54:12:88:dc
bge1: <HP Ethernet 1Gb 2-port 330i Adapter, ASIC rev. 0x5717100> mem 0xfabc0000-0xfabcffff,0xfabb0000-0xfabbffff,0xfaba0000-0xfabaffff irq 17 at device 0.1 on pci3
bge1: APE FW version: NCSI v1.1.15.0
bge1: CHIP ID 0x05717100; ASIC REV 0x5717; CHIP REV 0x57171; PCI-E
bge1: firmware handshake timed out, found 0x4b657654
miibus1: <MII bus> on bge1
brgphy1: <BCM5717C 1000BASE-T media interface> PHY 2 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge1: Ethernet address: 9c:b6:54:12:88:dd
pcib6: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci14: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 28.7 on pci0
pci1: <ACPI PCI bus> on pcib7
pci1: <base peripheral> at device 0.0 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf9000000-0xf9ffffff,0xfbfe0000-0xfbfe3fff,0xfb000000-0xfb7fffff irq 16 at device 0.1 on pci1
pci1: <base peripheral> at device 0.2 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x4000-0x401f irq 16 at device 0.4 on pci1
usbus1 on uhci0
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xface0000-0xface03ff irq 20 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
pcib8: <PCI-PCI bridge> at device 30.0 on pci0
pci20: <PCI bus> on pcib8
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0x10c0-0x10c7,0x10c8-0x10cb,0x10d0-0x10d7,0x10d8-0x10db,0x10e0-0x10ff mem 0xfacd0000-0xfacd07ff irq 17 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ichwd0 on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0 at port 0x930-0x937,0x960-0x97f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8-0x2ff irq 3 on isa0
wbwd0: DevID 0x00 DevRev 0x01, will not attach, please report this.
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 236300002100
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 236300002100
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 236300002100
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 236300002100
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe0002f87200 with IPv6 initialized (100409)
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
usbus0: 480Mbps High Speed USB v2.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
ugen2.1: <Intel> at usbus2
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen1.1: <0x103c> at usbus1
uhub1: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen0.1: <Intel> at usbus0
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3500820AS SD81> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST3500320AS SD04> ATA-7 SATA 1.x device
ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
cd0 at ahcich4 bus 0 scbus4 target 0 lun 0
cd0: <hp DVDROM DH51N IE00> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, SMP: AP CPU #3 Launched!
ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1546518015 Hz quality 1000
Root mount waiting for: usbus2 usbus1 usbus0
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
Root mount waiting for: usbus2 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen2.2: <vendor 0x8087> at usbus2
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
Root mount waiting for: usbus2 usbus0
uhub3: 6 ports with 6 removable, self powered
uhub4: 6 ports with 6 removable, self powered
Root mount waiting for: usbus2 usbus0
ugen2.3: <vendor 0x0424> at usbus2
uhub5: <vendor 0x0424 product 0x2660, class 9/0, rev 2.00/8.01, addr 3> on usbus2
uhub5: 2 ports with 1 removable, self powered
ugen0.3: <JetFlash> at usbus0
umass0: <JetFlash Mass Storage Device, class 0/0, rev 2.00/11.00, addr 3> on usbus0
Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
mountroot: waiting for device /dev/ufs/FreeNASs1a ...
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <JetFlash Transcend 8GB 1100> Removable Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 7728MB (15826944 512 byte sectors: 255H 63S/T 985C)
da0: quirks=0x2<NO_6_BYTE>
GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware

 

[cyberjock]

Ok, I got no good ideas, so let's shootin the dark, shall we?

-Download the RAM testing ISO and run it from www.memtest.org. See if it actually is testing the full 8GB of RAM. You don't need to let it complete any passes if you don't want to. There's no harm in it, except that it will take some hours to complete.

-Also try reseating your RAM. This fixed the problem for someone a few days ago that had a similar situation.

-Check out your BIOS settings and see if they are correct. In particular, look closely at any settings related to memory remapping and such. Also check you have the latest BIOS. Maybe try doing a "reset to defaults" in your BIOS and then setup the proper settings again.

 

[Dusan]

I have an idea. Did you enable autotune at one of the systems? Please check your Sysctls and Tunables. Are they the same on both systems?​

 

[MoB]

Hello and thanks for the help !

I ran memtests and checked the bios, no joy here.

Autotune wasn't turn on, on either system. Sadly after some reboot the full usage of memory was back; without performance improvement. So maybe a bug in the report interface ? It's very annoying to experience kind of non deterministic event like this...

Anyway, thank you again for the quick and helpfull hand.