Reporting Database Full

[Chris Tobey]

Hello,

I am running into an issue where the reporting database is full, and I am not sure how to proceed.

I am running FreeNAS 11.1-U6.

It is clear that /var/db/collectd/rrd is full, but how can I increase it? In System > System Dataset, my System Dataset Pool is set to SG1, and Syslog and Reporting Database are both checked.

Code:

root@stargate:~ # df
Filesystem                                                   1K-blocks        Used       Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-f962a6faf7146399176f845cd687c089   111220088     1165328   110054760     1%    /
devfs                                                                1           1           0   100%    /dev
tmpfs                                                            32768       10660       22108    33%    /etc
tmpfs                                                             4096           8        4088     0%    /mnt
tmpfs                                                         67059436     1642816    65416620     2%    /var
freenas-boot/grub                                            110062780        8020   110054760     0%    /boot/grub
fdescfs                                                              1           1           0   100%    /dev/fd
SG1                                                         9065162304         351  9065161953     0%    /mnt/SG1
SG1/users                                                  18306563485  9241401532  9065161953    50%    /mnt/SG1/users
SG1/.system                                                 9065622734      460781  9065161953     0%    /var/db/system
SG1/.system/cores                                           9065176842       14889  9065161953     0%    /var/db/system/cores
SG1/.system/samba4                                          9065180391       18438  9065161953     0%    /var/db/system/samba4
SG1/.system/syslog-e2eccb3703ad46d2b19f2e4809443384         9065467438      305485  9065161953     0%    /var/db/system/syslog-e2eccb3703ad46d2b19f2e4809443384
SG1/.system/rrd-e2eccb3703ad46d2b19f2e4809443384            9065706772      544819  9065161953     0%    /var/db/system/rrd-e2eccb3703ad46d2b19f2e4809443384
SG1/.system/configs-e2eccb3703ad46d2b19f2e4809443384        9065295886      133933  9065161953     0%    /var/db/system/configs-e2eccb3703ad46d2b19f2e4809443384
tmpfs                                                          1048576     1048576           0   100%    /var/db/collectd/rrd

/var/log/messages is filled with these:

Code:

May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190518.1400-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190519.1900-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190519.1900-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190520.1600-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190520.1600-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190520.1200-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190521.1500-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190521.1500-4w): No space left on device
May 22 12:10:02 stargate collectd[11620]: check_create_dir: mkdir (/var/db/collectd/rrd/stargate/df-mnt-SG1-users-.zfs-snapshot-auto-20190521.1500-4w): No space left on device

 

[Chris Moore]

What device is SG1? Is that you storage pool because it doesn't make sense that it would be full. When did you reboot the system last?

Do you have your system configured to keep snapshots? How much space is that taking up?

Please share with us the full output of zpool list -v

 

[Chris Tobey]

Hi Chris,

Here is the zpool list -v:

Code:

root@stargate:~ # zpool list -v
NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
SG1                                     43.5T  29.5T  14.0T         -    34%    67%  1.00x  ONLINE  /mnt
  raidz2                                21.8T  15.3T  6.42T         -    33%    70%
    gptid/55ef2498-ac64-11e5-b6ed-0025905800c4      -      -      -         -      -      -
    gptid/56ce660c-ac64-11e5-b6ed-0025905800c4      -      -      -         -      -      -
    gptid/57c2220a-ac64-11e5-b6ed-0025905800c4      -      -      -         -      -      -
    gptid/b61ba108-3f7e-11e9-a885-0cc47a1e4734      -      -      -         -      -      -
    gptid/59a42bc4-ac64-11e5-b6ed-0025905800c4      -      -      -         -      -      -
    gptid/5a9d3eee-ac64-11e5-b6ed-0025905800c4      -      -      -         -      -      -
  raidz2                                21.8T  14.1T  7.61T         -    36%    65%
    gptid/f47a731e-ffea-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -
    gptid/f58f6ee3-ffea-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -
    gptid/f699307f-ffea-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -
    gptid/f79e727b-ffea-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -
    gptid/f8aab7e9-ffea-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -
    gptid/f9c2c4df-ffea-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -
log                                         -      -      -         -      -      -
  gptid/0d56f7b3-c8f7-11e8-a885-0cc47a1e4734   348G  22.2M   348G         -     0%     0%
freenas-boot                             111G  2.58G   108G         -      -     2%  1.00x  ONLINE  -
  mirror                                 111G  2.58G   108G         -      -     2%
    gptid/166272df-def1-11e5-8ae9-0cc47a1e4734      -      -      -         -      -      -
    gptid/0407fa6e-e241-11e5-ab2b-0cc47a1e4734      -      -      -         -      -      -

 

[Chris Tobey]

Code:

root@stargate:~ # uptime
12:33PM  up 228 days, 17:07, 2 users, load averages: 1.30, 1.15, 1.18

 

[Chris Moore]

The zpool list shows that you have 14T of free space on SG1, so it isn't full... I would try a reboot if it isn't too inconvenient.

 

[Chris Moore]

I am running FreeNAS 11.1-U6.

You might also consider updating to U7... Unless you are worried it will break something.

 

[Chris Tobey]

There are 50+ servers using the storage, so it will have to wait for a weekend if a reboot is required.
Is there anything specific in the update?

 

[Chris Moore]

Is there anything specific in the update?

There were actually two updates. U6.3 and U7. The 6.3 update was out of cycle because of some important security patches.

Here is the change log from 6.3:

Code:

26131     Add "Auxiliary arguments" field to middleware
26695     Use 4k jumbo cluster pool for all jumbo frames until the allocator problem is fixed for the 9k jumbo cluster pool
28063     Write local SID to correct DB file
29020     Encrypt cloud credentials in configuration database
30696     Fixes for Windows AD User Base entries
32055     Remove warning that vfs_full_audit may cause transfer problems
33054     Automatically create bridge with default route for iocage jails
33453     Fix unnecessary AD restarts caused by enabling service monitor
33645     Add ability to stop winmsa from changing owner
34134     Fix corruption of first byte in AFP_AfpInfo stream/xattr in Samba
34264     Add support for configuring a custom S3 endpoint
34276     NVMe SMART monitoring workaround
34396     Use FreeNAS-specific fork of iocage
34459     Recompile minio to allow S3 service to start
34522     Allow reset of SED password in UI
34603     Fix traceback when setting IPMI VLAN ID
34684     Fix quota exceeded emails being sent every minute
35215     Fix smart.nawk script in freenas-debug SMART section
35404     Fix race condition in SMART
35973     Remove faulty enabled/disabled logic from freenas-debug
37143     Remove unnecessary pam_sss errors from /var/log/auth.log
37878     Add sysctls to disable winbind and sssd enumeration
37902     Remove 'net usersidlist' from freenas-debug
38898     Improve text of unauthorized email alert message
39178     Ensure that U6 installs the correct version of the Guide
39628     Fix function for storing SIDs
40572     Allow Samba to also listen on loopback when specifying a Bind IP
40636     Allow /etc/find_alias_for_smtplib.sh to be used by sudoers
40640     Monitor mdnsd and restart if necessary
40644     Get samba_server status when checking if AD started
40648     Load catia VFS module before zfs_space and zfsacl
40664     Log netatalk messages to /var/log/afp.log
40668     Regenerate /etc/resolv.conf when disabling Domain Controller service
40672     Write out pam configuration files in /etc/pam.d/ if they don't already exist
40680     Kerberos authentication fixes for LDAP servers
40684     Allow NIS to be ID provider for Active Directory
40688     Fix typo and lack of clarity in NTLMv1 warning
40692     At boot, have MDNS wait for /etc/resolv.conf to be available and valid
40696     Remove leftover NT4 code
40704     Do not grant extra privileges to users when a Directory Service is enabled
40708     Add unix_primary_group and unix_nss_info to idmap_ad configuration to address how Samba now handles groups
40716     Disable SMB1 by default
40720     Replace (nss|pam)_ldap with nss-pam-ldapd
40724     Allow unsigned 64-bit serials for certificates which allows AFP auth against macOS LDAP/KRB5
40728     Add experimental sysctl to enable multichannel Samba
40768     Revert recent zilstat commit as there is no zil_lwb_write_start() function in FreeNAS 11.1
40968     Fix traceback when trying to add a Cloud Sync task
41028     Patch FreeBSD CVE-2018-6922 "Resource exhaustion in TCP reassembly"
41044     Fix traceback when creating an internal certificate
41052     Bug fix for zvol/dataset traceback
41244     Fix check to see whether the SMB service is started
41272     Properly encode passwords in inadyn config
41332     Check for iocage host release being less than jail release when creating regular jail
41385     Update Samba port to address August CVEs
41410     Fix "AttributeError: 'Undefined' object has no attribute 'call'` traceback
41772     Add patch to address FreeBSD-SA-18:10.ip
41880     Improve service status wording in freenas-debug

You can get all the same updates by going to U7 but there were supposed to be some additional bug fixes. I am not able to find the difference between U6.3 and U7 because the change log appears to be the same between the two. Probably a documentation oversight.
Here is the path to the documentation if you want to look: https://download.freenas.org/11/11.1-U7/
I am still using U7 at work and the upgrade from U6 to U7 was completely painless. I have nothing on the server besides pure storage.

 

[Chris Moore]

Is there anything specific in the update?

Here is a better link to show the changes:
https://www.ixsystems.com/documentation/freenas/11.1/intro.html#u6