how to manually install Graylog 5.#

[dwchan69]

I manage to manually install Graylog 4.3.x to a TrueNAS 13.0 jail using pkg update and "pkg install -y p5-Digest-SHA pwgen mongodb44 elasticsearch7 graylog". However, is it possible to install or push Graylog 5.x? I mean what the repository URL I can see if Graylog is even an option (I recall you used to be able to do that with RPM on RedHat), Or is this tie back to a repo that TrueNAS managed? Any insight would be appreciated

 

[jgreco]

TrueNAS jails are basically just FreeBSD hosts. You can check under http://www.freebsd.org/ports for additional details on the current status of a given port. Packages are basically just precompiled versions of ports. This is documented on the FreeBSD doc site.

Chapter 4. Installing Applications: Packages and Ports

FreeBSD provides two complementary technologies for installing third-party software: the FreeBSD Ports Collection, for installing from source, and packages, for installing from pre-built binaries

docs.freebsd.org

which also talks about FreshPorts, pkg-search, how repos are managed, and related issues. Since I don't use packages, I only have vague familiarity with that stuff, so I realize this may not be entirely helpful to you. But you did say "any insight".

 

[Whattteva]

I manage to manually install Graylog 4.3.x to a TrueNAS 13.0 jail using pkg update and "pkg install -y p5-Digest-SHA pwgen mongodb44 elasticsearch7 graylog". However, is it possible to install or push Graylog 5.x? I mean what the repository URL I can see if Graylog is even an option (I recall you used to be able to do that with RPM on RedHat), Or is this tie back to a repo that TrueNAS managed? Any insight would be appreciated

FreshPorts -- sysutils/graylog: Tool for centralized log collection

Graylog is a centralized log server that lets you search and analyze logs using a web interface or a REST HTTP API. It handles many input types and sources, both structured and unstructured, and stores data in OpenSearch, either single node or cluster.

www.freshports.org

According to this, the current version available on the ports tree is 5.0.2.

To answer your question: Yes, you can, but you will have to update your pkg source from "quarterly" to "latest", which is a very simple process explained in the following page in chapter 4.4.2. https://docs.freebsd.org/en/books/handbook/ports/#pkgng-intro

After following those steps, you should end up with this:

Code:

root@playground:~ # cat /usr/local/etc/pkg/repos/FreeBSD.conf
# $FreeBSD$
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
root@playground:~ # pkg search graylog
graylog-5.0.2                  Tool for centralized log collection

You can even build from source using the ports tree if you need some exotic compiler flags... But that's a topic for another day and I'm sure you probably don't want to busy yourself with such labor-intensive and involved process.

Don't you just love FreeBSD and its very well-written handbook?

@jgreco You basically nearly got him almost all the way there!!!

 

[dwchan69]

FreshPorts -- sysutils/graylog: Tool for centralized log collection

Graylog is a centralized log server that lets you search and analyze logs using a web interface or a REST HTTP API. It handles many input types and sources, both structured and unstructured, and stores data in OpenSearch, either single node or cluster.

www.freshports.org

According to this, the current version available on the ports tree is 5.0.2.

To answer your question: Yes, you can, but you will have to update your pkg source from "quarterly" to "latest", which is a very simple process explained in the following page on chapter 4.4.2. https://docs.freebsd.org/en/books/handbook/ports/#pkgng-intro

After following those steps, you should end up with this:

Code:

root@playground:~ # cat /usr/local/etc/pkg/repos/FreeBSD.conf
# $FreeBSD$
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
root@playground:~ # pkg search graylog
graylog-5.0.2                  Tool for centralized log collection

You can even build from source using the ports tree if you need some exotic compiler flags... But that's a topic for another day and I'm sure you probably don't want to busy yourself with such labor-intensive and involved process.

Don't you just love FreeBSD and its very well-written handbook?

@jgreco You basically nearly got him almost all the way there!!!

Thanks, a lot of info and been years since I touch FreeBSD. So in short, I am changing / redirecting the jail VM where to look for as its main Repo?

 

[Whattteva]

Thanks, a lot of info and been years since I touch FreeBSD. So in short, I am changing / redirecting the jail VM where to look for as its main Repo?

That's correct. Also, a jail is NOT a VM although it may appear like so.

 

[Patrick M. Hausen]

I'd recommend not to edit /etc/pkg/FreeBSD.conf but instead just override the settings:

Code:

mkdir -p /usr/local/etc/pkg/repos
echo 'FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest" }' >/usr/local/etc/pkg/repos/FreeBSD.conf

 

[Whattteva]

I'd recommend not to edit /etc/pkg/FreeBSD.conf but instead just override the settings:

Code:

mkdir -p /usr/local/etc/pkg/repos
echo 'FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest" }' >/usr/local/etc/pkg/repos/FreeBSD.conf

Uh.... the very first line in my code tag should tell you that there is no overriding of the sort taking place......

root@playground:~ # cat /usr/local/etc/pkg/repos/FreeBSD.conf

Neither does the handbook in chapter 4.4.2 state to do that. I'm not sure where you're getting that impression....

 

[Patrick M. Hausen]

Ah - sorry, missed that. OK, the finer point is - you do not need to copy the entire file. It's possible to override just single settings like e.g. the URL.

 

[jgreco]

@jgreco You basically nearly got him almost all the way there!!!

Yeah, yeah, I know, it's just that I don't use pkg for anything and so it's much easier to just point people at the correct documentation when I don't really have a complete answer. Over here, we have a fiat list of ports installed on all FreeBSD systems, and specific server applications such as Apache are installed from original project sources.

 

[Whattteva]

Yeah, yeah, I know, it's just that I don't use pkg for anything and so it's much easier to just point people at the correct documentation when I don't really have a complete answer. Over here, we have a fiat list of ports installed on all FreeBSD systems, and specific server applications such as Apache are installed from original project sources.

That's curious. Is that because you guys modify the source so it behaves slightly differently or is it because the package isn't built with the correct feature flags?

 

[dwchan69]

Good news and back news. I was able to pkg update to the latest repo and upgrade graylog to 5.# However, in the process, something broke

With that being said, I am going for a do-over, but curious about the following
1. Doing a new jail using the 13.0 release.
2. update the repo from quarterly to latest right out of the gate
3. install Graylog 5.* right out of the gate. But will I need to install all its dependencies also, like what I used with the default quarterly repo

pkg install -y p5-Digest-SHA pwgen mongodb44 elasticsearch7 graylog

or will the package manager handle the installation for all its dependencies like MongoDB, Elastic, etc

 

[jgreco]

That's curious. Is that because you guys modify the source so it behaves slightly differently or is it because the package isn't built with the correct feature flags?

The pkg system is possibly nice for beginners who are looking for a quick and easy way to get a system up and running in short order.

It is not, however, a particularly good idea in other ways. It is no good at building complicated servers; it spams crap throughout the host's base OS, making it extremely difficult down the road to determine what's been installed. At one point, FreeBSD had three or four different Apache ports, one for SSL, one for (IIRC) fastcgi, etc. And these are moving targets as time goes on.

A few of you here may be aware that I created the first freely available FreeBSD appliance OS, intended to boot off floppy to create FreeBSD-based Xterminals (for X11 window system), and that evolved into PicoBSD when Andrzej Bialecki took it and generalized it a bit. That was the precursor to NanoBSD, which underpinned FreeNAS in the early days. I've included a few links to convince you that I'm crazy.

Anyways, I have a long history of appliance-ified UNIX systems, and part of this is the idea that the system should not have crap spammed throughout it, which is typically what many BSD and Linux package managers have done. It makes it very unpleasant to identify what has been done to the system.

So instead, I split the system up into a base OS, it'd look familiar to you, except that it is generated by an automated build system from a database of configuration directives designed to make it relatively easy to take a host up to the next release of FreeBSD. This is hardened, firewalled, IDS'd, and secured. This host then sponsors applications, in a root subdirectory (such as /www, /pgsql, /postfix, etc.) where the application is installed from the ground up, only the necessary files; all libraries and dependencies for the app are compiled within the tree and no external links are allowed. Most of these are also jailed, and the environments do not include a /bin/sh, so remote stack smash attacks are really hard, and most other types of attacks fail too.

As for ports, they're marginally useful. Stuff like iperf, kermit, rsync, gmake, etc., are handy to have in the base. Other stuff such as quagga/frr are usually a bit rotted and may need a custom patched port. In any case, the system builder is responsible for both building a consistent system image, and then doing baseline configuration of the generated system for all that pesky stuff that people often forget. The builder is over 300KB of Bourne.

Anyways, it ends up giving you a very clean system design; you might even manage to figure out where stuff is just from df:

Code:

Filesystem       1024-blocks    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs       555036  223436  287200    44%    /
devfs                      1       1       0   100%    /dev
/dev/gpt/usrfs       1120540  640072  390828    62%    /usr
/dev/gpt/lclfs        902300  658432  171684    79%    /usr/local
/dev/gpt/varfs       1765276   71588 1552468     4%    /var
/dev/gpt/homefs       643336     428  591444     0%    /export/home/u0
tmpfs                  65536       8   65528     0%    /tmp
devfs                      1       1       0   100%    /var/named/dev
/dev/gpt/pgsql       8104668   80096 7376200     1%    /pgsql
/dev/gpt/pgsql-c     8104668       4 7456292     0%    /pgsql/conf
/dev/gpt/pgsql-d     8104668   41568 7414728     1%    /pgsql/data
/dev/gpt/pgsql-l     8104668       4 7456292     0%    /pgsql/logs
/dev/gpt/www         8104668 2343400 5112896    31%    /www
/dev/gpt/www-c       8104668     508 7455788     0%    /www/conf
/dev/gpt/www-d       8104668   44144 7412152     1%    /www/data
/dev/gpt/www-l       8104668     752 7455544     0%    /www/logs

Just a webserver I happen to be working on setting up.

 

[Whattteva]

pkg install -y p5-Digest-SHA pwgen mongodb44 elasticsearch7 graylog

or will the package manager handle the installation for all its dependencies like MongoDB, Elastic, etc

pkg should handle the dependencies automatically and you should only need to do

Code:

pkg install -y graylog

 

[dwchan69]

OK, stay tune. Wish the community graylog plugin is working which be make this a whole lot easier

 

[Whattteva]

Anyways, it ends up giving you a very clean system design; you might even manage to figure out where stuff is just from df:

Code:

Filesystem       1024-blocks    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs       555036  223436  287200    44%    /
devfs                      1       1       0   100%    /dev
/dev/gpt/usrfs       1120540  640072  390828    62%    /usr
/dev/gpt/lclfs        902300  658432  171684    79%    /usr/local
/dev/gpt/varfs       1765276   71588 1552468     4%    /var
/dev/gpt/homefs       643336     428  591444     0%    /export/home/u0
tmpfs                  65536       8   65528     0%    /tmp
devfs                      1       1       0   100%    /var/named/dev
/dev/gpt/pgsql       8104668   80096 7376200     1%    /pgsql
/dev/gpt/pgsql-c     8104668       4 7456292     0%    /pgsql/conf
/dev/gpt/pgsql-d     8104668   41568 7414728     1%    /pgsql/data
/dev/gpt/pgsql-l     8104668       4 7456292     0%    /pgsql/logs
/dev/gpt/www         8104668 2343400 5112896    31%    /www
/dev/gpt/www-c       8104668     508 7455788     0%    /www/conf
/dev/gpt/www-d       8104668   44144 7412152     1%    /www/data
/dev/gpt/www-l       8104668     752 7455544     0%    /www/logs

Just a webserver I happen to be working on setting up.

Looks nice and clean, but kinda' defies FreeBSD Hier and the custom arbitrary directory structure reminds me of why I loathe Linux , unfortunately.

That being said, I do understand the drive behind it to make a hardened lean and highly secure web server though.

 

[jgreco]

Looks nice and clean, but kinda' defies FreeBSD Hier and the custom arbitrary directory structure reminds me of why I loathe Linux , unfortunately.

FreeBSD has no mechanism at all for dealing with this. FHS at least has /opt, which covers Linux, Solaris also does /opt, etc. The big deal here is that you can create new root level directories on almost any OS.

The goal is to put things in consistent locations in a consistent design. You might be able to guess where webserver data is stored, or PgSQL logs are stored, or their configuration bits can be found. Can you do that for FreeBSD? No. It's been a moving target over the years. As an added bonus, the app partition itself can be made read-only to make it even more secure. This has secondary and tertiary benefits such as ease of filesystem expansion in virtualized environments as well.

Anyways, if you were really offended by this, you should be objecting to the nefarious misuse of /mnt/ by TrueNAS, so I'm just going to troll you a bit before blowing off your complaint as specious. Heh. ;-)

 

[Whattteva]

The goal is to put things in consistent locations in a consistent design. You might be able to guess where webserver data is stored, or PgSQL logs are stored, or their configuration bits can be found. Can you do that for FreeBSD? No. It's been a moving target over the years.

That's interesting cause I think this is actually one area where FreeBSD does well.
Configuration files, for example, are pretty reliably in /etc/ or /usr/local/etc/. Logs are typically found in /var/log/.
I do agree that things like webserver data could be a moving target, but usually, you can find that information either in the /usr/local/etc/ configuration file or the rcvar setting on /etc/rc.conf.

Anyways, if you were really offended by this, you should be objecting to the nefarious misuse of /mnt/ by TrueNAS, so I'm just going to troll you a bit before blowing off your complaint as specious. Heh. ;-)

Haha, offended is a strong word... perhaps more like intrigued.
Curious, why do you think TrueNAS's use of /mnt/ nefarious? /mnt/ is kind of a place for general mount points, though I do understand that it's generally used for a less permanent file systems like removable disks or network mounts, but it's not entirely inappropriate.

 

[jgreco]

That's interesting cause I think this is actually one area where FreeBSD does well.
Configuration files, for example, are pretty reliably in /etc/ or /usr/local/etc/. Logs are typically found in /var/log/.
I do agree that things like webserver data could be a moving target, but usually, you can find that information either in the /usr/local/etc/ configuration file or the rcvar setting on /etc/rc.conf.

I once ran across a client's box that had

/etc/apache2/httpd.conf
/etc/apache/httpd.conf
/etc/apache-ssl/httpd.conf
/usr/local/etc/apache2/httpd.conf

and I believe a few others too. The problem gets to be identifying where crap is coming from, and this gets more fun when there are also multiple copies of Apache httpd installed with all sorts of mess. This was Linux, I believe, but I saw a similar disaster during the early years of FreeBSD as well, as there were multiple maintainers for ports who had radically different ideas about how to do things.

/mnt/ is kind of a place for general mount points

No, it isn't. According to hier(7), which you just quoted before,

"/mnt/ empty directory commonly used by system administrators as a temporary mount point"

In TrueNAS usage, it is neither empty nor temporary.

 

[Whattteva]

I once ran across a client's box that had

/etc/apache2/httpd.conf
/etc/apache/httpd.conf
/etc/apache-ssl/httpd.conf
/usr/local/etc/apache2/httpd.conf

and I believe a few others too. The problem gets to be identifying where crap is coming from, and this gets more fun when there are also multiple copies of Apache httpd installed with all sorts of mess. This was Linux, I believe, but I saw a similar disaster during the early years of FreeBSD as well, as there were multiple maintainers for ports who had radically different ideas about how to do things.

Yes, this is very common in Linux. Linux tends to put everything in /etc/. Occasionally, some applications will play nice and put it in /usr/local/etc. But in general, there is no such separation between base OS and third-party and they just put everything in /etc/. Of course, this is kinda' proper I suppose, as there's no such thing as "base OS" in Linux in the first place.

On modern FreeBSD, only stuff bundled in base OS can go to /etc/ /bin/ /usr/bin/ etc. All extra stuff installed through ports must go under /usr/local/ and I've observed this since probably FreeBSD 8. Of course, I can't vouch for the early days. In your example with Apache, it should only reside in that last path (/usr/local/etc/apache24/httpd.conf) and the www root is at /usr/local/www/apache24/ but nothing should reside outside of /usr/local/.

No, it isn't. According to hier(7), which you just quoted before,
"/mnt/ empty directory commonly used by system administrators as a temporary mount point"
In TrueNAS usage, it is neither empty nor temporary.

Yes yes, I know, which is why I put more nuance in my post that you unfairly cut out!!! It's certainly not ideal placement since it's not temporary, but it's not completely off base like putting it under /usr/bin/ or something.

 

[dwchan69]

Interesting, I got a number of notification as follow. Fair to assume go ahead and add them?
Message from python39-3.9.16:

--
Note that some standard Python modules are provided as separate ports
as they require additional dependencies. They are available as:

py39-gdbm databases/py-gdbm@py39
py39-sqlite3 databases/py-sqlite3@py39
py39-tkinter x11-toolkits/py-tkinter@py39

Message from graylog-5.0.2:

--
Please see /usr/local/etc/graylog for sample versions of graylog.conf, log4j.xml, and
and adjust them for your configuration.

For GeoIP support you need to install the net/GeoIP port and
configure the path to the GeoIP databases in the Graylog Web Interface.

When running graylog in a jail, you need to set enforce_statfs for the jail.

For a single-node installation, install:

- databases/mongodb50 or higher
- textproc/opensearch 1.3, 2.0 or higher

And ensure that the elasticsearch cluster name matches that used by graylog.

See https://go2docs.graylog.org/5

p.s.
How does one set enforce_statfs for the jail.?