In the following article, I will share with you what I learned when I have converted my servers from Debian to FreeBSD. I will explain the common problems you could have, what you should consider before changing the system and what motivated me for this “adventure”.
To begin this article, I need to set up the background, so that you can fully understand what follows. I am a system administrator in a small company. I work every day with two developers using my servers. Their knowledge about operating systems is quite good. I had to renew a whole set of servers, used for a high availability service for our biggest client.
So far, everything is running with Debian GNU/Linux, but I am always talking about FreeBSD to my colleagues. This time, I proposed switching to FreeBSD for our new servers!
What follows is about decisions, choices and “philosophy”. This article is not a how-to switch from Debian to FreeBSD. It may help you decide what to use; it may help you find answers; but you won’t find any commands or scripts.
I am trying to write this article while being as neutral as possible. I may make some mistakes in my choices and reasons, but they are mine. You can send me an email if you want to share a thought or if you think that I said something wrong.
Why change?
I knew that I would need a good reason to switch operating systems, because this will certainly create extra work with regard to compatibility, finding packages, and learning the operations of the operating system. That is why I made a Pros/Cons list to explain why I would like to use FreeBSD for the next few years. I tried to be as honest as possible, not trying to sway you to FreeBSD with only pros, while Linux would have had only cons. Linux has been working well, so why change it?
Pros and Cons list:
• FreeBSD and Linux are equivalents and they basically perform the same. As we are using open-source software only, we should not encounter compatibility problems, but we must be aware that if we need to use proprietary tools, it may not work.
• Switching to FreeBSD may not be that easy. We may have to change some of our source code and may need to teach people new operations or features. Some user commands may be different.
• FreeBSD is a Unix system, keeping Unix philosophy. System administration does not change with everynew release. Linux, over the last few years, has been evolving and moving away from the Unix philosophy. Every new release changes many components, or includes more services and applications. I have the feeling that things may become more complicated in the future.
• FreeBSD is able to provide high-availability service, with two lines of config per host. We do not have to struggle with heartbeat/pacemaker/corosync.
• FreeBSD is using a port tree. Compilation takes time, but it will be centralized with the software Poudriere. We will be able to choose compilation flags, the versions of postgresql and PHP that we want to use. We have had surprises with Debian in the past: old PostgreSQL, PHP missing some features.
• FreeBSD is developed as a complete package and is coherent, with man pages that are up-to-date. The FreeBSD handbook is a good source of documentation and best practices also.
As you see, my list is mostly talking about philosophy concerns rather than performance or technical differences. In my company, we like to keep things simple, and we have the feeling that Linux is going in the opposite direction.
The deal
My boss accepted the idea that we move to FreeBSD. We agreed that this should work with as few modifications as possible of scripts / software / makefiles / users habits, etc. For example, For example, I do not have time to fix every Bash script, but adding a symlink could fix the problem -> hello /bin/bash! I really care about my users (the developers); if they want Bash, I have no problem with it and I will provide them with Bash with the same alias as before.
The migration: from Debian to FreeBSD
In this part, I shall share some issues that I had, and how I fixed them.
The Shell
An important part of the system is the shell. By default, FreeBSD uses the unfriendly csh, which does not keep history and no color. I will not be able to advocate FreeBSD to my users with just csh. I decided to install bash as we are used to and importing our bash config files.
Where are my commands?
You would think that people would understand that if you make a change to the operating system, then they will need to learn about this, but they don’t. A developer asked me “The command free doesn’t work, why?”
Under Linux, this command shows you information about memory usage. I just told him: “free does not exist in FreeBSD. You can use the vmstat command and you will get everything you need. You can also use top.” Users will have to learn a few new tools.
What’s wrong with putty?
On Windows, you are likely to use the ssh client Putty. I have to admit that Putty with the default FreeBSD profile is not user friendly at all. A lot of keys are not doing what you would expect them to do. For example, typing “home” or “end” will display a “~”, typing “delete” will add a character, console dialogs (like the one in ports or the frame in tmux) are made of weird characters etc…
After a quick search on the Internet, the fix was made on the server-side. People are now happy with their Putty.
Paths of death
One extremely important change between the systems is the filesystem hierarchy. In FreeBSD, configurations files are separated between /etc and /usr/local/etc, while Debian stores everything in /etc. The folder /var/www does not exist, it is replaced by /usr/local/www. Packages binaries are under /usr/local/bin/.
Everything installed on the system with packages is installed in /usr/local/. Files outside this folder come from the base system.
B.1 Compilers
Our work uses compilers and we rely on C and C++. We had a problem with a lot of makefiles, using the compiler “gcc”. “gcc” command was a 4.6 GCC on the previous system, while on FreeBSD “gcc” is a 4.2.1 version. If we add a new compiler, its full name must be used e.g.: gcc46 for GCC 4.6 etc.
I had to fix manually some scripts where “gcc” was not compiling, due to compatibility problems.
In addition to the outdated GCC, I have been struggling with include and libraries paths too. The linker could not find the libraries located in /usr/local/lib instead of /usr/lib as before. The same kind of error was happening with includes files.
After some hours, I decided to use an environment variable telling GCC to add a path by default to fix the problem. I know this solution makes the makefile not very portable because the paths will not be right without the environment variable. But, the makefile was not actually portable, and we do not plan to share our code.
We have scripts!
We have a lot of scripts, doing various things. Every script
is using #!/bin/bash!
You can imagine how boring it would be to fix every script manually. That is why someone wrote a “sed” script. I preferred to link Bash to /bin/bash, which is the path that our previous scripts were expecting. People did not have to waste time searching why their scripts using /bin/bash do not work.
Please, when you write scripts, use #!/usr/bin/env bash instead of bash full path. And if you use Linux, if you link to /bin/sh, be sure to really be using /bin/sh and not a disguised Bash.
FreeBSD crash course
When the time comes to change users tools, at least, be kind. Explain to them what changed and why you did it and teach them the basis of their new tool. Like I said before, the command “free” does not exist, binaries are in /usr/local/bin/ path, services scripts are either in /etc/rc.d or /usr/local/etc/rc.d, /etc/rc.conf is the main configuration file etc. My users were happy to learn a few things about their new FreeBSD system, they did not feel lost or abandoned. Take time to explain to them what changed, give them some time to learn and it is all good.
Where are my packages
I had some fun (or bad news, I do not know) with the packages. We are mostly using open-source software, or our own programs.
Hello, I am an old closed source Linux library
I found ONE program relying on a 1997 closed source Linux library that we MUST use. Even on the previous Debian server, we had to do some hacks to make it work! This library is very important to our business and we must use it. If we cannot use it, bye bye FreeBSD.
Luckily, I got it working with Linux emulation and some dirty hacks that I would have preferred to skip. Also, in a few months, we will trash it because it is getting replaced.
This is a bit frustrating to spend a lot of time getting a program working, when you know it will be removed soon.
Hello, I am away from ports
The other case is funny too. I was installing the dependencies of our application and I knew exactly what was needed, installing them one by one.
…Until that moment when I could not find the latest library in ports! I have been looking on my favorite search engine “mylibrary + freebsd”, and found no answer.
I have been cursing the developers. “Where did you find this? No one is using it! It’s not even in the freebsd ports!” And then I had to learn how to write a port that compiled. (I know I should share my port, but it is an ugly one actually).
Surprises!
Now, all my dependencies are installed, services are running awaiting to be checked. (Yes, you should really consider checking if everything works as expected when you change your operating system and your servers!
Then, when everything is done, someone comes to tell you he found something strange.
My program A does not work as expected
We have a program that stopped terminating correctly on FreeBSD. This seems to come from a difference with threads, and exiting was not killing all the threads. We made a little change so the application was terminating correctly with Ctrl-C.
My program B consumes 30% less memory
I do not have an explanation for this one. One of our applications consumes a lot of memory by loading shape files (with libshape), and with the same data, the new FreeBSD consumes 30% less memory than the previous Debian system.
I assume this difference is coming from the shapelib, which is newer on the FreeBSD system, and which is compiled with optimizations for the server itself. We were very happy to see this behavior!
Valgrind needs to be recompiled…
To debug the application, I was telling before, we used valgrind. After a day spent running the program under valgrind, we were not able to use the results – valgrind gave us an error. Valgrind has been monitoring too much memory that a compile-time limitation was reached. We had to recompile Valgrind with a modification in the source code to increase a value.
I have been happy to tell the developers that ten minutes later, I compiled valgrind from ports with a patch of mine, and that it was installed on the servers!
Conclusion
In a nutshell, I would say that FreeBSD gives us more freedom than our previous systems. I can really provide the developers with the tools they want.
But, when it comes to using proprietary tools, I have to admit that FreeBSD may not be the best option. I have not had this problem (except for one library).
If I would have been using Solaris, NetBSD, or another Linux distribution, I think I could have had a working result. But FreeBSD comes with good performance, server hardware compatibility, a complete handbook, incredible stability, control over packages and a clean and well-designed system.
About the Author
Charles Rapenne is a Linux, and now FreeBSD system administrator in a small company in France. He enjoys using *BSD systems and writing applications in Common LISP. His servers rely only on open source software.
This article was re-published with the permission of BSD Magazine. To Learn More about iXsystem’s commitment to open source check us out here: https://www.ixsystems.com/about-ix/
