How to build FreeNAS 9.10?

Status
Not open for further replies.

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Yep, looks like it's FreeBSD 11 needed now.

I'll update the doc once I've gone through things and verified it's ok after testing. Hopefully in the next few days. :)
 
Last edited:

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
I'll nuke my build system and re-install it as FreeBSD11 and try it also. I'll let you know.
 

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Well, I tried a few ways (root-on-ZFS, UFS), ran out of disk space once, out of swap another time. Eventually I was able to reproduce the exact error message I was getting before from js-freenas.dispatcher.

I had been running the build in a tmux session, and when it was done I had the following message on the machine console:

Code:
kernel: Failed to write core file for process conftest (error 14)


In /var/log/messages I found this:

Code:
Dec 21 19:43:10 tripod kernel: pid 54829 (try), uid 0: exited on signal 10 (core dumped)
Dec 21 21:52:24 tripod kernel: Limiting closed port RST response from 255 to 200 packets/sec
Dec 21 21:53:06 tripod kernel: Limiting closed port RST response from 227 to 200 packets/sec
Dec 21 22:05:31 tripod kernel: Failed to write core file for process conftest (error 14)
Dec 21 22:05:33 tripod kernel: pid 70403 (conftest), uid 0: exited on signal 11
Dec 21 23:08:15 tripod kernel: pid 19707 (readonly.exe), uid 0: exited on signal 11


So now I'm wondering if it has something to do with my particular version of vSphere that underlies this virtual build machine. I'm running ESXIi 5.5.0-1623387 which is quite old at this point, and possibly not fully compatible. Unfortunately I cannot readily upgrade this box to the latest version of vSphere but I'm guessing that that is what might be necessary at this point.

I'll set up an actual piece of hardware for this build box at some point, then try again.
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Yeah, that's probably worth a shot. I haven't updated the build docs to say "use FreeBSD 11" yet, nor tested with 11 either. Need to get around to it, but other priorities atm. (Learning data visualisation using D3 and AngularJS) ;)
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I installed FreeBSD 10.3-RELEASE in an ESXi v6.0 VM and was able to successfully build FreeNAS 9.10 using @JustinClift's instructions. I'll try building version 10... next year! Ha! :smile:
 

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Spearfoot, I'm very curious if you can get FreeNAS10 to build using Justin's instructions.

I just tried building FreeNAS9.10 on the same VM where I have the FreeNAS10 failures, and it built fine. I installed the ISO and got a working system out of it. The build VM is FreeBSD11 based. There is definitely something "different" about FreeNAS10 that causes my issues.

By the way, is there a way to serialize the build so it uses only one builder process? When I had the system setup with 4 cores, it spun up 5 or 6 builders and used ridiculous amounts of RAM.
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Spearfoot, I'm very curious if you can get FreeNAS10 to build using Justin's instructions.

I just tried building FreeNAS9.10 on the same VM where I have the FreeNAS10 failures, and it built fine. I installed the ISO and got a working system out of it. The build VM is FreeBSD11 based. There is definitely something "different" about FreeNAS10 that causes my issues.
Well... I've been working on this for a couple of days, without success.

I see the same kernel: Failed to write core file for process conftest (error 14) error message that @Durkatlon reports, no matter how I go about skinning the cat. It shows up at varying points in the build process, so it's inconsistent.

I'm using FreeBSD 11.0-RELEASE, patched to the 'Latest'n'Greatest'.

On VMware ESXi 6.0U2, I have the FreeBSD VM configured with 4 vCPUs and 12GB of RAM with 8GB of swap. This system builds FreeNAS 9.10 without any problems. FreeNAS 10 builds invariably fail, sometimes locking the VM up so that I have to power it down gracelessly and run fsck to repair the filesystem.

Suspecting that the problem might lie with ESXi (as @Durkatlon suspected), I tried a VirtualBox VM next, on my workstation PC. This VM I configured with 2 vCPUs and 8GB of RAM with 12GB of swap. But builds fail on this system, too, with the same error ( kernel: Failed to write core file for process conftest (error 14)).

As I mentioned upthread, my old FreeBSD 10.3-RELEASE ESXi VM builds FreeNAS 9.10 without problems, but fails when building FN10. I sorta expected that, because iXsystems seems to be using FreeBSD 11 to build both 9.10 and 10 these days.
By the way, is there a way to serialize the build so it uses only one builder process? When I had the system setup with 4 cores, it spun up 5 or 6 builders and used ridiculous amounts of RAM.
I believe this is the Poudriere package building system. It seems to allocate N+1 builder processes, where N is the number of cores on the system. This iXsystems Jenkins build log implies that there's a POUDRIERE_JOBS setting that specifies the number of builders - you might be able to set this as an environment varilable. I dunno...
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Okay, because I'm stubborn as a mule, and a glutton for punishment... I tore down and rebuilt the FreeBSD 11.0-RELEASE VM on my main ESXi AIO. I provisioned it with:
  • 12GB of RAM
  • 4 vCPUs
  • 256GB drive
  • 16GB swap
This is twice the disk size and swap space I'd used previously. Before I'd always used the FreeBSD swap default (usually 4GB) and later added an add'l 4GB of swap with a swap file per the FreeBSD docs.

The VMware tools have been giving me grief, so I installed the Open VM tools with no X11 support.

I then followed @JustinClift's build guide on Git and was able to build both versions of FreeNAS: 9.10 and 10.0!

While the FreeNAS 9.10 build went flawlessly, I still saw the same old error code when building FreeNAS 10. However, it ran to completion and produced the ISO image and other deliverables. Here are the relevant error messages from /var/log/messages:
Code:
Jan  3 15:45:20 beesty11 kernel: Failed to write core file for process conftest (error 14)
Jan  3 15:45:20 beesty11 kernel: pid 42493 (conftest), uid 0: exited on signal 11
Jan  3 16:10:14 beesty11 kernel: pid 81192 (readonly.exe), uid 0: exited on signal 11
Jan  3 16:10:18 beesty11 kernel: pid 84850 (readonly.exe), uid 0: exited on signal 11

The core file write failure for conftest is what seems to have intermittently locked up the VM beforehand. But it ran fine just now, after I doubled the disk size and swap. Why doesn't it behave consistently?! I rationalize it this way: FreeNAS is a large, complex product and FreeBSD needs a lot of RAM - more than I have available on my meager servers - in order to build it smoothly.

This is the kind of baffling, unexplained mystery that has given me a head full of grey hair over the years. Well, my daughters may have contributed a little to the greying, too. :)
 
Last edited:

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Very interesting. I don't have that much RAM to give to the box, but my VM also has a phenomenal amount of swap memory just to get through the build (I think I ended up with 50GB of swap before it would get through a build once I upgraded to FreeBSD11). It helped when I reduced VCPUs from 4 down to 2, so I ended up with 3 builders instead of 5 during the ports building.

I'll give it another go with the Poudriere environment limited to a single builder.
 

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Yay! Finally got it working!! I limited the number of builders to 1 (based on the hint from Spearfoot) which causes the build to take a long time, but limits the amount of RAM that it needs to finally be within a limit that my VM can deal with. The build took about 16 hours but finished with an ISO that results in a working system.

To limit builders to 1 in csh:
Code:
# setenv POUDRIERE_JOBS 1
# make release
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Excellent, that's extremely useful info. :)

Do either of you have time to update the build guide on GitHub to mention that?

Something like "If you're building FreeNAS 10 on a system with less than x GB of RAM, run 'setenv POUDRIERE_JOBS 1' before 'make release'. This will make the build single threaded (slower) but will keep the ram usage lower. For many multicore systems with a lot of ram (32GB+), [some wording here about not needing that setting ;) ]". ?
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Excellent, that's extremely useful info. :)

Do either of you have time to update the build guile on GitHub to mention that?

Something like "If you're building FreeNAS 10 on a system with less than x GB of RAM, run 'setenv POUDRIERE_JOBS 1' before 'make release'. This will make the build single threaded (slower) but will keep the ram usage lower. For many multicore systems with a lot of ram (32GB+), [some wording here about not needing that setting ;) ]". ?
I'll take a stab at it...
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Looks good to me. The one possible source of confusion could come if the user decided to change root's shell. Use of "setenv" assumes a csh style shell (vs "export" in bash for instance).

I guess if the user knows enough to change the default shell for root, they will know how to properly set an environment variable in their shell of choice.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Looks good to me. The one possible source of confusion could come if the user decided to change root's shell. Use of "setenv" assumes a csh style shell (vs "export" in bash for instance).

I guess if the user knows enough to change the default shell for root, they will know how to properly set an environment variable in their shell of choice.
Yeah, I wondered about whether or not to address that... can we safely assume folks who are savvy enough to build FreeNAS are also savvy enough to know how to set environment variables appropriately? I think so. But... I could be wrong! :)
 

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Yeah, I wondered about whether or not to address that... can we safely assume folks who are savvy enough to build FreeNAS are also savvy enough to know how to set environment variables appropriately? I think so. But... I could be wrong! :)
I would say if you follow the instructions to the letter, you end up with "csh" as the shell for root, so the setenv will work. If you deviate from the instructions, you better know what you're doing.
 
Status
Not open for further replies.
Top