Migrating dnsmasq to iocage jail

[Egon Olieux]

Does anyone here have experience running dnsmasq in an iocage jail? (freenas 11.2).
I had dnsmasq working in a warden jail, but was forced to move it to iocage, and can't seem to get it working.

/usr/local/etc/dnsmasq.conf:

Code:

no-hosts
addn-hosts=/etc/hosts_dnsmasq
expand-hosts
local=/lan/
bogus-priv
strict-order
domain-needed
domain=lan

/etc/resolv.conf:

Code:

search lan
nameserver 127.0.0.1 # Force local queries to be handled by dnsmasq.
nameserver 8.8.8.8
nameserver 8.8.4.4

Although dnsmasq seems to be running:

Code:

[root@dnsmasq /]# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 192.168.1.16.53        *.*                    LISTEN
udp4       0      0 192.168.1.16.53        *.*          

I can't seem to be able to get a response from dnsmasq:

Code:

egonolieux@z97:~$ nslookup z97.lan 192.168.1.16
;; connection timed out; no servers could be reached

I assume the same thing happens in the dnsmasq jail itself, as nslookup uses the 2nd entry from resolv.conf:

Code:

[root@dnsmasq /]# nslookup z97.lan
Server:        8.8.8.8
Address:    8.8.8.8#53

** server can't find z97.lan: NXDOMAIN

When looking in the logs, dnsmasq reports that the max number of concurrent queries has been reached:

Code:

Aug 18 23:21:05 dnsmasq dnsmasq[26390]: Maximum number of concurrent DNS queries reached (max: 150)
Aug 18 23:21:45 dnsmasq last message repeated 4 times
Aug 18 23:23:55 dnsmasq last message repeated 13 times
Aug 18 23:33:56 dnsmasq last message repeated 30 times
Aug 18 23:43:48 dnsmasq last message repeated 59 times

I have no idea where this comes from though. Does anyone know what might be going on here?

 

[Fredda]

This looks more like a networking issue rather than a dnsmasq issue.

Are you able to connect to the internet from within your jail?
Can you ping 8.8.8.8?
What is the output of ifconfig and netstat -r?
What is you network setup in the jail? VNET? raw sockets allowed?

 

[Egon Olieux]

Forgot to post this yesterday:

Piging to the jail from the local network works (enabled "allow_raw_sockets" for this):

Code:

egonolieux@z97:~$ ping 192.168.1.16
PING 192.168.1.16 (192.168.1.16) 56(84) bytes of data.
64 bytes from 192.168.1.16: icmp_seq=1 ttl=64 time=0.213 ms
64 bytes from 192.168.1.16: icmp_seq=2 ttl=64 time=0.280 ms

Connecting to port 53 also works on the local network:

Code:

egonolieux@z97:~$ telnet 192.168.1.16 53
Trying 192.168.1.16...
Connected to 192.168.1.16.
Escape character is '^]'.

There is also internet access from within the jail:

Code:

[root@dnsmasq /]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=51 time=29.877 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=26.320 ms

The jail is using shared IP (not VNET)

Output of ifconfig:

Code:

[root@dnsmasq /]# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    ether 00:15:17:d2:48:7a
    hwaddr 00:15:17:d2:48:7a
    inet 192.168.1.16 netmask 0xffffff00 broadcast 192.168.1.255 
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:15:17:d2:48:7b
    hwaddr 00:15:17:d2:48:7b
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    groups: lo 
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:c8:4f:e2:5b:00
    groups: bridge 
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0

Route table:

Code:

[root@dnsmasq /]# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
dnsmasq            link#1             UHS         lo0

 

[Fredda]

Hmm. Most of the output looks OK, IMHO.

Route table:

Code:

[root@dnsmasq /]# netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 192.168.1.16.53        *.*                    LISTEN
udp4       0      0 192.168.1.16.53        *.*          

[root@dnsmasq /]# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
dnsmasq            link#1             UHS         lo0

Only these last two points seem a little bit odd. Is dnsmasq not listening on 127.0.0.1?
Also I'm missing the default route here, but maybe this is a thing due to the shared IP?
(I'm using VNET for all my jails).

Also this here sounds strange:

Aug 18 23:21:05 dnsmasq dnsmasq[26390]: Maximum number of concurrent DNS queries reached (max: 150)
Aug 18 23:21:45 dnsmasq last message repeated 4 times
Aug 18 23:33:56 dnsmasq last message repeated 30 times

This looks like dnsmasq is flooded with requests. You could increase the logging level in dnsmasq, to see where they
come from. Maybe there is a strange feedback loop? Set "log-queries" in the dnsmasq.conf.

Check if dnsmasq is really listening on localhost in the jail (telnet 127.0.0.1 53).

You could also try to set a different resolv.conf file for dnsmasq. Set the following paramet in the dnsmasq.conf file:

Code:

# Change this line if you want dns to get its upstream servers from
# somewhere other that /etc/resolv.conf
resolv-file=/etc/resolv.conf_dnsmasq

and put there your upstream DNS-Servers.

 

[Egon Olieux]

Only these last two points seem a little bit odd. Is dnsmasq not listening on 127.0.0.1?
Also I'm missing the default route here, but maybe this is a thing due to the shared IP?
(I'm using VNET for all my jails).

This seems to be a thing related to shared IP jails indeed.
Although the socket is listening on 192.168.1.16, I seem to be able to connect to localhost on port 53 (isn't this a little weird since the socket is listening on 192.168.1.16 explicitly, and not localhost?).

Based on your suggestion regarding resolv.conf, I specified the servers dnsmasq should use in-config (server=8.8.8.8 and server=8.8.4.4) and also added the no-resolv option. The only entry left in resolv.conf is now localhost. Doing this seems to have solved the "max number of queries" error, which makes me think dnsmasq somehow got stuck in a loop on localhost. Querying dnsmasq both from the local network and localhost still doesn't work though; same error as before (timeout).

Maybe I should try virtualizing the network (VNET)? I'm afraid I don't know much about FreeBSD internals and what benefits this gives, aside from the possibility of dnsmasq working as intended. However, when I try to create a VNET jail, no IP address gets assigned to it and I only have the loopback interface available in the jail. Are there additional configuration steps required? I explicitly configured the interface, IP and default route in the setup, but none of this is showing up.

 

[Fredda]

This seems to be a thing related to shared IP jails indeed.
Based on your suggestion regarding resolv.conf, I specified the servers dnsmasq should use in-config (server=8.8.8.8 and server=8.8.4.4) and also added the no-resolv option. The only entry left in resolv.conf is now localhost. Doing this seems to have solved the "max number of queries" error, which makes me think dnsmasq somehow got stuck in a loop on localhost. Querying dnsmasq both from the local network and localhost still doesn't work though; same error as before (timeout).

Yes, that was pretty much what I was suggesting in my previous post. One thing that just occured to me, you try to look up "z97.lan"?
Does the resolving with dnsmasq not work at all or is it just local addresses? What happens with e.g. "nslookup xkcd.com"?
For resolving local addresses please check the "address" config directive.
For logging please check "log-queries" and "log-facility"

Maybe I should try virtualizing the network (VNET)? I'm afraid I don't know much about FreeBSD internals and what benefits this gives, aside from the possibility of dnsmasq working as intended. However, when I try to create a VNET jail, no IP address gets assigned to it and I only have the loopback interface available in the jail. Are there additional configuration steps required? I explicitly configured the interface, IP and default route in the setup, but none of this is showing up.

VNET will give the jail an independent IP- Address, so you can start all the services you'll need and don't have to share ports with the FreenNAS.
In the jail properties select [X] VNET, ipv4-Interface:vnet0 and assign IP, netmask and default route.

 

[Egon Olieux]

Yes, that was pretty much what I was suggesting in my previous post. One thing that just occured to me, you try to look up "z97.lan"?
Does the resolving with dnsmasq not work at all or is it just local addresses? What happens with e.g. "nslookup xkcd.com"?
For resolving local addresses please check the "address" config directive.
For logging please check "log-queries" and "log-facility"

That's what I meant with the timeout error; both local (z97.lan) and domain queries (google.com) result into a connection timeout error, without dnsmasq logging anything (despite "log-queries"). No query ever seems to reach dnsmasq.

 

[Fredda]

Try running dnsmasq in debug mode instead as a demon:
dnsmasq -d -q
In a second shell directly query the dnsmasq server, eg:
nslookup slashdot.org ip.of.your.jail

 

[Egon Olieux]

While running in debug mode, it appears dnsmasq is able to resolve corectly:

Code:

dnsmasq: query[A] z97.lan from 192.168.1.11
dnsmasq: /etc/hosts_dnsmasq z97.lan is 192.168.1.50
dnsmasq: query[A] z97.lan from 192.168.1.11
dnsmasq: /etc/hosts_dnsmasq z97.lan is 192.168.1.50
dnsmasq: query[A] z97.lan from 192.168.1.11
dnsmasq: /etc/hosts_dnsmasq z97.lan is 192.168.1.50

It also forwards the queries correctly to 8.8.8.8 for real domains.
The weird thing is that each query is repeated 3 times, and after that, it times out:

Code:

freenas# nslookup z97.lan 192.168.1.16
;; connection timed out; no servers could be reached

 

[Fredda]

Your setup is confusing.

In your prior posts you mentioned you set up 8..8.8.8 and 8.8.4.4 as you DNS server, now there is suddenly a new server 192.168.1.11?

If you have a DSL-router or something involved doing DHCP for your local net, do not use 8.8.8.8 as DNS server.
Google knows a lot about you, but not how to resolve your local net.

I see three options to this right.

• configure dnsmasq to forward only to your (other) local DNS-Server which does DHCP
• turn off DHCP in your local DNS-server and let dnsmasq do the DHCP
• configure your DNS-server to always give out the same IP-addresses and use the "address" option in dnsmasq

Anyway, this is getting a little bit offtopic, there is no problem with the network config in your jail.
Use one of the three options to fix you dnsmasq setup.

 

[Egon Olieux]

Not sure where you get the idea from I have a second DNS server running somewhere. 192.168.1.11 is the IP of the FreeNAS machine I made the queries from (sorry, forgot to mention this). The dnsmasq jail is 192.168.1.16 and my personal computer is 192.168.1.50 (all static addresses).

I use Googles DNS for dnsmasq to forward queries it cant resolve (which are not in "/etc/hosts_dnsmasq"). These addresses are set using the "server" option in dnsmasq.conf. My local DHCP server has the dnsmasq IP set for DNS, not Google.

The thing is the exact same config has worked for years, literally nothing changed; the only thing I did was move to iocage jails because I'm somewhat forced to because of recent updates.

 

[Fredda]

Sorry, I misread the debug log. I read it forwards the request to 192.168.1.11 and not that it receives the request from there.
So it basically works as intended if you run in debug mode.

Could it be something completely different? When you start in debug mode it probably runs as root?
How is it started in normal mode? With rc scripts? Which user runs dnsmasq? Can dnsmasq read the config files?

 

[Egon Olieux]

There's no difference between running it in debug mode and as a daemon in practice; it only gives the appearance of working in debug mode because of whats logged. See post #9.

Normally dnsmasq is started using rc scripts yes, running as the nobody user. The config file is set to 644, so it should be able to read it.

 

[Fredda]

Hmm, I'm running out of ideas. So the lookup of internet addresses works, it's just the local not working?

You could try to strip down the config file, IMHO of all you parameter only the addn-hosts is really needed.

 

[colmconn]

Don't you need to add listen-address=127.0.0.1 to your dnsmasq config file to make it listen on localhost? Alternatively, if it defaults to listening on your 169... address shouldn't you use that as the first entry in your resolv.conf file?

 

[Egon Olieux]

Hmm, I'm running out of ideas. So the lookup of internet addresses works, it's just the local not working?

You could try to strip down the config file, IMHO of all you parameter only the addn-hosts is really needed.

The logs show that the domain names are translated to the correct ip addresses, but the response from nslookup remains ";; connection timed out; no servers could be reached"; thats the weird thing.

Don't you need to add listen-address=127.0.0.1 to your dnsmasq config file to make it listen on localhost? Alternatively, if it defaults to listening on your 169... address shouldn't you use that as the first entry in your resolv.conf file?

Yes, I would assume dnsmasq wouldnt even respond to any queries comming from localhost as its not listening on that address, but strangely enough, it also logs queries coming from localhost.

I already tried running without any config parameters in debug mode and now tried it again for the daemon, but still the same result.

 

[Fredda]

The logs show that the domain names are translated to the correct IP addresses, but the response from nslookup remains ";; connection timed out; no servers could be reached"; thats the weird thing.

Really weird. Just tried it here on a 11.2u5 iocage jail. Works like a charm. Installed dnsmasq, did not touch config file, start it, make a lookup everything is fine.

Code:

root@jail:~ # dnsmasq -q --log-facility /tmp/dnsmasq.log
root@jail:~ # nslookup www.google.de localhost
Server:        localhost
Address:    127.0.0.1#53

Non-authoritative answer:
Name:    www.google.de
Address: 108.177.126.94
Name:    www.google.de
Address: 2a00:1450:4013:c01::5e

root@jail:~ # tail /tmp/dnsmasq.log
Aug 21 16:38:19 dnsmasq[35902]: query[A] www.google.de from 127.0.0.1
Aug 21 16:38:19 dnsmasq[35902]: forwarded www.google.de to ....
Aug 21 16:38:19 dnsmasq[35902]: reply www.google.de is 108.177.126.94
Aug 21 16:38:19 dnsmasq[35902]: query[AAAA] www.google.de from 127.0.0.1
Aug 21 16:38:19 dnsmasq[35902]: forwarded www.google.de to ....
Aug 21 16:38:19 dnsmasq[35902]: reply www.google.de is 2a00:1450:4013:c01::5e

Perhaps you could quickly add another jail with a vanilla dnsmasq with default config to see if the problem persists?

 

[Egon Olieux]

Same result, any query times out. Again, the logs seem fine here as well. I'm starting to think that somehow dnsmasq isn't able to send the response back. The weird thing is that I'm running a few other jails which are practically configured in the exact same way (an nginx proxy, plex and transmission), and none of those seem to have any network related problems.

 

[Fredda]

I must admit I'm out of ideas. Works here™

But it is basically just dnsmasq which is not working? If you are addressing the upstream servers directly the resolution works?
And it's also not just a nslookup problem? dig @localhost www.google.de also runs in a timeout?
The FreeBSD version of the jail matches the FreeBSD version FreeNAS system?

 

[Egon Olieux]

Querying google dns directly works without issue (I.e. not using dnsmasq at all). dig also gives the same timeout error. Both the host and guest machines are 11.2-STABLE.