Apple devices (and possibly other vendors) lose connection to VNET jails because the jail ARP replies are not going through. These replies correspond to ARP requests that are padded with more than 18 zeros, like macOS or iOS are sending (looks like VNET has a bug that corrupts ARP replies when the incoming request is padded with many zeros). Connection can be restored when pinging from inside the jail to the other device (since it is accompanied with a new ARP message), but the connection is lost again when the ARP cache is invalidated on the Apple OS (typically in 20 minutes).
Similar issues have been reported previously several times here and I believe that's the root cause. Note that this behaviour is the same in warden jails, but in warden jails non-VNET network worked with Plex for example, where on iocage Plex must use VNET network stack. You can read below how to reproduce it (I have written more about the investigation here). I am not aware of any solution for this but as a workaround you can set a script in the jail that periodically sends ARP announce messages to the network or leave a ping running. Not ideal by any means.
To reproduce it create a jail with scapy installed and have another VNET jail (any existing VNET jail is fine). We will use the scapy jail to simulate an ARP request on behalf of another real machine on the network. You need the IP of a VNET jail (say 192.168.1.2) and the IP (say 192.168.1.3) and MAC address (say 00:11:22:33:44:55) of a real machine connected to the network.
1. Run tcpdump on the machine connected to the network (this listens to ARP traffic to/from the VNET jail). Replace 'en0' with the right interface name):
2. From the scapy jail (not the VNET jail we are trying to debug), run the following in scapy (this sends an ARP request on behalf of the real machine):
You can see that the tcpdump shows only the ARP request (the ARP reply is not going through):
However, when repeating this with
Similar issues have been reported previously several times here and I believe that's the root cause. Note that this behaviour is the same in warden jails, but in warden jails non-VNET network worked with Plex for example, where on iocage Plex must use VNET network stack. You can read below how to reproduce it (I have written more about the investigation here). I am not aware of any solution for this but as a workaround you can set a script in the jail that periodically sends ARP announce messages to the network or leave a ping running. Not ideal by any means.
To reproduce it create a jail with scapy installed and have another VNET jail (any existing VNET jail is fine). We will use the scapy jail to simulate an ARP request on behalf of another real machine on the network. You need the IP of a VNET jail (say 192.168.1.2) and the IP (say 192.168.1.3) and MAC address (say 00:11:22:33:44:55) of a real machine connected to the network.
1. Run tcpdump on the machine connected to the network (this listens to ARP traffic to/from the VNET jail). Replace 'en0' with the right interface name):
tcpdump -XXvi en0 arp host 192.168.1.22. From the scapy jail (not the VNET jail we are trying to debug), run the following in scapy (this sends an ARP request on behalf of the real machine):
Code:
station_mac='00:11:22:33:44:55' station_ip='192.168.1.3' jail_ip='192.168.1.2' padding=19 arp=ARP(hwsrc=station_mac,psrc=station_ip,pdst=jail_ip) ether=Ether(dst='ff:ff:ff:ff:ff:ff',type=0x0806) sendp(ether/arp/Padding(load='\x00'*padding))
You can see that the tcpdump shows only the ARP request (the ARP reply is not going through):
Code:
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.2 tell 192.168.1.3, length 48
0x0000: ffff ffff ffff 0011 2233 4455 0806 0001 ..........`#....
0x0010: 0800 0604 0001 0011 2233 4455 c0a8 0103 ..........`#...i
0x0020: 0000 0000 0000 c0a8 0102 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 0000 0000However, when repeating this with
padding=18 you can see both ARP request+reply:Code:
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.2 tell 192.168.1.3, length 46
0x0000: ffff ffff ffff 0011 2233 4455 0806 0001 ..........`#....
0x0010: 0800 0604 0001 0011 2233 4455 c0a8 0103 ..........`#...i
0x0020: 0000 0000 0000 c0a8 0102 0000 0000 0000 ................
0x0030: 0000 0000 0000 0000 0000 0000
ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.2 is-at d1:23:99:f9:6f:6c, length 46
0x0000: 0011 2233 4455 d123 99f9 6f6c 0806 0001 ....`#.P..ol....
0x0010: 0800 0604 0002 d123 99f9 6f6c c0a8 0102 .......P..ol....
0x0020: 0011 2233 4455 c0a8 0103 0000 0000 0000 ....`#...i......
0x0030: 0000 0000 0000 0000 0000 0000
Last edited: