fibs are using wrong arps on replies to incoming connections

[kerbstrike]

I have created multiple fibs to split my management network from my user and jails area on freenas.
For internet access this works just fine jail and the base freenas can ping the internet and get file etc. The problem comes in when I attempt to send traffic from a device on the management network to an iocage jail on the jails area with a non default fib. (Note: sending traffic from the jail works and I can ping properly out the fib through the router and to my box on management net)

See attached pic for the fib tunables setup (Note: having net.add_addr_allfibs as a loader type used to work for warden jails in 9.3, but now makes it so even the jails cant reach out to management net, setting as sysctl type got the jails to connect to management net)

relevent iocage setup
ip4:new
ip4_addr:cxgb0|172.19.0.121/24
ip4_saddrsel:1
exec_fib:1
vnetff
defaultrouter:172.19.0.1


when I ping from the jail to the management net we see proper mac addressing:
00:07:43:08:43:3e > 00:01:2e:6e:60:26, ethertype IPv4 (0x0800), length 98: 172.19.0.* > 192.168.1.*: ICMP echo request, id 42033, seq 0, length 64
17:00:19.961282 00:01:2e:6e:60:26 > 00:07:43:08:43:3e, ethertype IPv4 (0x0800), length 98: 192.168.1.* > 172.19.0.*: ICMP echo reply, id 42033, seq 0, length 64

60:26 is the mac of the default router for 172.19.0.*



when I ping from the management net we see the wrong mac (the mac of the management net machine itself)
17:02:08.162923 00:01:2e:6e:60:26 > 00:07:43:08:43:3e, ethertype IPv4 (0x0800), length 74: 192.168.1.* > 172.19.0.*: ICMP echo request, id 13, seq 46071, length 40
17:02:08.163111 00:1b:21:67:ff:5c > 4c:cc:6a:f8:8a:61, ethertype IPv4 (0x0800), length 74: 172.19.0.* > 192.168.1.*: ICMP echo reply, id 13, seq 46071, length 40

the 8a:61 should be the 60:26 we see in the original request (and what we see it use when it originates the ping.

I don't know what other information you might need, but I have searched the forum and freebsd forums. The only thing that looks to reference this type of issue is a bug from way back in 2012/13:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=167947

But a lot of that code is no longer in the main train for freebsd that I can find.

 

[dlavigne]

Were you able to resolve this?

 

[kerbstrike]

I was able to work around it, but not fix it.

I switched to using the bridge method with vnets, and that is currently able to properly get traffic flowing.