New NIC, webUI, SSH works nothing else

[mrreload]

So I upgraded to 12.0-U5 Core from U4 today. Immediately I noticed the NIC I had added to the system a couple months ago (complete failure at the time) is showing up without adding a tunable, etc.

The NIC is Asus XG-C100C 10G.
x.x.69.x/24 is primary network aq0
x.x.2.x/24 is my secondary/test network em1
Subnets are physically segregated/ separate switches
Routing between takes place on my pfSense firewall

I moved my old primary NIC on the secondary subnet and assigned a new address to it. All works in this config
I then put the new NIC with old NICs IP and am able to get to Web UI and SSH to server on this IP. x.x.69.143
BUT no other services appear to work. SMB, NFS, jails, VMs etc
I have 3 jails from plugins and 4 Bhyve VMs normally running.
Windows VM can ping the host, host can ping the VM, VM cannot ping anything beyond host nor can anything else on network ping the VM.
I did shutdown all VMs and jails to bind them to new NIC.

I also tried removing all config from em1 and restarting, same. I can access UI, not much else.

TrueNAS did auto create a bridge1 for the new NIC and the Jails appear to be attaching to it as members along with aq0.

I feel like the TrueNas host is somehow blocking or not forwarding the traffic to/from the VM/jails. I think IF I can get this solved The other services will follow suite.

I reverted to a backup to keep system in production until I can figure this out. Will likely test new NIC on secondary subnet next.

System specs are in my Sig.

 

[sretalla]

BUT no other services appear to work. SMB, NFS, jails, VMs etc

Most (maybe all) of those things would be bound to the old interface by NIC name.

You'll need to edit the config of the jails and swap NICs over.

For VMs, you need to edit the NIC device.

For SMB, check the Bind IP addresses under Services. It may be best to remove any selections (which makes it bind to all).

 

[mrreload]

I actually did check the bind addresses and NICs on all VMs/Jails and Services. For SMB I have tried to bind to specific IP and with all selections removed. (bind all)
VM network devices are working as they can access the host.

There is a place in the UI to change the NIC for each Jail. What config files should I be checking/editing for the jails?

Most (maybe all) of those things would be bound to the old interface by NIC name.

This is what I am thinking and there must be some config file that is not getting written to or written to properly.

 

[sretalla]

In the jail "Edit" setings, on the Basic page, you have vnet_default_interface

In a VM, Devices, NIC, Edit, you have Nic to attach.

I suspect that all of your jail problems are that your new NIC is on its own bridge and not added to the bridge that everything wants to use (usually bridge0)

Setting that NIC as the vnet_default_interface on a jail that auto-starts should sort that out.

 

[mrreload]

I did change all of what you suggested.
I even shutdown all Jails and VMs and attempted testing a single jail.
ifconfig from the shell showed vnet0 attaching to bridge1 as expected once the jail was started again. But again the jail can only access the host.
I even created a new test jail with same results.

Is there something more global that could be the issue? Even with all jails, VMs and services bound to the correct NIC or IP, I still get no joy.

When I 1st got this NIC (on U4 at the time) I had the same issue plus it was not recognized. Adding a tunable enabled the NIC but no traffic beyond host. At the time I even tried upgrading to Scale but while the NIC was recognized, networking with it was only sporadic. (Sometimes worked then completely stopped (Outside of UI) next day.

Please note this server have 2 onboard NICs and 1 2 port NIC (All Intel), plus IPMI. Would adding a 5th interface cause this issue? Should I attempt to remove the dual port card?