SOLVED Issue connecting to other network machines via second NICs subnet

[Spa]

Hi there,

First, here's my current setup for reference:

I have an all-in-one ESXi machine with the following VMs: Windows, Ubuntu, FreeNAS (using PCIE passthrough, don't worry).
Each VM has two NICs attached to it. One is on the 192.168.1.xx subnet (my main network) and one is on the 172.16.0.xx subnet (which I'm trying to configure).

IPs for the machines are as follows:
Windows: 172.16.0.15/24
Ubuntu: 172.16.0.12/24
FreeNAS: 172.16.0.10/24

All of these machines are using the VMXNet3 adapter.

My issue is that while I can communicate with ease on the first subnet (192.168.1.xx), I cannot get any communication between FreeNAS and any other machine on the second subnet (172.16.0.xx) to work. I can freely ping between the Windows and Ubuntu machines with no issues, but while FreeNAS can self ping on it's 172.16.0.10 IP just fine, it both cannot ping other machines on the subnet, and other machines cannot ping it. From what I can tell from Googling, FreeNAS itself doesn't have firewall rules to contend with (although these are threads from 2014 stating this), so I don't think it's a firewall issue. Here is my ESXi configuration, and here is my FreeNAS config. I'm not really a networking guy, I've been learning as I go along, so maybe there's something obvious here I'm missing. Can anyone help me figure out why the FreeNAS machine won't talk with anyone else on the second subnet?

 

[Elliot Dierksen]

Would you please provide more detail regarding the rest of your network configuration? I assume you have some kind of router/firewall for internet access. Are you trying to get FreeNAS to do the routing between networks? There is no way to say without greater detail.

 

[Spa]

I am not trying to get FreeNAS to do any routing, simply be open to communication on a second subnet.

I'm happy to provide other information, but what other information would be useful to you?

I'll do a more explicit summary of my goals/setup in order to better explain my situation. Please note all subnets/IPs given are /24.

Right now, I have FreeNAS virtualized under ESXi in my single physical server. I, up to this point, have been content with utilizing the 1gb NIC assigned to FreeNAS to share back storage for VMs via iSCI. Recently, I added a pair of NVME drives as a mirror to FreeNAS, and I'm hoping to avoid having to go out and buy 10gb NICs/switches to get full bandwidth out of them.

My solution to the above, based on some Googling and talking with my work's IT guy, was to create a network internal to ESXi. That means creating a virtual port (IntraPort) linked to a virtual switch (IntraSwitch) - but without a physical NIC attached to it. ESXi will do whatever it's magic is to ensure the VMs using that same port as the passed-through NIC can talk to each other (according to another user's benchmark, at at least 33gbps).

With that in mind, I went to work to get this system setup. I am using 3 VMs are test subjects. Each VM is now equipped with 2 virtual NICs - one for accessing the wider intranet (passed through to real, physical NICs, and operating on the 192.168.1.xx subnet) , and the second (connected to the IntraPort, with no physical NIC, and operating on 172.16.0.xx subnet) for communication within the ESXi server's realm.

At this stage, two of my VMs can communicate across that 172.16.0.xx subnet perfectly fine. My Ubuntu (Plex) and Windows (Windows10) servers will happily ping and be pinged from one to another. The FreeNAS VM, on the other hand, is unwilling to communicate with either server over the 172.16.0.xx subnet (virtual link), only on the 192.168.1.xx subnet (physical link). The FreeNAS vm will happily self-ping on it's given 172.16.0.10 IP address however, leading me to believe it's not a NIC issue. Additionally, the results of successful communication between Ubuntu and Windows over the subnet lead me to believe it is not an ESXi or firewall issue. This leads me to wonder if there's something that needs to be done special for FreeNAS to communicate over two distinct subnets, or if it has any hidden firewall rules or something that would prevent it from receiving inbound pings or sending pings out to the other VMs.


Hopefully that wall of text provides better context for the issue at hand and why I'm attempting this.

 

[Elliot Dierksen]

My FreeNAS boxes have 3 different IP's on their different NIC's, so it is certainly capable of doing that. What does a network summary look like from the network tab? Also, there are a couple CLI commands that would be of interest. ifconfig -a will list all the interfaces. netstat -rn will show all routes. arp -a will list any ARP entries. It would also be helpful to see what the networking config looks like in ESXi.

 

[Spa]

Network summary is posted in OP, but it's embedded in a "here" (same for ESXi config), so it might have been easy to miss depending on how this forum renders for you. Here's direct links: https://imgur.com/IGb49fP https://imgur.com/tZQUiDw https://imgur.com/Wtctn77

Results of ifconfig -a:

Code:

root@freenas:~ # ifconfig -a
vmx0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:a6:79:62
        hwaddr 00:0c:29:a6:79:62
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:a6:79:4e
        hwaddr 00:0c:29:a6:79:4e
        inet 192.168.1.144 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:a6:79:58
        hwaddr 00:0c:29:a6:79:58
        inet 172.16.0.10 netmask 0xffffff00 broadcast 172.16.0.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Results of netstat -rn:

Code:

root@freenas:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.2        UGS        vmx1
127.0.0.1          link#4             UH          lo0
172.16.0.0/24      link#3             U          vmx2
172.16.0.10        link#3             UHS         lo0
192.168.1.0/24     link#2             U          vmx1
192.168.1.144      link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#4                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

Results of arp -a:

Code:

root@freenas:~ # arp -a
? (172.16.0.10) at 00:0c:29:a6:79:58 on vmx2 permanent [ethernet]
? (192.168.1.43) at 04:a1:51:1e:24:78 on vmx1 expires in 1018 seconds [ethernet]
? (192.168.1.2) at e8:fc:af:fb:59:aa on vmx1 expires in 941 seconds [ethernet]
? (192.168.1.26) at d0:50:99:5c:0f:1a on vmx1 expires in 166 seconds [ethernet]
? (192.168.1.144) at 00:0c:29:a6:79:4e on vmx1 permanent [ethernet]
? (192.168.1.19) at 08:62:66:c0:5b:e8 on vmx1 expires in 1199 seconds [ethernet]
? (192.168.1.53) at 00:50:56:6f:bb:73 on vmx1 expires in 513 seconds [ethernet]
? (192.168.1.21) at 00:0c:29:8e:73:f6 on vmx1 expires in 1173 seconds [ethernet]

 

[Elliot Dierksen]

You have a vmx0 interface that has no IP address on it. You can post images in the post which makes things much easier to follow. Making a guess based on this, I wonder if your other 172.16.0.0/24 VM's are attached to a different Vswitch than FreeNAS. That would make them unable to talk to one another.

 

[Spa]

Huh... So I had vmx0 as a second physically-attached NIC I was going to play around with for teaming that I had disabled/removed via the console. I removed the interface entirely via ESXi and rebooted FreeNAS and now I can ping on both the 192.168.1.xx and the 172.16.0.xx subnets. I'm not sure how removing the NIC effected that, so if you have the time and the knowledge, I'd love to learn how that chain of cause and effect worked. Thanks for the help!

Re: embedded images; it's been forever since I've actively posted on a traditional forum and not Reddit, so I honestly forgot you could embed as the full image and not just a hyperlink.