Network Traffic from Default Gateway Dropped

[Brent Bohmont]

I'm experiencing a network issue on my recently built server. Everything works correctly on my local network, but the server is unable to contact the network's default gateway or anything beyond (e.g. the Internet). I've dug into this a little and believe the network traffic is being correctly routed, but responses are being dropped. There's some bit of configuration I'm missing, perhaps?

Here's my troubleshooting...

FreeNAS Version - FreeNAS-9.3-STABLE-201502110455
ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO> ether xx:xx:xx:xx:xx
inet 172.21.120.10 netmask 0xffffff00 broadcast 172.21.120.255
inet6 fe80::225:90ff:fed7:2a1b%em0 prefixlen 64 scopeid 0x1
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

Ping default gateway
$ ping 172.21.120.1
PING 172.21.120.1 (172.21.120.1): 56 data bytes
--- 172.21.120.1 ping statistics ---
34 packets transmitted, 0 packets received, 100.0% packet loss

ping tcpdump trace
# tcpdump -nS host 172.21.120.1
09:45:55.157429 IP 172.21.120.10 > 172.21.120.1: ICMP echo request, id 45512, seq 0, length 64
09:45:55.158036 IP 172.21.120.1 > 172.21.120.10: ICMP echo reply, id 45512, seq 0, length 64
09:45:56.158376 IP 172.21.120.10 > 172.21.120.1: ICMP echo request, id 45512, seq 1, length 64
09:45:56.158889 IP 172.21.120.1 > 172.21.120.10: ICMP echo reply, id 45512, seq 1, length 64
09:45:57.159377 IP 172.21.120.10 > 172.21.120.1: ICMP echo request, id 45512, seq 2, length 64
09:45:57.159943 IP 172.21.120.1 > 172.21.120.10: ICMP echo reply, id 45512, seq 2, length 64

Ping generates a request and a reply, so the router is working correctly. I get similar results with a http GET request (e.g. curl http://www.google.com).

Firewall, perhaps? My grasp of FreeBSD networking is shaky at best, but it seems that any firewall is connected through hooks exposed by the ng_ether kernel module. Doesn't seem to be much there, however...zero hooks on each of four nodes.

# ngctl list
There are 4 total nodes:
Name: em0 Type: ether ID: 00000001 Num hooks: 0
Name: igb0 Type: ether ID: 00000002 Num hooks: 0
Name: ipfw0 Type: ether ID: 00000003 Num hooks: 0
Name: ngctl52556 Type: socket ID: 00000006 Num hooks: 0

The ipfw configuration seems empty, too.

#ipfw list
65535 allow ip from any to any

 

[dlavigne]

There's no firewall enabled on FreeNAS (the ipfw stuff is just there to do the nat/bridging thing for jails), so we can't blame that. You shouldn't be getting 100% packet loss pinging the default gateway. Does the gateway have any firewall rules or restrictions on MAC addresses? Does the gateway show as up and with the correct IP in netstat -rn?

 

[Brent Bohmont]

The gateway is clean, no applicable firewall or routing rules. The gateway shows up with the correct address in netstat -rn.

# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 172.21.120.1 UGS 0 619 em0
127.0.0.1 link#6 UH 0 6088374 lo0
172.21.120.0/24 link#1 U 0 28694072 em0
172.21.120.10 link#1 UHS 0 11 lo0

 

[Brent Bohmont]

I was able to get connectivity, but there is still something not quite right. Here are the steps I tried and the results:

1. Try the other NIC - The motherboard, a SuperMicro X10SLL-F has two NIC's. Initially, I configured interface em0 (selected more-or-less at random) for a static IP using the web interface. Using the system console, I deleted that interface, reconnected via a DHCP assigned address and added the static IP to the igb0 NIC. Presto - I was able to ping the default gateway.
2. I applied the latest update and rebooted (not sure if this was technically necessary; system rebooted on it's own). Unable to ping gateway after reboot.
3. I switched the primary NIC back to em0. Once again, I was unable to ping the gateway.
4. Went to bed. Oddly, I received an e-mail overnight from the server. When I checked network connectivity in the morning everything seems to be working.

I have no idea what happened overnight. Just happy that everything is working and the issue appears to be resolved.

 

[SweetAndLow]

Are you sure the static IP you are assigning is from a rang not in your routers dhcp range? You didn't configure any kind of nic binding did you?

 

[Brent Bohmont]

The static IP is outside of the DHCP range. Not sure what you mean by "NIC Binding". I haven't configured anything at the OS level; I'm stuck with the web GUI and console menu.

Right now, routing is working correctly but the NIC seems to have both a DHCP assigned (.113) and statically assigned (.10) address. The web GUI only shows one static IP address.