SOLVED FreeNAS 11.3-U4.1 fails to create network connection to another FreeNAS Server

MattTaylor2

Dabbler
Joined
Oct 7, 2020
Messages
12
Hi all,

I am unable to create an SSH connection between our two FreeNAS boxes (Both on 11.3-U4.1). A static route has been created on host 1 and both connected interfaces show as active.

Have followed the documentation on how to create an SSH Connection and Keypair however when we try to "Discover Remote Host Key" the following error occurs:
(IP addresses and Hostnames obfuscated)

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1084, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/keychain.py", line 509, in remote_ssh_host_key_scan
    raise CallError(f"ssh-keyscan failed: {proc.stderr!r}")
middlewared.service_exception.CallError: [EFAULT] ssh-keyscan failed: "connect (`172.16.0.2'): Invalid argument\r\nconnect (`172.16.0.2'): Invalid argument\r\nconnect (`172.16.0.2'): Invalid argument\r\n"


Below are the network settings for the NIC on host1:
Code:
root@hostname:~ # ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Replication
        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:5d:53:90
        hwaddr a0:36:9f:5d:53:90
        inet 172.16.0.1 netmask 0xfffffff8 broadcast 172.16.0.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active

Network settings for NIC on host2:
Code:
root@hostname:~ # ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Replication
        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:60:7c:48
        hwaddr a0:36:9f:60:7c:48
        inet 172.16.0.2 netmask 0xfffffff8 broadcast 172.16.0.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active


Output of netstat -rn host1
Code:
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default              *.*.*.*                 UGS       lagg0
*.*.*.*                 link#8               U           lagg0
*.*.*.*                 link#8              UHS        lo0
127.0.0.1           lo0                   UHS        lo0
172.16.0.0/29    172.16.0.2         UGS        ix0
172.16.0.1         link#1               UHS        lo0


netstat -rn host2:
Code:
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default               *.*.*.*               UGS        igb0
*.*.*.*                    link#3             U          igb0
*.*.*.*                   link#3             UHS         lo0
127.0.0.1          lo0                UHS         lo0
172.16.0.0/29      172.16.0.1     UGS         ix0
172.16.0.2         link#1             UHS         lo0



Both NICS are directly cabled together.

We have two other FreeNAS boxes that have the same setup as these two, and we have an SSH connection between these two for replication. It has a static route on host 1 as per the problematic NAS but I have noticed that in the routing table the gateway is set to "link#l" on the working NAS. When a static route is created via the gui on the NAS with the issue, the gateway in the routing table is set to host2 IP rather than "link#l".

/var/log/messages shows the following:
Code:
Oct  7 08:40:59 hostname kernel: arpresolve: can't allocate llinfo for 172.16.0.2 on ix0


Any help would be much appreciated.

Thanks
Matt
 

Kcaj

Contributor
Joined
Jan 2, 2020
Messages
100
Im not sure why you need a static route at all. If your LAGG0 and ix0 interfaces are in different networks (they should be) then it should work just fine.

Can you please post complete ifconfig of both boxes
 

MattTaylor2

Dabbler
Joined
Oct 7, 2020
Messages
12
Thanks for the reply, please see below:
Host 1
Code:
 # ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Replication     options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:5d:53:90
        hwaddr a0:36:9f:5d:53:90
        inet 172.16.0.1 netmask 0xfffffff8 broadcast 172.16.0.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:5d:53:92
        hwaddr a0:36:9f:5d:53:92
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
ix2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:a2:b6
        hwaddr 0c:c4:7a:59:a2:b6
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
ix3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500      options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:a2:b7
        hwaddr 0c:c4:7a:59:a2:b7
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: member of lagg0
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 0c:c4:7a:45:a7:98
        hwaddr 0c:c4:7a:45:a7:98
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: member of lagg0
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 0c:c4:7a:45:a7:98
        hwaddr 0c:c4:7a:45:a7:99
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: lagg0
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 0c:c4:7a:45:a7:98
        inet 10.*.*.* netmask 0xfffffe00 broadcast 10.*.*.*
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto failover lagghash l2,l3,l4
        laggport: em0 flags=5<MASTER,ACTIVE>
        laggport: em1 flags=0<>


Host 2
Code:
root@ # ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Replication      options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:60:7c:48
        hwaddr a0:36:9f:60:7c:48
        inet 172.16.0.2 netmask 0xfffffff8 broadcast 172.16.0.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:60:7c:4a
        hwaddr a0:36:9f:60:7c:4a
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: lag01
 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:79:5c:c0
        hwaddr 0c:c4:7a:79:5c:c0
        inet 10.*.*.* netmask 0xffffff00 broadcast 10.*.*.*
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:79:5c:c1
        hwaddr 0c:c4:7a:79:5c:c1
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
 

MattTaylor2

Dabbler
Joined
Oct 7, 2020
Messages
12
Thanks for the reply, please see below:
Host 1
Code:
 # ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Replication     options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:5d:53:90
        hwaddr a0:36:9f:5d:53:90
        inet 172.16.0.1 netmask 0xfffffff8 broadcast 172.16.0.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:5d:53:92
        hwaddr a0:36:9f:5d:53:92
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
ix2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:a2:b6
        hwaddr 0c:c4:7a:59:a2:b6
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
ix3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500      options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:a2:b7
        hwaddr 0c:c4:7a:59:a2:b7
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: member of lagg0
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 0c:c4:7a:45:a7:98
        hwaddr 0c:c4:7a:45:a7:98
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: member of lagg0
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 0c:c4:7a:45:a7:98
        hwaddr 0c:c4:7a:45:a7:99
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: lagg0
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 0c:c4:7a:45:a7:98
        inet 10.*.*.* netmask 0xfffffe00 broadcast 10.*.*.*
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto failover lagghash l2,l3,l4
        laggport: em0 flags=5<MASTER,ACTIVE>
        laggport: em1 flags=0<>


Host 2
Code:
root@ # ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Replication      options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:60:7c:48
        hwaddr a0:36:9f:60:7c:48
        inet 172.16.0.2 netmask 0xfffffff8 broadcast 172.16.0.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:60:7c:4a
        hwaddr a0:36:9f:60:7c:4a
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: lag01
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:79:5c:c0
        hwaddr 0c:c4:7a:79:5c:c0
        inet 10.*.*.* netmask 0xffffff00 broadcast 10.*.*.*
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:79:5c:c1
        hwaddr 0c:c4:7a:79:5c:c1
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

Yes the NIC's are on different networks.
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925

MattTaylor2

Dabbler
Joined
Oct 7, 2020
Messages
12
Removed static route from host 1, checked netstat -rn to make sure the route was gone.

Deleted the interface from the FreeNAS gui on both host1 and host2.

Recreated the interfaces from the gui. Was able to ping and create ssh connection between the two boxes.
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
Thank you.
 
Top