iocage and tunables

[mrgtstr]

I will try to best describe my situation....I just recently updated my freenas SAN to version 11.2. It has a dual port 10gb network card that is directly connected to two hypervisors and configured with its own subnet in order to create iscsi sharing. In order to get the two hypervisors to see the SAN I had to create tunables to create a network bridge. I have attached a screen shot of the tunables.

That being said when I configure an iocage jail I am unable to ping my gateway and believe it maybe related to the tunables. I am wondering if I need to create a new interface for the iocage jails or use a different one. Below is a screen shot of the pings and the iocage settings that I have.


root@test11:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=8<VLAN_MTU>
ether 02:ff:60:85:75:ab
hwaddr 02:24:10:00:16:0b
inet 172.31.1.40 netmask 0xffffff00 broadcast 172.31.1.255
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair


root@test11:~ # ping 172.31.1.40
PING 172.31.1.40 (172.31.1.40): 56 data bytes
64 bytes from 172.31.1.40: icmp_seq=0 ttl=64 time=0.028 ms
64 bytes from 172.31.1.40: icmp_seq=1 ttl=64 time=0.037 ms
64 bytes from 172.31.1.40: icmp_seq=2 ttl=64 time=0.023 ms
64 bytes from 172.31.1.40: icmp_seq=3 ttl=64 time=0.042 ms
64 bytes from 172.31.1.40: icmp_seq=4 ttl=64 time=0.022 ms
64 bytes from 172.31.1.40: icmp_seq=5 ttl=64 time=0.025 ms
^C
--- 172.31.1.40 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.022/0.029/0.042/0.007 ms
root@test11:~ #

root@test11:~ # ping 172.31.1.1
PING 172.31.1.1 (172.31.1.1): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- 172.31.1.1 ping statistics ---
10 packets transmitted, 0 packets received, 100.0% packet loss
root@test11:~ #


root@san1:~ # iocage get -a test11.2
CONFIG_VERSION:14
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
available:readonly
basejail:no
boot off
bpf:no
children_max:0
cloned_release:11.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize off
count:1
cpuset off
cputime off
datasize off
dedup off
defaultrouter:172.31.1.1
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:test11.2
host_hostuuid:test11.2
host_time:yes
hostid:4c4c4544-004a-4b10-8058-c3c04f365331
hostid_strict_check off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|172.31.1.40/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs off
jail_zfs_dataset:iocage/jails/test11.2/data
jail_zfs_mountpoint:none
last_started:2019-02-18 16:02:48
login_flags:-f root
mac_prefix:02ff60
maxproc off
memorylocked off
memoryuse off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued off
msgqsize off
nmsgq off
notes:none
nsemop off
nshm off
nthr off
openfiles off
origin:readonly
owner:root
pcpu off
priority:99
pseudoterminals off
quota:none
release:11.2-RELEASE-p9
reservation:none
resolver:/etc/resolv.conf
rlimits off
securelevel:2
shmsize off
stacksize off
state:up
stop_timeout:30
swapuse off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse off
vnet on
vnet0_mac:02ff608575aa 02ff608575ab
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:none
vnet_interfaces:none
wallclock off
root@san1:~ #

 

[mrgtstr]

If this thread needs to be moved to a different section, please do.

 

[dlavigne]

Were you able to resolve this?

 

[sretalla]

It may be useful to see the ifconfig from the host in addition to the jail... it seems that the jail network is OK.
I suspect the bridge on the host isn't properly set.

 

[mrgtstr]

Were you able to resolve this?

Haven't yet...

 

[mrgtstr]

It may be useful to see the ifconfig from the host in addition to the jail... it seems that the jail network is OK.
I suspect the bridge on the host isn't properly set.

Here is the output of ifconfig of the host. The vnet adpaters that are showing up I would of thought would of been deleted when the jail is deleted.

root@san1:~ # ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 78:2b:cb:34:60:49
hwaddr 78:2b:cb:34:60:49
inet 172.31.1.35 netmask 0xffffff00 broadcast 172.31.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 78:2b:cb:34:60:4a
hwaddr 78:2b:cb:34:60:4a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
cxgbe0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
ether 00:07:43:10:3c:30
hwaddr 00:07:43:10:3c:30
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
status: active
cxgbe1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
ether 00:07:43:10:3c:38
hwaddr 00:07:43:10:3c:38
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 02:57:af:28:b5:00
inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:17 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000
member: vnet0:16 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 20 priority 128 path cost 2000
member: vnet0:15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 18 priority 128 path cost 2000
member: vnet0:12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 16 priority 128 path cost 2000
member: vnet0:11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000
member: vnet0:6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: vnet0:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000
member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: cxgbe1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000
member: cxgbe0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: plex
options=8<VLAN_MTU>
ether 02:ff:60:14:fa:09
hwaddr 02:24:10:00:07:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:60:14:fa:0a
hwaddr 02:24:10:00:08:0b
ether 02:ff:60:14:fa:0a
hwaddr 02:24:10:00:08:0b
ether 02:ff:60:ae:1b:76
hwaddr 02:24:10:00:08:0b
ether 02:ff:60:51:a2:47
hwaddr 02:24:10:00:08:0b
ether 02:ff:60:51:a2:47
hwaddr 02:24:10:00:08:0b
ether 02:ff:60:51:a2:47
hwaddr 02:24:10:00:08:0b
ether 02:ff:60:51:a2:47
hwaddr 02:24:10:00:08:0b
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
vnet0:5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: plex
options=8<VLAN_MTU>
ether 02:ff:60:14:fa:09
hwaddr 02:24:10:00:09:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: test
options=8<VLAN_MTU>
ether 02:ff:60:ae:1b:75
hwaddr 02:24:10:00:0a:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: Test-11_1
options=8<VLAN_MTU>
ether 02:ff:60:51:a2:46
hwaddr 02:24:10:00:0e:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: Test-11_1
options=8<VLAN_MTU>
ether 02:ff:60:51:a2:46
hwaddr 02:24:10:00:10:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:15: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: Test-11_1
options=8<VLAN_MTU>
ether 02:ff:60:51:a2:46
hwaddr 02:24:10:00:12:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:16: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: Test-11_1
options=8<VLAN_MTU>
ether 02:ff:60:51:a2:46
hwaddr 02:24:10:00:14:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:17: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: associated with jail: test11_2
options=8<VLAN_MTU>
ether 02:ff:60:85:75:aa
hwaddr 02:24:10:00:0d:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
root@san1:~ #

 

[sretalla]

Your bridge has an IP address... I wasn't expecting that.

It looks like the only NIC you have with an IP address, bce0, isn't a member of the bridge... probably why you're locked in.

Perhaps it would be good to understand what is plugged to which switch...

 

[mrgtstr]

Your bridge has an IP address... I wasn't expecting that.

It looks like the only NIC you have with an IP address, bce0, isn't a member of the bridge... probably why you're locked in.

Perhaps it would be good to understand what is plugged to which switch...

Will try to explain my network configuration the best I can...

I have two onboard network cards and I am only using one for my local LAN (172.31.1.0/.24). I have a add-on dual port 10Gb network card that is directly connected to two xenservers to allow for iscsi sharing between the two xenservers. To my understanding In order for me to get the iscsi sharing to work with both xenservers I had to create a bridge using tunables in order for the dual port card to only have one IP address (172.16.1.1).

 

[mrgtstr]

Do I need create another bridge for the jails I create?

 

[sretalla]

I think you want to do LAGG (there's a whole different thing for that) to get the 2 NICs to cooperate. The bridge for your jails needs to bridge the jails and the LAGG interface (and should not need an IP address).

 

[mrgtstr]

I think you want to do LAGG (there's a whole different thing for that) to get the 2 NICs to cooperate. The bridge for your jails needs to bridge the jails and the LAGG interface (and should not need an IP address).

After briefly checking into LAGG, I will see if that will help my issue. The only caveat I see is the network cards are not going into a switch for VLAN tagging and are directly connected to another server.

Thanks for your time! Will see what happens.

 

[mrgtstr]

Out of curiosity after changing the tunable settings for the bridge adapter from 0 to 1 the network for the jail started working. If it happens again will see about configuring LAGG.