Network monitoring by IP

[iliak]

systat -ifstat give only total network usage, i would like to filter by IP is it possible ?

 

[droeders]

I've used iftop for this under FreeBSD. Not sure if this is available under FreeNAS or not.

 

[iliak]

nop, it is not available and i cant install it under root user

 

[droeders]

You could install it in a FreeBSD jail with the pkg command.

 

[iliak]

i don't know how,, can direct to the help page or give me more details how to do it

 

[Kevin Horton]

i don't know how,, can direct to the help page or give me more details how to do it

Read the Jails section of the FreeNAS docs.

 

[iliak]

Read the Jails section of the FreeNAS docs.

thanks ill give it a try. still learning my way around freenas and DevOps..

 

[iliak]

Kevin Horton

I have managed to make it work,

But how i can configure multiple interfaces for jail?

UPDATE:

I configured static ip, on one select interface (the one i want to monitor)
i now have ping to other computers trough the network
but running iftop gives me this error:

Code:

iftop  -i mlxen0
interface: mlxen0
IP address is: 192.168.11.34
MAC address is: 00:02:ffffffc9:fffffff0:0b:11
pcap_open_live(mlxen0): mlxen0: (cannot open device) /dev/bpf: No such file or directory

when running without -i it gives:

Code:

interface: igb0
Unable to get IP address for interface: igb0
ioctl(SIOCGIFADDR): Can't assign requested address
MAC address is: 00:25:ffffff90:ffffff89:ffffffc3:2a
pcap_open_live(igb0): igb0: (cannot open device) /dev/bpf: No such file or directory

while the ip is configured via mlxen0

 

[droeders]

I would make sure the Berkeley Packet Filter setting for your jail is checked. I see some other settings in the docs that might be needed, but I'd start with this.

See here:

https://www.ixsystems.com/documentation/freenas/11.2/jails.html#jail-basic-props-tab

 

[iliak]

I would make sure the Berkeley Packet Filter setting for your jail is checked. I see some other settings in the docs that might be needed, but I'd start with this.
.....

on the secondary ip i don't have dhcp server ( all ips are statically set)

i tried multiple settings combinations, none worked.

 

[droeders]

on the secondary IP i don't have dhcp server ( all ips are statically set)

I don't see the relevance of static vs DHCP assignment for your IP address. We just need raw sockets/access to the network interface you want to monitor with iftop. That's what BPF does in BSD, so you'll definitely want that enabled for your jail.

i tried multiple settings combinations, none worked.

This doesn't tell me much at all. I only suggested that you enable the Berkeley Packet Filter setting. I like to troubleshoot things one step at a time rather than change a bunch of things at once.

Some questions:
- I assume you're using the root user when you run iftop in your jail. Is that correct?

- I assume you have mlxen0 selected in the 'IPv4 Interface' setting of the jail basic properties?

- When you run this command in your jail, does it show the correct interface and IP settings?

ifconfig mlxen0


Besides the BPF setting, I think you're going to need /dev mounted in your jail as well. To do this, I think you need at least the following settings:
- allow_mount
- allow_mount_devfs
- enforce_statfs - set to 0 or 1
- mount_devfs

Then you'll have to restart your jail and make sure that /dev/ is mounted and populated in your jail.

 

[iliak]

Some questions:
- I assume you're using the root user when you run iftop in your jail. Is that correct?

yes

- I assume you have mlxen0 selected in the 'IPv4 Interface' setting of the jail basic properties?

yes

ifconfig mlxen0
- When you run this command in your jail, does it show the correct interface and IP settings?

root@monitoring:~ # ifconfig mlxen0
ifconfig: interface mlxen0 does not exist

ifconfig mlxen0
Besides the BPF setting, I think you're going to need /dev mounted in your jail as well. To do this, I think you need at least the following settings:
- allow_mount
- allow_mount_devfs
- enforce_statfs - set to 0 or 1
- mount_devfs
[/CODE]

i dont know if it needed cause if i use main interface iftop is working well


no i remember i had to set tunables for sys.device.mlx4_core0.mlx4_port1
i think i should do it in the jail. but when i tried to run

Code:

 sysctl sys.device.mlx4_core0.mlx4_port1=eth

i get

Code:

 
sysctl: sys.device.mlx4_core0.mlx4_port1=eth: Operation not permitted

 

[droeders]

root@monitoring:~ # ifconfig mlxen0
ifconfig: interface mlxen0 does not exist

I assume the mlxen0 interface shows up outside of your jail, right? Better yet, please send the output from this, both inside and outside of the jail:

ifconfig -a

i don't know if it needed cause if i use main interface iftop is working well

What interface is working with iftop? And you're sure that mlxen0 is selected for your jail, correct?

no i remember i had to set tunables for sys.device.mlx4_core0.mlx4_port1
i think i should do it in the jail. but when i tried to run

Code:

 sysctl sys.device.mlx4_core0.mlx4_port1=eth

i get

Code:

sysctl: sys.device.mlx4_core0.mlx4_port1=eth: Operation not permitted

I'm not familiar with these tunables, what they do, or with these Mellanox cards in general. I think the bigger issue is getting your network interface visible within the jail.

 

[iliak]

@droeders

ifconfig outside jail:

Code:

bxe0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:25:90:89:95:40
    hwaddr 00:25:90:89:95:40
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
bxe1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
    ether 00:25:90:89:95:42
    hwaddr 00:25:90:89:95:42
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:89:c3:2a
    hwaddr 00:25:90:89:c3:2a
    inet XXX.XXX.XXX.232 netmask 0xffffff00 broadcast XXX.XXX.XXX.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:89:c3:2b
    hwaddr 00:25:90:89:c3:2b
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:02:c9:f0:0b:11
    hwaddr 00:02:c9:f0:0b:11
    inet XXX.XXX.XXX.232 netmask 0xffffff00 broadcast XXX.XXX.XXX.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
    status: active
mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:02:c9:f0:0b:12
    hwaddr 00:02:c9:f0:0b:12
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:8a:db:25:b9:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 20000

the mlxen0 is select as the interface in the jail settings
ifconfig inside jail

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo 

these tunables force the melanox device to work in ethernet mode instead of infiniband

 

[droeders]

I don't know why your mlxen0 interface isn't available in the jail. Unfortunately, I only run vanilla FreeBSD so I can't try anything on my machines without more effort than I'm willing to expend.

Maybe try passing through one of your other interfaces instead, like igb0 or bxe0?

Any other forum members using physical network interfaces in their jails?

 

[iliak]

all the other interfaces are not in use,, only two are, the default igb0 (which iftop works perfectly when i am not configuring any interface) and the mlxen0 (that holds all the load 500-1000MBs and growing,, added more servers to the cluster )