freenas.local kernel log messages: arp: packet with invalid ethernet address length 0 received on bge0

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
OS Version:
FreeNAS-11.2-U6
(Build Date: Sep 17, 2019 0:16)

HP ProLiant Microserver Gen 8
Processor:

Intel(R) Celeron(R) CPU G1610T @ 2.30GHz (2 cores)
Memory:
16 GiB

Setup:
4x 1.36TB Samsung HDD's in a RAID5 setting (ada0,ada1,ada2,ada3)
Boot Pool on ada4
Grub Loader on da0 (needed to be able to boot from ada4 without the use of Hardware-RAID [Gen8-Specific] )

Overall Status:
Device: /dev/ada2, Self-Test Log error count increased from 8 to 9
Device: /dev/ada3, Self-Test Log error count increased from 4 to 5
Device: /dev/ada3, 1 Offline uncorrectable sectors
Device: /dev/ada2, 1 Offline uncorrectable sectors
Device: /dev/ada2, 2 Currently unreadable (pending) sectors

Smartctl for ada2:
Code:
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   100   099   051    -    57
  3 Spin_Up_Time            POS---   073   073   011    -    8780
  4 Start_Stop_Count        -O--CK   092   092   000    -    8168
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   100   100   051    -    2
  8 Seek_Time_Performance   P-S--K   100   100   015    -    12978
  9 Power_On_Hours          -O--CK   086   086   000    -    70424
 10 Spin_Retry_Count        PO--CK   100   100   051    -    0
 11 Calibration_Retry_Count -O--C-   100   100   000    -    16
 12 Power_Cycle_Count       -O--CK   100   100   000    -    202
 13 Read_Soft_Error_Rate    -OSR--   100   099   000    -    56
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        PO--CK   100   100   000    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    503
188 Command_Timeout         -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   075   056   000    -    25 (Min/Max 16/26)
194 Temperature_Celsius     -O---K   075   055   000    -    25 (Min/Max 16/28)
195 Hardware_ECC_Recovered  -O-RC-   100   100   000    -    268752977
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O--C-   100   100   000    -    2
198 Offline_Uncorrectable   ----CK   100   100   000    -    1
199 UDMA_CRC_Error_Count    -OSRCK   100   100   000    -    0
200 Multi_Zone_Error_Rate   -O-R--   100   100   000    -    0
201 Soft_Read_Error_Rate    -O-R--   099   099   000    -    1

SMART Extended Self-test Log Version: 1 (2 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       70%      4877         557853748
# 2  Short offline       Completed: read failure       20%      4802         2757152908
# 3  Short offline       Completed: read failure       20%      4586         2757152907
# 4  Short offline       Completed: read failure       20%      4418         2757152907
# 5  Extended offline    Completed: read failure       90%      4395         2757152907
# 6  Short offline       Completed: read failure       20%      4253         2757152908
# 7  Extended offline    Completed: read failure       50%      4161         1121334378
# 8  Short offline       Completed without error       00%      4085         -
# 9  Short offline       Completed without error       00%      3845         -
#10  Short offline       Completed without error       00%      3677         -
#11  Extended offline    Completed: read failure       50%      3658         1216029813
#12  Short offline       Completed without error       00%      3509         -
#13  Extended offline    Completed: read failure       50%      3418         1216029813
#14  Short offline       Completed without error       00%      3341         -
#15  Short offline       Completed without error       00%      3125         -
#16  Short offline       Completed without error       00%      2957         -
#17  Extended offline    Completed: read failure       50%      2938         1216029813
#18  Short offline       Completed without error       00%      2789         -
#19  Extended offline    Completed: read failure       50%      2698         1216029813
#20  Short offline       Completed without error       00%      2621         -
#21  Short offline       Completed without error       00%      2382         -


Smartctl for ada3:
Code:
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   100   099   051    -    57
  3 Spin_Up_Time            POS---   073   073   011    -    8780
  4 Start_Stop_Count        -O--CK   092   092   000    -    8168
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   100   100   051    -    2
  8 Seek_Time_Performance   P-S--K   100   100   015    -    12978
  9 Power_On_Hours          -O--CK   086   086   000    -    70424
 10 Spin_Retry_Count        PO--CK   100   100   051    -    0
 11 Calibration_Retry_Count -O--C-   100   100   000    -    16
 12 Power_Cycle_Count       -O--CK   100   100   000    -    202
 13 Read_Soft_Error_Rate    -OSR--   100   099   000    -    56
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        PO--CK   100   100   000    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    503
188 Command_Timeout         -O--CK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   075   056   000    -    25 (Min/Max 16/26)
194 Temperature_Celsius     -O---K   075   055   000    -    25 (Min/Max 16/28)
195 Hardware_ECC_Recovered  -O-RC-   100   100   000    -    268752977
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O--C-   100   100   000    -    2
198 Offline_Uncorrectable   ----CK   100   100   000    -    1
199 UDMA_CRC_Error_Count    -OSRCK   100   100   000    -    0
200 Multi_Zone_Error_Rate   -O-R--   100   100   000    -    0
201 Soft_Read_Error_Rate    -O-R--   099   099   000    -    1

SMART Extended Self-test Log Version: 1 (2 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       70%      4877         557853748
# 2  Short offline       Completed: read failure       20%      4802         2757152908
# 3  Short offline       Completed: read failure       20%      4586         2757152907
# 4  Short offline       Completed: read failure       20%      4418         2757152907
# 5  Extended offline    Completed: read failure       90%      4395         2757152907
# 6  Short offline       Completed: read failure       20%      4253         2757152908
# 7  Extended offline    Completed: read failure       50%      4161         1121334378
# 8  Short offline       Completed without error       00%      4085         -
# 9  Short offline       Completed without error       00%      3845         -
#10  Short offline       Completed without error       00%      3677         -
#11  Extended offline    Completed: read failure       50%      3658         1216029813
#12  Short offline       Completed without error       00%      3509         -
#13  Extended offline    Completed: read failure       50%      3418         1216029813
#14  Short offline       Completed without error       00%      3341         -
#15  Short offline       Completed without error       00%      3125         -
#16  Short offline       Completed without error       00%      2957         -
#17  Extended offline    Completed: read failure       50%      2938         1216029813
#18  Short offline       Completed without error       00%      2789         -
#19  Extended offline    Completed: read failure       50%      2698         1216029813
#20  Short offline       Completed without error       00%      2621         -
#21  Short offline       Completed without error       00%      2382         -

------------------------------------------------------------------------------------------------------------------

Hello everyone,

ten days ago I started getting security run outputs with the following message:

freenas.local kernel log messages:
arp: packet with invalid ethernet address length 0 received on bge0
arp: packet with invalid ethernet address length 0 received on bge0

-- End of security output --


In this example I have two arp log messages – that was on the first day. The days after that it would be more, sometimes up to 11 times,
and always with the exact same log message.

The server is connected to the Internet, and it now runs for about a little less then a year – but I can't recall of ever having a
security output like that one.

As mentioned in the overall status above, the disks in my RAID seem to be slowly dying.
I don't know if this has something to do with it – since the output seems to have not – but I thought I'll include it also, just to give you a full picture of the situation.

When using arp -a, I don't see devices, that shouldn't be there.
In the ifconfig I couldn't find anything, that would lead to the problem:

Code:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 1c:98:ec:0f:96:74
        hwaddr 1c:98:ec:0f:96:74
        inet 192.168.1.30 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active


On the web I found next to nothing regarding this log message, and nothing on how to solve it.

Thanks to everyone in advance for helping me on this one.


Best regards,
FreeBruce
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
The server is connected to the Internet

Seriously? You connected your server directly to the Internet without any firewalls or other protection? Those log messages means someone's trying to pwn the server by mangling packets in hope of lucking into a buffer overflow or other way in.
 

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
It's behind a sophos firewall – so no, it's not directly connected to the internet.

So you think this message means just that – even with a firewall in front of the server?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
What would be your suggestion on what to do now, after pulling the plug – which I just did.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
  1. First, update your firewall to the latest release.
  2. The quickest way to recover FreeNAS is to reinstall to new boot media, and then to reload the configuration and reimport your pools. If you don't have a config saved, then you're stuck typing in everything again by hand.
  3. Also assume your data pool is compromised. If you have snapshots from before the messages started, consider rolling back to those snapshots.
  4. Don't forget to check your PCs and clients for FreeNAS. Scan them all with multiple security suites (e.g., MalwareBytes + your preferred antivirus + a rootkit detector). If you have backups, consider rolling back to the last backup before the messages started.
  5. Longer-term consider moving to an open-source firewall, like pfSense or OpenWRT, which are updated much more frequently than the commercial home/small business firewalls.
 

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
Thank you for your fast reply and help.

I ran some checks over the weekend, and all hot fixes were present on the Sophos firewall – it gets them automacially. Also I have no forwarding set anywhere – you can only communicate to the outside – and even IF someone or something would try to get its way inside – it would 1.) definitely not work a second time, especially not periodically, and 2.) I would would have logs by the firewall, that would tell me about it.

But here I have no logs, and no setup that allows incoming traffic whatsoever.

Also all the machines, that communicate with the NAS are macOS based – so it's very unlikely that they could have been infected by some kind of trojan – and I am super careful with what sites I visit – even on a mac.

With that being said, I believe it is very unlikely, that I have been pwned by someone.

Are there other reasons, that could cause this kind of security log?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
If you're confident you're not pwned, then there's some quirk on your local network that's garbling ARPs. This old FreeBSD bug indicates this can happen with D-Link switches with loopback detection enabled on the uplink port. Can you describe your LAN topology, and what devices you're using?
 

Fredda

Guru
Joined
Jul 9, 2019
Messages
608
Apart from the security issues: you should also take care of your drives. They show several SMART errors and
the self tests don't even finish successfully. After 8 years this can happen ....
 

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
Ok I checked all the hardware, and this is how the topology looks like:

Internet Connection -> Server With Sophos -> Main Managed Switch (Cisco) -> Switch (Netgear)

Alongside with the Internet connection, FreeNAS and all my Macs are connected to the Netgear switch:

Netgear Switch = [Mac Pro, G4, Zyxel Wlan Router]

All the other switches that are present are either from Cisco or Netgear – so not one from D-Link.

Sidenote:
The log messages are not always the same in number, for example yesterday I got 15, an this morning it were 10.
Maybe that helps.

@Fredda
Yeah I am about to change the drives – they are clearly dying now.
 
Last edited:

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
Could it have something to do with pending updates for FreeNAS?
It gave me an update alert for quite some time, which I ignored. Now that I have installed the update, the kernel log message report is
gone for the first time.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398

FreeBruce

Cadet
Joined
Jul 10, 2020
Messages
7
Do you have a theory what else it could be, apart from pwn or D-Link loopback error?

After the update this is what I got alongside with my daily run:

freenas.local changes in mounted filesystems:
11c11
< freenas-boot/ROOT/default / zfs rw,noatime,nfsv4acls 0 0
---
freenas-boot/ROOT/11.2-U8 / zfs rw,noatime,nfsv4acls 0 0

freenas.local kernel log messages:
FreeBSD 11.2-STABLE #0 r325575+4710c8b6420(HEAD): Fri Feb 14 13:59:19 UTC 2020
root@tnbuild02.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
CPU: Intel(R) Celeron(R) CPU G1610T @ 2.30GHz (2294.84-MHz K8-class CPU)
Timecounter "TSC-low" frequency 1147420130 Hz quality 1000
est: cpu_vendor GenuineIntel, msr 1b0f00001700
est: cpu_vendor GenuineIntel, msr 1b0f00001700
ugen1.1: <0x1912 XHCI root HUB> at usbus1
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub1: <0x1912 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub1: 6 ports with 6 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub4 on uhub2
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen3.2: <vendor 0x8087 product 0x0024> at usbus3
uhub5 on uhub0
uhub5: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus3
uhub5: 6 ports with 6 removable, self powered
uhub6 on uhub5
umass0 on uhub4
random: unblocking device.
Trying to mount root from zfs:freenas-boot/ROOT/11.2-U8 []...

-- End of security output --


Just in case it could be of value.
 
Top