Hi there. I have set up an OpenWrt VM in TrueNAS Scale, and I have been experiencing internet drop everyday in the morning so far. At first I thought maybe there is something wrong with the VM so I just reboot the VM but that doesn't help at all and I have to restart TrueNAS to bring the internet back up. When I check out the log, I find the NIC shifting between up and down all the time. Below is just an excerpt and there is no error detected even if I have shutdown the VM using it and resetting its configuration. One of the major reasons for me to build this new NAS is just making the router, the files and all my services in a same place and the router VM even underperforming my 6 years old Netgear router is driving me crazy.
Honestly what it happening? Now I am ridiculously relying on the Wi-Fi as the fail-over connection in my OpenWrt VM which has been frowned upon so many time here because it's considered far less stable than a cabled connection but it is the cable connection that fails all the time.
The NIC involved is a
which is just a very common onboard 1Gbps NIC. Removing this PCI device and rescan doesn't help either. I have read about it somewhere (in PVE forums although I installed TrueNAS on bare metal and use it as the hypervisor) that maybe there is a bug in using offloading in the NIC but I tried turning it off and it makes no difference. Maybe it works only if I have turned off offloading before things go wrong shall I be optimistic. I have set up a post-init script as follows and will report back if the problem persists.
Edit:
So I find I219-V NIC is not being officially supported? OK, I missed that when picking up the components. So the problem I am seeing is supposed to continue happening and never will be fixed? What about passing the NIC through to the VM directly as a PCI device instead of binding to it in TrueNAS? Is that possible?
Edit 2:
Also switching to e1000e instead of virtio to see if things will change as I heard that e1000e exchange some performance for compatibility and I219-V uses e1000e driver on the host machine anyway, but I guess this is only helpful if the root cause of my problem is how the VM uses the NIC rather than TrueNAS handling the physical NIC itself.
Edit 3:
So far, after I disabled offloading in a post-init script, I haven't seen any reset of the NIC in the system log yet, but I think I still need to wait for at least several days before concluding. If the hardware setup is of interest:
No interface down has been reported from TrueNAS so far but the OpenWRT VM does see a link down just now, which I get passed by simply restarting the interface. I am not sure if this is a problem with my new setup or my modem or the ISP because I used to encounter similar issues when I was using my Netgear router. A bit of a digression but to automate the interface restart when the link is down, you can put a script in in OpenWRT for that.
Edit 5:
Despite many other problems with TrueNAS Scale (as I just upgraded to Cobia Beta) that makes me reboot the server from time to time, the OpenWRT VM is now working just fine.
Edit 6:
Another week with no networking problems has passed. I am marking this as solved.
Code:
Aug 13 10:58:50 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 10:59:04 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 10:59:17 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 10:59:32 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 10:59:47 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:00:01 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:00:16 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:00:30 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:00:45 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:00:59 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:01:14 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:01:28 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:01:42 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:01:57 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:02:12 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:02:27 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:02:41 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:02:56 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:03:10 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:03:24 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:03:38 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:03:53 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:04:07 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:04:22 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:04:31 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Aug 13 11:04:40 TrueNAS kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Honestly what it happening? Now I am ridiculously relying on the Wi-Fi as the fail-over connection in my OpenWrt VM which has been frowned upon so many time here because it's considered far less stable than a cabled connection but it is the cable connection that fails all the time.
The NIC involved is a
Code:
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (17) I219-V (rev 11) Subsystem: ASRock Incorporation Ethernet Connection (17) I219-V Flags: bus master, fast devsel, latency 0, IRQ 147, IOMMU group 12 Memory at 75400000 (32-bit, non-prefetchable) [size=128K] Capabilities: [c8] Power Management version 3 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Kernel driver in use: e1000e Kernel modules: e1000e
which is just a very common onboard 1Gbps NIC. Removing this PCI device and rescan doesn't help either. I have read about it somewhere (in PVE forums although I installed TrueNAS on bare metal and use it as the hypervisor) that maybe there is a bug in using offloading in the NIC but I tried turning it off and it makes no difference. Maybe it works only if I have turned off offloading before things go wrong shall I be optimistic. I have set up a post-init script as follows and will report back if the problem persists.
Code:
ethtool -K enp0s31f6 tso off gso off
Edit:
So I find I219-V NIC is not being officially supported? OK, I missed that when picking up the components. So the problem I am seeing is supposed to continue happening and never will be fixed? What about passing the NIC through to the VM directly as a PCI device instead of binding to it in TrueNAS? Is that possible?
Edit 2:
Also switching to e1000e instead of virtio to see if things will change as I heard that e1000e exchange some performance for compatibility and I219-V uses e1000e driver on the host machine anyway, but I guess this is only helpful if the root cause of my problem is how the VM uses the NIC rather than TrueNAS handling the physical NIC itself.
Edit 3:
So far, after I disabled offloading in a post-init script, I haven't seen any reset of the NIC in the system log yet, but I think I still need to wait for at least several days before concluding. If the hardware setup is of interest:
- Motherboard: Asrock B660M Pro RS
- CPU: i3 13100
- RAM: 16Gx4 at 3200
- Hard drive: Silicon Power 1T SP001TBP34A60M28 for boot; Kingston KC3000 1T stripe; Seagate Exos X14 14Tx4 in RAIDz-1
- Hard disk controllers: N/A
- Network cards: on-board I219-v; Intel X540-AT2
No interface down has been reported from TrueNAS so far but the OpenWRT VM does see a link down just now, which I get passed by simply restarting the interface. I am not sure if this is a problem with my new setup or my modem or the ISP because I used to encounter similar issues when I was using my Netgear router. A bit of a digression but to automate the interface restart when the link is down, you can put a script in
Code:
/etc/hotplug.d/net/
Edit 5:
Despite many other problems with TrueNAS Scale (as I just upgraded to Cobia Beta) that makes me reboot the server from time to time, the OpenWRT VM is now working just fine.
Edit 6:
Another week with no networking problems has passed. I am marking this as solved.
Last edited: