Fresh install of FreeNAS 11.2-U4.1, networking works fine and the system is otherwise behaving normally, but when I attempt to search for available plugins, I get the error:
None: Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by None)
Expanding that error message gets the following python traceback:
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 894, in get_plugin_version
'iocage_rpkgdict')
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 103, in call_sync
return self.client.call(method, *params, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 451, in call
raise c.py_exception
KeyError: 'iocage_rpkgdict'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
OSError: [Errno 65] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x81b7564e0>: Failed to establish a new connection: [Errno 65] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='pkg.cdn.trueos.org', port=80): Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x81b7564e0>: Failed to establish a new connection: [Errno 65] No route to host',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 131, in main_worker
res = loop.run_until_complete(coro)
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 91, in _run
return await self._call(name, serviceobj, methodobj, params=args, job=job)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 84, in _call
return methodobj(*params)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 84, in _call
return methodobj(*params)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 554, in list_resource
pv = self.get_plugin_version(plugin[2])
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 900, in get_plugin_version
f'http://pkg.cdn.trueos.org/iocage/{branch}/All')
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='pkg.cdn.trueos.org', port=80): Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x81b7564e0>: Failed to establish a new connection: [Errno 65] No route to host',))
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 166, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1097, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1042, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1062, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 996, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 972, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
requests.exceptions.ConnectionError: None: Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by None)
The pertinent section seems to be that it is unable to create a route to host pkg.cdn.trueos.org. Other machines on this same network can browse successfully to http://pkg.cdn.trueos.org/iocage/11.2-RELEASE/All/ and see a long list of available packages. Testing connectivity from the freenas server itself seems to indicate a routing failure:
[mrb@freenas ~]$ telnet pkg.cdn.trueos.org 80
Trying 68.70.205.4...
telnet: connect to address 68.70.205.4: Connection refused
Trying 68.70.205.3...
telnet: connect to address 68.70.205.3: Connection refused
Trying 68.70.205.2...
telnet: connect to address 68.70.205.2: Connection refused
Trying 68.70.205.1...
telnet: connect to address 68.70.205.1: Connection refused
Trying 2a0b:4d07:2::4...
telnet: connect to address 2a0b:4d07:2::4: No route to host
Trying 2a0b:4d07:2::3...
telnet: connect to address 2a0b:4d07:2::3: No route to host
Trying 2a0b:4d07:2::2...
telnet: connect to address 2a0b:4d07:2::2: No route to host
Trying 2a0b:4d07:2::1...
telnet: connect to address 2a0b:4d07:2::1: No route to host
telnet: Unable to connect to remote host
Other network information:
[mrb@freenas ~]$ ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether d8:9e:f3:1f:37:ab
hwaddr d8:9e:f3:1f:37:ab
inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
[mrb@freenas ~]$ cat /etc/resolv.conf
# Generated by resolvconf
search local internal.lan
nameserver 192.168.0.10
nameserver 192.168.0.1
[mrb@freenas ~]$ netstat -r
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.0.1 UGS em0
localhost lo0 UHS lo0
192.168.0.0/24 link#1 U em0
192.168.0.40 link#1 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 localhost UGRS lo0
localhost lo0 UHS lo0
::ffff:0.0.0.0/96 localhost UGRS lo0
fe80::/10 localhost UGRS lo0
fe80::%em0/64 link#1 U em0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
ff02::/16 localhost UGRS lo0
To my eyes, that all appears to be correct. There are no vlans or anything at play, this is a pretty straightforward home network. I'm pretty stumped. Any input would be greatly appreciated.
None: Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by None)
Expanding that error message gets the following python traceback:
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 894, in get_plugin_version
'iocage_rpkgdict')
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 103, in call_sync
return self.client.call(method, *params, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py", line 451, in call
raise c.py_exception
KeyError: 'iocage_rpkgdict'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
OSError: [Errno 65] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x81b7564e0>: Failed to establish a new connection: [Errno 65] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='pkg.cdn.trueos.org', port=80): Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x81b7564e0>: Failed to establish a new connection: [Errno 65] No route to host',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 131, in main_worker
res = loop.run_until_complete(coro)
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 91, in _run
return await self._call(name, serviceobj, methodobj, params=args, job=job)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 84, in _call
return methodobj(*params)
File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 84, in _call
return methodobj(*params)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 554, in list_resource
pv = self.get_plugin_version(plugin[2])
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 900, in get_plugin_version
f'http://pkg.cdn.trueos.org/iocage/{branch}/All')
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='pkg.cdn.trueos.org', port=80): Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x81b7564e0>: Failed to establish a new connection: [Errno 65] No route to host',))
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 166, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1097, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1042, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1062, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 996, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 972, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
requests.exceptions.ConnectionError: None: Max retries exceeded with url: /iocage/11.2-RELEASE/All (Caused by None)
The pertinent section seems to be that it is unable to create a route to host pkg.cdn.trueos.org. Other machines on this same network can browse successfully to http://pkg.cdn.trueos.org/iocage/11.2-RELEASE/All/ and see a long list of available packages. Testing connectivity from the freenas server itself seems to indicate a routing failure:
[mrb@freenas ~]$ telnet pkg.cdn.trueos.org 80
Trying 68.70.205.4...
telnet: connect to address 68.70.205.4: Connection refused
Trying 68.70.205.3...
telnet: connect to address 68.70.205.3: Connection refused
Trying 68.70.205.2...
telnet: connect to address 68.70.205.2: Connection refused
Trying 68.70.205.1...
telnet: connect to address 68.70.205.1: Connection refused
Trying 2a0b:4d07:2::4...
telnet: connect to address 2a0b:4d07:2::4: No route to host
Trying 2a0b:4d07:2::3...
telnet: connect to address 2a0b:4d07:2::3: No route to host
Trying 2a0b:4d07:2::2...
telnet: connect to address 2a0b:4d07:2::2: No route to host
Trying 2a0b:4d07:2::1...
telnet: connect to address 2a0b:4d07:2::1: No route to host
telnet: Unable to connect to remote host
Other network information:
[mrb@freenas ~]$ ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether d8:9e:f3:1f:37:ab
hwaddr d8:9e:f3:1f:37:ab
inet 192.168.0.40 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
[mrb@freenas ~]$ cat /etc/resolv.conf
# Generated by resolvconf
search local internal.lan
nameserver 192.168.0.10
nameserver 192.168.0.1
[mrb@freenas ~]$ netstat -r
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.0.1 UGS em0
localhost lo0 UHS lo0
192.168.0.0/24 link#1 U em0
192.168.0.40 link#1 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 localhost UGRS lo0
localhost lo0 UHS lo0
::ffff:0.0.0.0/96 localhost UGRS lo0
fe80::/10 localhost UGRS lo0
fe80::%em0/64 link#1 U em0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
ff02::/16 localhost UGRS lo0
To my eyes, that all appears to be correct. There are no vlans or anything at play, this is a pretty straightforward home network. I'm pretty stumped. Any input would be greatly appreciated.
