UI returning a 404 error on 11.2

[skullitor13]

So when I go to the freenas box it is now giving me a 404 error. The server is up and running and I can SSH into it and the shares are still working just the UI seems to be broken. I tried to restart the box and it did not fix the issue.

The UI was working before and I don't think I made any changes. Anyone have any ideas?

 

[dlavigne]

Were you able to resolve this? If not, is there anything in /var/log/messages when you SSH into the system?

 

[dropio]

I'am facing the same issue at the moment.
Installing freenas a couple of days before, start uploading my data and sort it a little bit via SMB from my fedora laptop and had no problems until yesterday. Today i'am getting a 404 on the freenas webgui (checked with Firefox and curl on command line). Restart ngnix and django reporting no problems, all fine. /var/log/messages holding a lot of hardware informations but nothing that seem related to the problem.

The legacy web gui is responding and i'am able to log me in. But just the html and js scripts can be loaded, all images failing with http status 403 (forbidden). That guides me to some nasty things i did before. Even I had to put a

Code:

chown -R nobody:nogroup

on my data directory, because without it i'am facing lots of permission issues while copying data via SMB to the FreeNAS and Yesterday I reset this with putting a

Code:

chown -R <myuser>:wheelCODE] to my data directory. Maybe this is related.

Or maybe this helps:
Code:
root@***[~]# l /var/run/log
srw-rw-rw-  1 root  wheel  0 Nov 10 14:21 /var/run/log

root@***[/usr/local/www]# l
total 13
drwxr-xr-x   6 root  wheel   6 Oct 30 08:49 apache24
drwxr-xr-x   6 root  wheel   7 Oct 30 08:49 dojo
drwxr-xr-x  27 www   www    34 Oct 30 08:50 freenasUI
lrwxr-xr-x   1 root  wheel  25 Oct 30 08:49 nginx -> /usr/local/www/nginx-dist
dr-xr-xr-x   2 root  wheel   5 Oct 30 08:49 nginx-dist
drwxr-xr-x   3 root  wheel   4 Oct 30 08:49 swagger-ui
drwxr-xr-x   4 root  wheel  39 Oct 30 08:50 webui

root@***[/usr/local/www]# l webui 
total 2678
-rw-r--r--  1 root  wheel   436612 Oct 30 08:50 0.4f2e38e8928582951e83.chunk.js
-rw-r--r--  1 root  wheel   125566 Oct 30 08:50 1.c40647e0833c4ab87bf8.chunk.js
-rw-r--r--  1 root  wheel    18157 Oct 30 08:50 10.5f463e03afcd6858fe95.chunk.js
-rw-r--r--  1 root  wheel    97198 Oct 30 08:50 11.e93f9cfd1a6497acaaa1.chunk.js
-rw-r--r--  1 root  wheel   180223 Oct 30 08:50 12.41d93db73b0b11029aee.chunk.js
-rw-r--r--  1 root  wheel   106252 Oct 30 08:50 13.110d4e107cdac163782e.chunk.js
-rw-r--r--  1 root  wheel    69206 Oct 30 08:50 14.7c764e755971de471758.chunk.js
-rw-r--r--  1 root  wheel   180716 Oct 30 08:50 15.6fbd8b8e77aa2d5f1d3c.chunk.js
-rw-r--r--  1 root  wheel     6440 Oct 30 08:50 16.243085ab6ce60c1d0852.chunk.js
-rw-r--r--  1 root  wheel    98714 Oct 30 08:50 17.42f6e02b1e44e23266a7.chunk.js
-rw-r--r--  1 root  wheel    17439 Oct 30 08:50 18.8d004a570384f91a9857.chunk.js
-rw-r--r--  1 root  wheel    57147 Oct 30 08:50 19.f072cb1f2b3a1c972e00.chunk.js
-rw-r--r--  1 root  wheel   398798 Oct 30 08:50 2.baf8293bcf059cc2e955.chunk.js
-rw-r--r--  1 root  wheel    94323 Oct 30 08:50 3.006704261c193b9a9e63.chunk.js
-rw-r--r--  1 root  wheel    69695 Oct 30 08:50 3rdpartylicenses.txt
-rw-r--r--  1 root  wheel   113298 Oct 30 08:50 4.a21348778375b28e5e5a.chunk.js
-rw-r--r--  1 root  wheel    97682 Oct 30 08:50 5.1b85b5027716704fce87.chunk.js
-rw-r--r--  1 root  wheel    15203 Oct 30 08:50 6.63ed06b3b77ee7327ee3.chunk.js
-rw-r--r--  1 root  wheel   134116 Oct 30 08:50 7.6da46359eb354729bc2b.chunk.js
-rw-r--r--  1 root  wheel     9651 Oct 30 08:50 8.54ede87a57fcef8e6f08.chunk.js
-rw-r--r--  1 root  wheel   261152 Oct 30 08:50 9.98840952f079e724db85.chunk.js
drwxr-xr-x  8 root  wheel       10 Oct 30 08:50 assets
-rw-r--r--  1 root  wheel    19163 Oct 30 08:50 color.c316ed3adb938b0eeed6.png
-rw-r--r--  1 root  wheel     2946 Oct 30 08:50 data-table.0dc1f75c4e8ffc447f9b.eot
-rw-r--r--  1 root  wheel     2666 Oct 30 08:50 data-table.b0aebd744ce7adb780a9.svg
drwxr-xr-x  2 root  wheel        6 Oct 30 08:50 icons
-rw-r--r--  1 root  wheel     2635 Oct 30 08:50 index.html
-rw-r--r--  1 root  wheel     1959 Oct 30 08:50 inline.ebe88ce6598b27a81c4f.bundle.js
-rw-r--r--  1 root  wheel    13388 Oct 30 08:50 line.efbb151ac96f01ffc3e0.gif
-rw-r--r--  1 root  wheel  3725315 Oct 30 08:50 main.91d3f9c4fb631de7493d.bundle.js
-rw-r--r--  1 root  wheel   100521 Oct 30 08:50 polyfills.4e42ab25a6112b333361.bundle.js
-rw-r--r--  1 root  wheel    32126 Oct 30 08:50 primeicons.259fb42d510585f9c01f.ttf
-rw-r--r--  1 root  wheel   115540 Oct 30 08:50 primeicons.38d77552b0353684a208.svg
-rw-r--r--  1 root  wheel    32308 Oct 30 08:50 primeicons.961bca12a77e1b31f2b5.eot
-rw-r--r--  1 root  wheel    32204 Oct 30 08:50 primeicons.e49d843aed5ca49f4fc0.woff
-rw-r--r--  1 root  wheel   533259 Oct 30 08:50 scripts.39cf4374a16bd35016b2.bundle.js
-rw-r--r--  1 root  wheel   307251 Oct 30 08:50 styles.113c07d08bf223b3b9d1.bundle.css

root@***[/usr/local/www]# l freenasUI                
total 67
-rw-r--r--   1 www   www   1365 Oct 30 08:50 __init__.py
drwxr-xr-x   2 www   www      9 Nov 10 14:27 __pycache__
drwxr-xr-x   5 www   www     12 Oct 30 08:50 account
drwxr-xr-x   4 www   www     17 Oct 30 08:50 api
-rw-r--r--   1 www   www  31137 Oct 30 08:50 choices.py
drwxr-xr-x   3 www   www     28 Oct 30 08:50 common
drwxr-xr-x   3 www   www      5 Oct 30 08:50 contrib
drwxr-xr-x   4 www   www     12 Oct 30 08:50 directoryservice
drwxr-xr-x   2 www   www      3 Oct 30 08:50 dojango
drwxr-xr-x   2 www   www      3 Oct 30 08:50 fnstatic
drwxr-xr-x   8 www   www     19 Oct 30 08:50 freeadmin
drwxr-xr-x   4 www   www     14 Oct 30 08:50 jails
lrwxr-xr-x   1 www   www     22 Oct 30 08:50 local_settings.py -> /etc/local_settings.py
drwxr-xr-x  65 www   www     65 Oct 30 08:50 locale
-rwxr-xr-x   1 www   www   1747 Oct 30 08:50 manage.py
drwxr-xr-x   3 www   www     12 Oct 30 08:50 middleware
drwxr-xr-x   4 www   www     13 Oct 30 08:50 network
drwxr-xr-x   5 www   www     14 Oct 30 08:50 plugins
drwxr-xr-x   4 www   www     11 Oct 30 08:50 reporting
drwxr-xr-x   4 www   www     13 Oct 30 08:50 services
-rw-r--r--   1 www   www   8441 Oct 30 08:50 settings.py
drwxr-xr-x   4 www   www     11 Oct 30 08:50 sharing
drwxr-xr-x   7 root  www      9 Oct 30 08:50 static
drwxr-xr-x   5 www   www     13 Oct 30 08:50 storage
drwxr-xr-x   3 www   www      9 Oct 30 08:50 support
drwxr-xr-x   5 www   www     14 Oct 30 08:50 system
drwxr-xr-x   4 www   www     13 Oct 30 08:50 tasks
drwxr-xr-x  17 www   www     21 Oct 30 08:50 templates
drwxr-xr-x   3 www   www     16 Oct 30 08:50 tools
-rw-r--r--   1 www   www   3737 Oct 30 08:50 urls.py
drwxr-xr-x   4 www   www     13 Oct 30 08:50 vm
-rw-r--r--   1 www   www    709 Oct 30 08:50 wsgi.py


I would be happy to provide you with more information.

Any help very apprechiated.

 

[dlavigne]

@dropio which browser(s) have you tried? Are they running any security extensions? Is there a firewall or proxy in the network?

 

[dropio]

@dropio which browser(s) have you tried?

Like i said:
> (checked with Firefox and curl on command line).

Are they running any security extensions? Is there a firewall or proxy in the network?

No, nothing. Just a Fritz.Box as ADSL Modem/Router with the FreeNAS box wired on it.

 

[dropio]

Is there absolutly nothing i can do?
Should i reinstall freenas and rebuild the pool?

 

[toxic2]

I have the exact same issue. I just installed FreeNAS11.2 U6 fresh a few days ago and after a few hours the UI shows 404 errors. I only configured a Pool and SMB and did not setup any Jails etc. I also tried swapping the boot drive from an ssd to an m.2 SATA but the issues appeared again.

nginx started reporting the following errors

Code:

2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:16:32 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:21:44 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:21:44 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:21:44 [alert] 80541#101723: connect() failed (13: Permission denied)
2019/11/20 20:21:44 [alert] 80541#101723: connect() failed (13: Permission denied)

 

[dlavigne]

Has anyone created a report yet at bugs.ixsystems.com? If so, what is the issue number?

 

[toxic2]

Has anyone created a report yet at bugs.ixsystems.com? If so, what is the issue number?

I just created a bug report #104050

 

[jgreifenhagen]

Good day all,
I can confirm the behaviour and added a note to the bug report stating what I did as good as I can recall before it broke...
I will try to "fix" it once more by rebooting my server and installing an "update" from USB. Last timeI at least had a real update from 11.2-U6 to U7.

 

[jgreifenhagen]

short update: The update install via usb stick worked (again) and I have a working system by now.
I took a snapshot of the boot pool to be sure to have an easy return path right now.
I did edit a samba share again, but this time nothing happened...
Let me know if you want a specific information.

 

[dropio]

Today I reinstall freenas (V11.2-U7), to get back the UI. Then I set the checkbox "default permissions" on my smb share. After that, the 404 error returns and i lost the UI again.

 

[anodos]

Today I reinstall freenas (V11.2-U7), to get back the UI. Then I set the checkbox "default permissions" on my smb share. After that, the 404 error returns and i lost the UI again.

Probably same issue as this one https://jira.ixsystems.com/browse/NAS-104050
TL;DR you most likely have a symlink in your pool that points back to '/'. The permissions reset tool follows it to the root of the pool and changes permissions there. I'll PM you a tool you can use to scan for the offending symlink and remove it (or you can wait for the next release). The recursive permissions change fails promptly after affecting `/` so you can probably fix it by running the command setfacl -b / followed by chmod 755 / if you have console access. 11.3 chroots the application that changes permissions into the target path (as well as having other protective measures in place).

 

[hondaman]

Probably same issue as this one https://jira.ixsystems.com/browse/NAS-104050
TL;DR you most likely have a symlink in your pool that points back to '/'. The permissions reset tool follows it to the root of the pool and changes permissions there. I'll PM you a tool you can use to scan for the offending symlink and remove it (or you can wait for the next release). The recursive permissions change fails promptly after affecting `/` so you can probably fix it by running the command setfacl -b / followed by chmod 755 / if you have console access. 11.3 chroots the application that changes permissions into the target path (as well as having other protective measures in place).

Do you mind if I can get that tool? I'm having exactly the same problem of setting up shares, rebooting and the GUI not being accessible.

 

[anodos]

Do you mind if I can get that tool? I'm having exactly the same problem of setting up shares, rebooting and the GUI not being accessible.

Sure. I'll attach it. Run it as follows: scanlnk -a full -p /mnt/tank where your pool is named 'tank'. 11.2-U8 and 11.3 will include the tool, but not be susceptible to this particular issue. In general the scan should be quite fast (but not perfect - optimized for speed), in my test VM it was checking files at a rate of about 20K files per second.

 

[Paul Lupa]

I have the same issue.
I see version 11.2-Stable but no 11.3 yet.
chmod on / allows the UI to work again.

Not sure where to go from here. Does the chmod on "/" fix it till the next reboot?

 

[anodos]

I have the same issue.
I see version 11.2-Stable but no 11.3 yet.
chmod on / allows the UI to work again.

Not sure where to go from here. Does the chmod on "/" fix it till the next reboot?

This will be fixed in 11.2-U8. The design issue was quite old (going back to 2014 or so), but it didn't become apparent until we changed what the default ACL was. The chmod will persist across reboots. You can run the tool I uploaded to find the problematic symlink(s) and remove them.

 

[Paul Lupa]

This will be fixed in 11.2-U8. The design issue was quite old (going back to 2014 or so), but it didn't become apparent until we changed what the default ACL was. The chmod will persist across reboots. You can run the tool I uploaded to find the problematic symlink(s) and remove them.

Running "scanlnk". Had to "grep -v 'lstat failed' " to get rid of the noise, but did find two files so far...

I'll report back, but I expect it will solve the issue.

 

[Paul Lupa]

I removed a few, but quickly ran into the issue of simlinks that should not be deleted.
Such as links in UNIX home directories that point to root locations.
Example:
/mnt/nas903/Homes/pXXXa/unix/pXXXXa/.blender/locale
which points to /usr/share/locale

I turned off SMB in hopes that it will prevent the ACL corruption of "/"

 

[Paul Lupa]

This will be fixed in 11.2-U8. The design issue was quite old (going back to 2014 or so), but it didn't become apparent until we changed what the default ACL was. The chmod will persist across reboots. You can run the tool I uploaded to find the problematic symlink(s) and remove them.

See earlier note, removing links will break various unix programs.

Implmented an ugly solution.
Added a cron job to correct the permissions on "/" every 15 seconds.... :(

Job runs every minute, corrects "/" ( sleeps 15 seconds corrects "/" ) .... 'lather and repeat'

Looking forward to the update.