Citadel - Build Plan and Log

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
After the last update (11.1-U4) logging in to the new UI brings up a modal error message:
Is this error generated when attempting to login to the "New" GUI or when logging into the (what I call) the normal GUI? The new one is still riddled with bugs because it is in development.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Is this error generated when attempting to login to the "New" GUI or when logging into the (what I call) the normal GUI? The new one is still riddled with bugs because it is in development.

Only the new GUI. Right now I'm only using the new one for jails administration, and the default GUI for everything else.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Only the new GUI.
You might want to check the bug tracking system. It could be something they have already fixed in the nightly releases, or it might be something new that they are not aware of. It might be best to submit a bug report, just in case, and let the developers make the call if it is a broken thing that needs fixing, because they are getting close to releasing the next version, if I recall correctly.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
It was filed against 11.1-U2 and U3 though, and the last comment sounds like it should be fixed in 11.1-U4.. Would that be something I'd file a separate bug report for?
It might be good to let them know that what they thought was fixed isn't actually fixed. They can't test every possibility, so it is good to give them feedback.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
I plan on filing that bug report, but want to dig around a little more before I do.

Yesterday I tried transferring some ~400GB of old system backups from my desktop to the FreeNAS box, and when it was done I had this error emailed with the daily security output:

Code:
May 29 00:00:00 bns-citadel syslog-ng[1585]: Configuration reload request received, reloading configuration;
May 29 00:50:05 bns-citadel MCA: Bank 8, Status 0x8c0000400001009f
May 29 00:50:05 bns-citadel MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
May 29 00:50:05 bns-citadel MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 0
May 29 00:50:05 bns-citadel MCA: CPU 0 COR (1) RD channel ?? memory error
May 29 00:50:05 bns-citadel MCA: Address 0x2a9c7880
May 29 00:50:05 bns-citadel MCA: Misc 0x10000000081281


From some brief searching, it looks like some of my RAM may be bad :eek:
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
From some brief searching, it looks like some of my RAM may be bad
Not all of it. This is the good thing about ECC memory. It told you that there was an error and the system was so kind as to tell you which bank (Bank 8) the error came from.
It is probably just a single stick of memory that needs to be replaced. CPU 0, memory slot 8. I can't recall if the slots are numbered 0-8 or 1-9 but you should be able to run the built-in memory diagnostic on the Dell system by pressing F-12 when the Dell logo first comes up. That should invoke the boot menu and one of the options should be to run diagnostics.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Thanks Chris, today after work I'll bring the box down and take a look at the RAM. Since it told me the address as well, I wonder if there's a way to test specifically for that error, so I can be sure I pulled the right memory stick. This wasn't caught by days of running memory tests, and I don't want to wash some dataset back and forth over the network just to try and reproduce the error..
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
In the meantime, I'm trying to figure out if the file transfer was successful.

I copied the files with something like
Code:
rsync -v -r -t -z -a -p -X --delete-delay --delay-updates -e "ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=yes" /mnt/systems/ "root"@citadel:"/mnt/main-pool/systems"

That I stole from a FreeNAS rsync task cronjob.

But the Web UI shows only ~350GiB, where du -hcs /mnt/systems/ on the original directory returns 495G. That should too much difference to be just units or compression.

If I use sshfs to mount the FreeNAS storage, I can run `du` on both:
Code:
$ du -csx /mnt/systems/																																			 
517712912	   total

Code:
$ du -csx sshfs/
516653036	   total

Closer, but the folders aren't exactly the same size. Since maybe just having the files on two different types of filesystem might affect the reported size, I tried running diff on the directories:
Code:
$ diff -qr --no-dereference /mnt/systems/ sshfs/

Which returned nothing, indicating identical folders, which I guess is good enough for me.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
But the Web UI shows only ~350GiB, where du -hcs /mnt/systems/ on the original directory returns 495G. That should too much difference to be just units or compression.
Not at all, compression can make an enormous difference, depending on how compressible the data is.
I have a server at work where the uncompressed data is 265TB, but the space on disk is only 115TB. It is a lot of small files with text and numbers...
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
That's good to know! I didn't mean to dismiss it out of hand. If I can get the numbers to line up, I'd like to try, so here goes:

Reported size on my desktop: 496GB

Reported size of dataset in FreeNAS: 386.5 GiB
Compression ratio of dataset: 1.05x
Compression ratio of pool: 1.08x
386.5 * 1.05 * 1.08 = 438.3 GiB => 470.6 GB

I don't know if that math is right, I just assume the compression ratios of sub-collections are on top of the ratio of the parent ones. At this point I'm mostly just curious to know what's causing the difference. Maybe du or FreeNAS does some rounding on the file sizes?
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Uhoh, I shutdown the system and opened the case: there's only 6 memory slots..

giphy.gif


Going to go try and actually read up on the error and see if it needs to be interpreted differently. Will return with findings.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Thought I was upgrading to 11.1-U4, but wound up with 11.1-U5. Ostensibly the upgrade seems to have gone well.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Whoops, never made the jump to NFS on Citadel. Doing that today and cutting the old NAS out.

I made the share enforce user 'nobody' and then chown'ed the existing data. I don't like how NFS handles permissions, but I guess this is the best I can do with my capabilities and what's available.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Got this in a security email today:

bns-citadel.csb.sh kernel log messages:
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 0
> MCA: CPU 0 COR (1) RD channel ?? memory error
> MCA: Address 0x2a9c7880
> MCA: Misc 0x10000000081180

-- End of security output --

I'm hoping it means that ECC caught something and all is well, but I don't know.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
I'm having trouble managing jails with pkg now. Some of the packages install broken.

I'm on FREENAS-11-STABLE, and have been waiting for it to tick up to 11.2, but I'm not sure if that's actually going to happen at all. Is jumping to FREENAS-11.2-STABLE the "correct" way to move forward?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,457
I'm on FREENAS-11-STABLE, and have been waiting for it to tick up to 11.2, but I'm not sure if that's actually going to happen at all.
The jails can never be on a newer FreeBSD version than the underlying system (older yes, newer no). If you upgrade to FreeNAS 11.2, and the jails are running under iocage, you'll be able to update them. FreeBSD 11.1 has been EOL for a while now, and it's impacting pkg availability and compatibility.
 

ctag

Patron
Joined
Jun 16, 2017
Messages
225
Thanks danb35.

I used the web UI to switch to the 11.2-stable track and upgraded the system. The new javascript-y UI is now default, and I can tell it's improved a lot since the last time I used it.

I followed instructions to switch the jail over to 11.2 as well, and now the node-js packages work!
 
Top