TrueNAS SCALE 24.04-BETA.1 (Dragonfish) has been released!

[Ericloewe]

ARC is (mostly? The details aren't clear to me) wired a bit oddly into the Linux kernel, as I understand it due to limitations that would have required substantial upstream patches that were (clearly going to be) rejected due to NIH syndrome and extreme hostility from the Linux kernel maintainers. So the kernel does not (did not?) fully understand that ARC is purely a cache, so the "hey, I need memory now" mechanism is one more geared towards applications that can do some cleanup (e.g. consolidate stuff into fewer memory pages) in some cases, and does not know how much memory can be made available.

 

[winnielinnie]

This means it can't be too slow as the memory allocation will not return until the ZFS ARC has coughed up enough memory.

This is what I'm trying to understand about SCALE and how it's supposedly fixed now.

It cannot simply be a matter of "SCALE (Linux) will now use up as much RAM as needed/possible for ARC, just like on FreeBSD". It must also address (seamlessly and with stability) the dynamic needs of non-ARC RAM pressure. I've noticed on FreeBSD ("Core"), that total used physical memory remains just shy of 1GB from capacity, in which the ARC dynamically adjusts (back and forth) based on other real-time requirements. And yet, this 1GB "cushion" remains relatively consistent. (Even with multiple jails involved.) Swap was never used.

If SCALE behaves this intelligently in regards to ARC/RAM, then one of the biggest hurdles, which separates ZFS on FreeBSD vs Linux, has been resolved. (That's a big deal.)

 

[Ericloewe]

My understanding is that things are better, but not completely on the level of the non-hostile kernels.

 

[morganL]

Uh, then it was not the same as Solaris & FreeBSD.

If I understand what Solaris & FreeBSD did, is that when a program requests memory and their is not enough to satisfy the request, ZFS ARC is reduced enough to satisfy the request. (As well as maintain a small buffer of free memory...) This means it can't be too slow as the memory allocation will not return until the ZFS ARC has coughed up enough memory.

The only time it can fail, (and I have seen this in Production on Solaris servers with broken Apps), is when a program mistakenly checks free memory BEFORE requesting it. Then, fails to start because their is not enough free memory.

Nothing is identical between OSes.

We've tested for TrueNAS use-cases.... including VMs and Apps being deployed. The focus was steady-state performance and reliability. We haven't tested for random applications. There is a reason for it being BETA and would like any feedback.

 

[Kris Moore]

What we did test was that memory pressure from services / Apps will indeed cause pressure in kernel and ARC to shrink dynamically to correspond to growing demand elsewhere. That was where the bulk of our testing resided and we feel confident that is working well. That plus a bunch of us have been running with 90%+ ARC on Cobia for months now and no issues, I think helps prove its working pretty decently so far :)

 

[truecharts]

Are TrueCharts apps supposed to work? Not asking for guarantee, just if they are known not to because of breaking changes?

We do not offer support for DragonFish as this time, due to breaking-changes over at iX-Systems.
This is reflected by being unable to install new TrueCharts Apps.

I'm not aware of any breaking changes from Cobia to Dragonfish; however, I'd recommend you ask the TrueCharts community if they have any datapoints.

Removal of OpenEBS ZFS LocalPV, is quite a huge breaking change.

There is potentially an outstanding issue we just discovered yesterday where TrueCharts can consume all available space in /var/run during chart validation on adding new catalog.

NAS-127213 / 24.10 / Do not use ix temp dir when creating catalogs by sonicaj · Pull Request #13100 · truenas/middleware

This PR fixes a problem where we were still using temp dir when a new catalog was being added so we could validate it before adding it to the system. With the changes, now we use the apps dataset t...

github.com

It's affecting some users on Cobia and will be fixed in 23.10.2 and 24.04-RC1.

Not limited to TrueCharts, though we are the only one bug enough, that we're aware of, to trigger these issues.
THe fix is hugely appreciated though!

Is it possible to override the installation error...

Code:

Your system version is greater then specified maximum scale version for the app version

You can hack this, but it would just result in non-functional truecharts apps

What we did test was that memory pressure from services / Apps will indeed cause pressure in kernel and ARC to shrink dynamically to correspond to growing demand elsewhere. That was where the bulk of our testing resided and we feel confident that is working well. That plus a bunch of us have been running with 90%+ ARC on Cobia for months now and no issues, I think helps prove its working pretty decently so far :)

We can confirm this as well, we've done about a 2 years of experiment, Apps and VM's correctly cause shrinkage and memory pressure gets handled well. Even with "unlimited" ARC maximums.

 

[CJRoss]

Can the version codename be added to the System Information widget? For someone just getting started with Scale it can be confusing to try and figure out if they're running Bluefish, Cobia, or Dragonfish.

 

[Jorsher]

Can the version codename be added to the System Information widget? For someone just getting started with Scale it can be confusing to try and figure out if they're running Bluefish, Cobia, or Dragonfish.

Nothing against it being added to the information widget, but for now you can see it in System Settings > Update.

 

[Arwen]

Can the version codename be added to the System Information widget? For someone just getting started with Scale it can be confusing to try and figure out if they're running Bluefish, Cobia, or Dragonfish.

You can always make a feature request using the "Report a Bug" link at the top of any forum web page. If you don't have an account on that other site, it's free to make one.

Generally if you do make a feature request or bug report, you post a link to it in the relevant forum thread. Here in this case.

Plus, you ask for people to "vote" for it. (Thumbs symbol, upper right of the feature request or bug report.) If a forum member's feature request or bug report gets 10 votes, that means it is important enough for iX staff to look at it.

 

[Kris Moore]

Can the version codename be added to the System Information widget? For someone just getting started with Scale it can be confusing to try and figure out if they're running Bluefish, Cobia, or Dragonfish.

Thats a good idea. I'll get a ticket made to get that listed.

 

[CJRoss]

Nothing against it being added to the information widget, but for now you can see it in System Settings > Update.

Are you referring to the train selection? I don't think that's really a valid equivalent.

You can always make a feature request using the "Report a Bug" link at the top of any forum web page. If you don't have an account on that other site, it's free to make one.

Generally if you do make a feature request or bug report, you post a link to it in the relevant forum thread. Here in this case.

Plus, you ask for people to "vote" for it. (Thumbs symbol, upper right of the feature request or bug report.) If a forum member's feature request or bug report gets 10 votes, that means it is important enough for iX staff to look at it.

Making bug reports is often painful because it's yet another account that you have to make which is different from your forum account, so I usually don't bother unless something really bugs me.

The ask for the codename to be added was just something that popped into my head as I was reading this thread and remembering my recent confusion trying to figure out if I was running the version people were complaining about or not. I literally just started dipping my toe into Scale this year after being a Core user for over a decade.

 

[Kris Moore]

Agreed, train names are nice, but it should be on the info widget as well for reference. Seems like a super easy ask, so we'll see if we can do that for RC.1.

 

[Jorsher]

Are you referring to the train selection? I don't think that's really a valid equivalent.

Apologies if you took it that way. I was not calling it an equivalent, it was suggested 'in the meantime.' It's clear that 2 clicks is infinitely more than 0.

Making bug reports is often painful because it's yet another account that you have to make which is different from your forum account, so I usually don't bother unless something really bugs me.

Painful indeed.

 

[morganL]

Apologies if you took it that way. I was not calling it an equivalent, it was suggested 'in the meantime.' It's clear that 2 clicks is infinitely more than 0.

Painful indeed.

Have you tried making bug reports from the UI in Dragonfish?

 

[danb35]

Have you tried making bug reports from the UI in Dragonfish?

Has that process improved compared to Cobia? Because the last time I tried to report a bug through the UI in Cobia, it took over two days before it actually made it into Jira.

 

[morganL]

Has that process improved compared to Cobia? Because the last time I tried to report a bug through the UI in Cobia, it took over two days before it actually made it into Jira.

The UI and debug capture process has improved. The back-end process may have a human filter to avoid noise in Jira.
You'll need to try it a few times to determine mean delay and standard deviation.

 

[joeschmuck]

I was glad to see smartmontools 7.4 on this build, big step forward. Unfortunately NVMe still cannot be configured in the GUI to run a SMART test.

 

[morganL]

I was glad to see smartmontools 7.4 on this build, big step forward. Unfortunately NVMe still cannot be configured in the GUI to run a SMART test.

I see you reported NAS-127306. Thanks

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

The same limitation on NVMe smart test exists for SCALE AFAIK

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

 

[Ericloewe]

NVMe devices seem pretty spotty in their support for SMART tests, which doesn't help.

 

[joeschmuck]

NVMe devices seem pretty spotty in their support for SMART tests, which doesn't help.

That is very true for older NVMe drives. I've found some that provide very little information. The new NVMe drives coming out "should" support he current NVMe specifications and the data will be there, hopefully.