solnet-array-test Discussion Thread

[MrGuvernment]

Hi,

Does this script run the tests in parallel? How do I know how far along it is?
I have it running about 12hrs now, on 6 HDDs.

It runs until you stop it, it is meant as a burn-in-aid so it will keep going and going until you say mercy.

 

[NugentS]

Keep it running for a few days at least. If the disks are large, a week?

 

[jgreco]

jgreco updated solnet-array-test with a new update entry:

I'm not sure how this Resource "update" thingy works. I guess we'll find out.

Merry Christmas to all you believer-in-Santa-Claus folks, Happy Holidays to everyone else, and geez if you're in the US right now with the once-in-a-lifetime bomb cyclone, I hope you're keeping warm next to a pile of happily spinning hard drives.

I've updated solnet-array-test to v3, which includes TrueNAS SCALE (Linux) compatibility, and adds a burn-in mode.

The tool is available via ftp from SOL's ftp archive server. You may use the UNIX "fetch" or "wget" commands to retrieve it. I...

Read the rest of this update entry...

 

[mistermanko]

It seems that the script consumes almost every bit of system memory, which gets me and the TrueNAS GUI kind of sweating. UI elements start to appear slower than usual.
Bluefin with a couple of apps and a VM of 8G.

EDIT: it is also grabbing every bit of swap. This can't be intended, right?

 

[mistermanko]

I now had a total system lock - GUI and IPMI - had to hard reset the server.

Perhaps the script is relatively untested on SCALE, the fact that parallel dd jobs are robbing RAM off of ZFS is something that needs to be addressed.
Maybe more people could test this on Bluefin and report back if they see similar behavior.
I have to find a different tool to burn-in my disk for now. Is badblocks SCALE compatible?

 

[NugentS]

badblocks does work - note that after a certain size it won't do an entire disk unless you expand the block size (usually 4096) which some say makes the results iffy. So you either have to do the disk in 2 runs or run a bs of 8192.

I have tested the solnet test program on a bunch of disks - seems to work without issue. I didn't notice any memory issue but I do have a quite a lot of memory.

 

[mistermanko]

I have tested the solnet test program on a bunch of disks

On SCALE Bluefin?

 

[NugentS]

Yes

 

[mistermanko]

Hm, I don't know why, but running even one iteration of dd if=/dev/${disk} of=/dev/null bs=1048576 fills up my 64G of memory instantaneous.
And killing it with CTRL+c isn't freeing that memory up again... it's hogging my memory. No wonder the system starts to act weird after a few hours of that.

 

[oddnoise]

I was trying to use this script to do a burn in of my new 5x seagate iron 4tb drives, but i always get

!!ERROR!! dd: failed to open '/dev/sda': Permission denied
!!ERROR!! dd: failed to open '/dev/sdb': Permission denied
!!ERROR!! dd: failed to open '/dev/sdc': Permission denied
!!ERROR!! dd: failed to open '/dev/sdd': Permission denied
!!ERROR!! dd: failed to open '/dev/sde': Permission denied

I did chmod +x beforehand to make it executable, also tried with bash solnet-array(...), same result.

I tried the internal TrueNas shell as well as ssh via Putty, same result.

sudo bash gives me
solnet-array-test-v3_n.sh: line 200: /tmp/sat.sda.err: Permission denied

right at the start.

What am i doing wrong?
Im running TrueNAS-SCALE-22.12.1.

Thanks in advance!

 

[NugentS]

Question - do sda, sdb, sdc etc exist in your /dev folder?
Are you logged on as root or some other admin user?

 

[oddnoise]

They do exist in the /dev folder apparently.
I am logged in as another admin user, but i can try as root!

 

[oddnoise]

Okay, getting a new error as root:

Performing initial serial array read (baseline speeds)
Fri Feb 24 09:58:51 PST 2023
solnet-array-test-v3.sh: line 200: /tmp/sat.sda.err: Permission denied

 

[NugentS]

Clear out your tmp folder by rebooting first would be my guess then try again.

 

[oddnoise]

Amazing! Seems to work.
Thank you very much! :)

 

[NugentS]

I'll chalk that one up to an educated guess - but you are welcome

 

[Stromkompressor]

It seems that the script consumes almost every bit of system memory, which gets me and the TrueNAS GUI kind of sweating. UI elements start to appear slower than usual.
Bluefin with a couple of apps and a VM of 8G.

EDIT: it is also grabbing every bit of swap. This can't be intended, right?

I get the same behavior and had to reboot over SSH because the webinterface couldn't even load the login page. This is what I got before the reboot: https://hastebin.com/share/ahanojudiq.yaml
If I had to guess with my limited knowledge, I think the read cache somehow doesn't get emptied?

I also got these alerts while the script was running: https://hastebin.com/share/odakubiwib

My memory was in maximum use for cache and swap was also used:

 

[Davvo]

My memory was in maximum use for cache and swap was also used: View attachment 65776

SCALE shouldn't be able to use more than half of max RAM for caching. Did you edit something? Tubables, bootloaders?

Edit: that graph shows buffered memory hogging.

 

[Stromkompressor]

SCALE shouldn't be able to use more than half of max RAM for caching. Did you edit something? Tubables, bootloaders?

Edit: that graph shows buffered memory hogging.

Ah thanks, yes it's actually buffered memory. But this script (V3) should only read from the disk and write to /dev/null/. I have a very fresh install of TrueNAS and only set up a pool, nothing else (so no smb sharing or whatever) nothing should be writing. I'll test manual dd command to analyze this further.

 

[Stromkompressor]

So I just ran one instance of dd if=/dev/sda of=/dev/null and the buffered memory fills up in ~30 s, this is the end result:

Code:

admin@truenas[~]$ cat /proc/meminfo     
MemTotal:       16349976 kB
MemFree:          371328 kB
MemAvailable:   14490520 kB
Buffers:        14230364 kB
Cached:           101068 kB
SwapCached:           88 kB
Active:            72784 kB
Inactive:       15187680 kB
Active(anon):       2748 kB
Inactive(anon):   937176 kB
Active(file):      70036 kB
Inactive(file): 14250504 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2095036 kB
SwapFree:        2094768 kB
[...]

I guess this is okay. It also doesn't bring down the system (like slow web interface loading). When starting 6 instances all on /dev/sda it already slows down the system considerably. I know this might not be a real-world scenario but at least that's what this script is doing and testing. top output when running 6 dd: https://hastebin.com/share/osaqivajan.yaml

I would just leave it like it is and do the burn-in maybe by copying large files over SMB? Any other suggestions?