Github repository for FreeNAS scripts, including disk burnin and rsync support

[svtkobra7]

Must be missing something ...

Complete:

• Created keys using PuTTYgen
  
• Public Key - Edited FreeNAS root user (copied "Public key for pasting into OpenSSH authorized_keys file" to "SSH Public Key" field)
  
• Public Key - Edited on ESXi HOST: /etc/ssh/keys-root/authorized_keys added same "Public key for pasting ..."
  
• Configured PuTTY saved session to use private key to SSH to FreeNAS VM = WORKS
• Configured PuTTY saved session to use private key to SSH to ESXi Host = WORKS

Incomplete:

• via SSH Console to FreeNAS = NOPE (see below)
• Do I need to copy the private key to the FreeNAS VM so it uses that to authenticate SSH to ESXi Host?
• I changed * something * on the ESXI Host sshd_config and ssh root@ESXiHost worked, but it prompted me for a password too. Not what we want. Should have made a copy of sshd_config as I've changed it so many times, I'm not sure what I changed.

root@FreeNAS:~ # ssh root@ESXiHost
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
BLAHBLAH
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /root/.ssh/known_hosts:1
RSA host key for ESXiHost.localdomain has changed and you have requested strict checking.
Host key verification failed.

 

[Ericloewe]

Do I need to copy the private key to the FreeNAS VM so it uses that to authenticate SSH to ESXi Host?

Yes, of course. Whatever is initiating the SSH session has to know where and how to grab the private key with which it can authenticate with the server on the other side of the SSH session.

 

[svtkobra7]

Yes, of course. Whatever is initiating the SSH session has to know where and how to grab the private key with which it can authenticate with the server on the other side of the SSH session.

Can you offer a resource that details how to do this? I'm struggling to configure properly (I can generate the key to login to Freenas VM just fine, but can't make the jump to allow FreeNAS to login to the ESXi host by way of @Spearfoot 's script or testing via ssh root@ESXiHost when connected to the FreeNAS VM via SSH).

Initially (prior to my last message), I thought the solution may lie in editing sshd_config on my ESXi host, which I borked and for lack of knowing how to correct that, I had to reset the ESXi host entirely). I actually did that more than once in an attempt to be self sufficient, but clearly I was headed down the wrong road.

(If I'm able to put the pieces together, I'm happy to document the process should it be helpful to others - I haven't found an end to end documentation on the forum and my Google skills haven't either)

Thanks in advance for any help you can offer.

 

[Spearfoot]

Can you offer a resource that details how to do this? I'm struggling to configure properly (I can generate the key to login to Freenas VM just fine, but can't make the jump to allow FreeNAS to login to the ESXi host by way of @Spearfoot 's script or testing via ssh root@ESXiHost when connected to the FreeNAS VM via SSH).

Initially (prior to my last message), I thought the solution may lie in editing sshd_config on my ESXi host, which I borked and for lack of knowing how to correct that, I had to reset the ESXi host entirely). I actually did that more than once in an attempt to be self sufficient, but clearly I was headed down the wrong road.

(If I'm able to put the pieces together, I'm happy to document the process should it be helpful to others - I haven't found an end to end documentation on the forum and my Google skills haven't either)

Thanks in advance for any help you can offer.

Compared to most UNIX-based environments, configuring SSH keys is a little different on ESXi - the authorized_keys file is in the /etc/ssh/keys-root directory for user root, as specified by the AuthorizedKeysFile directive (/etc/ssh/keys-%u) in the ssh configuration file. Add the public key you've generated on your FreeNAS system to this file and you should be able to start an SSH session from the FreeNAS server to the ESXi server without having to enter a password. The contents of /etc/ssh/keys-root/authorized_keys will look something like this:

Code:

ssh-rsa AAAAB3NzaC1yc2E ...encrypted gibberish snipped... Il068N12j706Q== root@yourfreenasservername

You will also need to enable the SSH service, and make sure it starts with the host.

 

[svtkobra7]

Compared to most UNIX-based environments, configuring SSH keys is a little different on ESXi - the authorized_keys file is in the /etc/ssh/keys-root directory for user root, as specified by the AuthorizedKeysFile directive (/etc/ssh/keys-%u) in the ssh configuration file. Add the public key you've generated on your FreeNAS system to this file and you should be able to start an SSH session from the FreeNAS server to the ESXi server without having to enter a password. The contents of /etc/ssh/keys-root/authorized_keys will look something like this:

Code:

ssh-rsa AAAAB3NzaC1yc2E ...encrypted gibberish snipped... Il068N12j706Q== root@yourfreenasservername

You will also need to enable the SSH service, and make sure it starts with the host.

• Thanks for the additional info!
• I think the issue is that I generated keys using PuTTY Key Gen and while that works great to log into both FreeNAS and ESXi, I'm getting that error because FreeNAS didn't create the key, and doesn't have a private key to authenticate against when attempting root@FreeNAS:~ # ssh root@ESXiHost?
• So the solve here would be to generate keys in FreeNAS (and then copy that public key to /etc/ssh/keys-root/authorized_keys on the ESXi Host)?

 

[Spearfoot]

So the solve here would be to generate keys in FreeNAS (and then copy that public key to /etc/ssh/keys-root/authorized_keys on the ESXi Host)?

Yes, sir! You need to provide the ESXi server a key generated on the machine from which you want to access it -- in this case, your FreeNAS server.

 

[rosabox]

I have a problem with the zpool_report.sh script.
After upgrading to FN 11.1, I get this error:

Code:

Failed conversion of ``13-on-Wed_Dec'' using format ``%Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
			[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
Failed conversion of ``2-on-Sat_Dec'' using format ``%Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
			[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]

Any ideas how to fix it?

Thank you.

 

[Spearfoot]

I have a problem with the zpool_report.sh script.
After upgrading to FN 11.1, I get this error:

Code:

Failed conversion of ``13-on-Wed_Dec'' using format ``%Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
			[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
Failed conversion of ``2-on-Sat_Dec'' using format ``%Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
			[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]

Any ideas how to fix it?

Thank you.

My guess is that the output of the zpool command has changed slightly and has broken this line of code:

Code:

scrubDate="$(zpool status "$pool" | grep "scan" | awk '{print $15"-"$12"-"$13"_"$14}')"

I haven't installed 11.1 yet, so I can't troubleshoot further at the moment, but this is where I suggest you start looking. Just run zpool status {pool} on your pool and examine the line containing "scan": it may be that we need to use the 16th, 13th, 14th, and 15th tokens from that line, instead of the current 15th, 12th, 13th, and 14th.

Good luck!

 

[danb35]

zpool status under 11.0-U4:

Code:

root@freenas2:~ # zpool status ssdpool
  pool: ssdpool
 state: ONLINE
  scan: scrub repaired 0 in 0h8m with 0 errors on Sun Dec 10 00:08:50 2017
config:

   NAME										  STATE	 READ WRITE CKSUM
   ssdpool									   ONLINE	   0	 0	 0
	 gptid/4573aa37-0c81-11e6-b877-002590caf340  ONLINE	   0	 0	 0

errors: No known data errors
root@freenas2:~ #

Under 11.1:

Code:

root@freenas11:~ # zpool status tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:38:35 with 0 errors on Sun Nov 19 01:43:37 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   tank											ONLINE	   0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/86c6263c-bc1d-11e7-9fae-00a0d1ec3348  ONLINE	   0	 0	 0
	   gptid/ffd26c25-446a-11e7-a5c6-00a0d1ec3348  ONLINE	   0	 0	 0

errors: No known data errors
root@freenas11:~ #

The output in 11.1 adds "0 days".

 

[Ericloewe]

Ooh, and seconds. That's great for... Well, not much, really, but I never liked seeing "scrub finished in 0h0m".

 

[Chris Moore]

My guess is that the output of the zpool command has changed slightly and has broken this line of code:

Code:

scrubDate="$(zpool status "$pool" | grep "scan" | awk '{print $15"-"$12"-"$13"_"$14}')"

I haven't installed 11.1 yet, so I can't troubleshoot further at the moment, but this is where I suggest you start looking. Just run zpool status {pool} on your pool and examine the line containing "scan": it may be that we need to use the 16th, 13th, 14th, and 15th tokens from that line, instead of the current 15th, 12th, 13th, and 14th.

Good luck!

These are the two lines that need to be fixed:

Code:

#	scrubErrors="$(zpool status "$pool" | grep "scan" | awk '{print $8}')"
	scrubErrors="$(zpool status "$pool" | grep "scan" | awk '{print $10}')"

#	scrubDate="$(zpool status "$pool" | grep "scan" | awk '{print $15"-"$12"-"$13"_"$14}')"
	scrubDate="$(zpool status "$pool" | grep "scan" | awk '{print $17"-"$14"-"$15"_"$16}')"

I was looking at the source code without considering that the zpool status had changed. Once I was directed to your post, it was an easy fix.

Thanks

 

[danb35]

Now, is it possible to make the script smart enough to auto-detect 11.0 vs. 11.1, and behave accordingly?

 

[Ericloewe]

There's a command somewhere that dumps out a lot of stuff. Maybe it's easier and more general to get the FreeBSD version. uname -r, perhaps?

 

[Chris Moore]

There's a command somewhere that dumps out a lot of stuff. Maybe it's easier and more general to get the FreeBSD version. uname -r, perhaps?

That could work, but I don't want to write something that is specific to only a couple versions.
uname -r gives the version of FreeNAS on 11.1 it = '11.1-STABLE'
uname -r gives the version of FreeNAS on 11.0 it = '11.0-STABLE'
An if statement would be easy but I want to find a way to make it more flexible.

 

[Ericloewe]

That could work, but I don't want to write something that is specific to only a couple versions.
uname -r gives the version of FreeNAS on 11.1 it = '11.1-STABLE'
uname -r gives the version of FreeNAS on 11.0 it = '11.0-STABLE'
An if statement would be easy but I want to find a way to make it more flexible.

I don't know enough sh to help out with the details, but parsing the string into two separate numbers you can compare against a reference (or, more generically, use to look up a table to use to look up the format) seems like the way to go.

 

[Chris Moore]

Now, is it possible to make the script smart enough to auto-detect 11.0 vs. 11.1, and behave accordingly?

Yes, here is the solution. If I recall correctly, the original line numbers were 82 and 83, where the variables scrubErrors and scrubDate are set, need to be replaced with the following code:

Code:

temp1="$(uname -r | sed 's/[^0-9]*//g')"
if [ $temp1  -gt  110 ]; then
	scrubErrors="$(zpool status "$pool" | grep "scan" | awk '{print $10}')"
	scrubDate="$(zpool status "$pool" | grep "scan" | awk '{print $17"-"$14"-"$15"_"$16}')"
else
	scrubErrors="$(zpool status "$pool" | grep "scan" | awk '{print $8}')"
	scrubDate="$(zpool status "$pool" | grep "scan" | awk '{print $15"-"$12"-"$13"_"$14}')"
fi

This determines the version of FreeNAS and executes the correct awk to get the numbers that are needed to fill in the table.

 

[rosabox]

I've encountered another problem, this time with the get_hdd_temp.sh script.
Only drive detection method 3 works, with methods 1 and 2 I get "-n/a-"

Edit: In the smart_report.sh script all 3 methods work.

 

[Spearfoot]

Spearfoot updated Github repository for FreeNAS scripts, including disk burnin with a new update entry:

Modified zpool_status.sh script to support FreeNAS 11.1

zpool output changed from FreeNAS version 11.0 to 11.1, breaking our parsing of the scrubErrors and scrubDate variables in the zpool_status.sh script. Added a conditional to check the FreeNAS version and parse these values accordingly.

We obtain the FreeBSD version using uname, as suggested by user Chris Moore here on the FreeNAS forum.

uname -K gives 7-digit OS release and version, e.g.:

Code:

FreeBSD 11.0 --> 1100512...

Read the rest of this update entry...

 

[rosabox]

I figured it out, the output of method 1 or 2 is "ada0 ada1 ..." of method 3 "/dev/ada0 /dev/ada1 ...".
I've replaced "$drive" with /dev/"$drive" in the script and borrowed method 3 from the smart_report.sh to make it work.
My scripting skills are near zero but I noticed in the smart_report.sh script on the line 48 this "get_smart_drives drives", I think it's an error?
Maybe on line 47, there should be "drives=$(get_smart_drives)"?

 

[VintageGold]

Using FreeNAS 11.1

I downloaded and tried the disk-burnin.sh script and am running into an error. When I run the script, I get: Syntax error: newline unexpected (expecting word)

Here are my steps. I based my commands on the ones jgreco included here for his HDD burn-in script: https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

Code:

cd /tmp
fetch https://github.com/Spearfoot/disk-burnin-and-testing/blob/master/disk-burnin.sh
chmod +x disk-burnin.sh
./disk-burnin.sh

Is this because I'm using 11.1 and the code needs an update?