Shell script - strings including quotes?

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
Sometimes, the things you think are simple, aren't that simple...

I'm trying to update my script for installing Nextcloud in a jail to use Caddy rather than Apache. In order to use DNS validation for the certs, I need to do something like this in the jail:
Code:
sysrc caddy_env="CLOUDFLARE_EMAIL=me@domain.com CLOUDFLARE_API_KEY=my_api_key"

...and to make that configurable, I'm putting this in the nextcloud-config file:
Code:
DNS_ENV='"CLOUDFLARE_EMAIL=foo@bar.baz CLOUDFLARE_API_KEY=blah"'

...with this later in the script:
Code:
iocage exec ${JAIL_NAME} sysrc caddy_env=${DNS_ENV}

Unfortunately, it doesn't seem to be working quite properly. If I add an echo $DNS_ENV to the script, I get output like this:
Code:
DNS_ENV is
"CLOUDFLARE_EMAIL=foo@bar.baz CLOUDFLARE_API_KEY=blah"
[: "CLOUDFLARE_EMAIL=foo@bar.baz: unexpected operator

...and when it gets to the part of the script that runs the sysrc command, I see this:
Code:
caddy_enable:  -> YES

caddy_cert_email:  -> foo@bar.baz

caddy_env:  -> 
/usr/sbin/sysrc: cannot create : No such file or directory

I'm kind of stumped here--any suggestions?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Try this.

Code:
DNS_ENV="CLOUDFLARE_EMAIL=foo@bar.baz CLOUDFLARE_API_KEY=blah"
iocage exec ${JAIL_NAME} sysrc caddy_env="${DNS_ENV}"


More likely to work but I didn't try it. Shell quoting is frustrating at times. Do not try to be overly clever. Try very hard to put variable expansions in double quotes even when you do not see the point.

i.e.
"${DNS_ENV}"
is guaranteed to parse out as a single argv[] element whereas
${DNS_ENV}
will not if there is a member of $IFS in there.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
Interesting. I'm still getting the [: CLOUDFLARE_EMAIL=blah: unexpected operator message, but the sysrc command is working, the jail is running, and Caddy is able to get a cert using the provided credentials. The warning message is curious, but it doesn't seem to be hurting things.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well yes you have a crapton of test commands in that script that use unquoted variable expansions.

if [ -z ${foo} ];

is bad syntax. You should do your tests as

if [ -z "${foo}" ];

which is less likely to generate a superfluous complaint due to expansion of a variable into something containing an IFS.

I am sorry that I am too lazy to debug your script but I am happy to criticize your style. :tongue: ;-)
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
I am happy to criticize your style.
You're criticizing in a way I can do something about, so that's a plus in my book. And what's the difference between $foo and ${foo}, anyway?
 

droeders

Contributor
Joined
Mar 21, 2016
Messages
179
And what's the difference between $foo and ${foo}, anyway?

I haven't looked at your original question, but I can answer this one. There's nothing functionally different between $foo and ${foo}, but it can make a difference if you're trying to use the variable in a string.

Run this example in a shell to illustrate it much quicker than I could explain in words:

Code:
foo="Testing"
echo "$foo_1_2_3"
echo "${foo}_1_2_3"


Basically, the first echo tries to reference variable $foo_1_2_3, but the second echo does what you want.
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
As someone who has written quite a bit of shell script including stuff meant to be run by any sh implementation, not just bash I cannot recommend shellcheck.net highly enough.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
You're criticizing in a way I can do something about, so that's a plus in my book. And what's the difference between $foo and ${foo}, anyway?

It's a readability and style thing in addition to what @droeders explained. Using doublequotes where possible when expanding variables is also a style choice, but has functional impact when there is whitespace or other IFS in there.

Code:
((rm "${portunstarted}" && ${chroot} ${basedir} sh -c "cd /usr/ports/${port} && make ${jflag} && make install" && rm "${portincomplete}" ) 2>&1 | tee "${portlogfile}" ) > ${portlogdev}


I submit that this is much easier to read when the variables are ${}.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
As someone who has written quite a bit of shell script including stuff meant to be run by any sh implementation, not just bash I cannot recommend shellcheck.net highly enough.

I applaud anyone who writes sh code and realizes that bash != sh.

Have you seen Seebach's Portable Shell Scripting?

shellcheck.net seems to kind of suck as it suggests $(...) instead of backticks/backquotes and doesn't seem to have an obvious way to disable that. $() makes for a lot of unreadability when you are already doing stuff with craptons of variables and a subshell or two inside the backticks. But that's a style choice. :smile:
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
Have you seen Seebach's Portable Shell Scripting?

shellcheck.net seems to kind of suck as it suggests $(...) instead of backticks/backquotes and doesn't seem to have an obvious way to disable that. $() makes for a lot of unreadability when you are already doing stuff with craptons of variables and a subshell or two inside the backticks. But that's a style choice.

I have not seen Seebach's Portable Shell Scripting; mostly I learned the hard way by writing Autorevision and reading the spec. ☺️

I seem to recall from when I looked into it that the $(...) and backticks/backquotes methods of doing things had functional differences that led me to prefer the use of $(...) where ever I used it, which is also likely why shellcheck.net recommends them.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I have not seen Seebach's Portable Shell Scripting; mostly I learned the hard way by writing Autorevision and reading the spec. ☺️

I seem to recall from when I looked into it that the $(...) and backticks/backquotes methods of doing things had functional differences that led me to prefer the use of $(...) where ever I used it, which is also likely why shellcheck.net recommends them.

Yeah, $(...) allows you to do certain things that backticks do not (or do not easily). However, the construct is also incredibly similar-looking to ${...} or (...) and when you have a complex command pipeline going on that you want to evaluate the output from,

for i in `(cat ${foo} | awk -F: '{print $1}'; echo ${bar}) | sort | uniq`; do

gives better visual context than

for i in $((cat ${foo} | awk -F: '{print $1}'; echo ${bar}) | sort | uniq); do

and I have a tendency to generate even more ()'y command pipelines with more arguments. The use of ` is nice because it doesn't have multiple meanings, and you can quickly see where the start and end of things are. Once you get to needing multiple levels of command substitution, you might be better off breaking it down into several steps anyways, so I'm not super-convinced that $(foo $(bar)) is all that useful in real world situations.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
Somewhat related to this question... My script downloads Caddy by way of a Caddy-supplied script. The download page gives a command of curl https://getcaddy.com | bash <options>. I'm using a variable of $DL_FLAGS to store those options; its contents default to "-s personal", and to that is added the plugin (if any) the user chooses for DNS validation. I implement this as:
Code:
fetch -o /tmp https://getcaddy.com
if ! iocage exec "${JAIL_NAME}" bash ${DL_FLAGS} < /tmp/getcaddy.com
then
    echo "Failed to download/install Caddy"
    exit 1
fi

...and it works just fine. But if I add quotes around the $DL_FLAGS variable like this:
Code:
fetch -o /tmp https://getcaddy.com
if ! iocage exec "${JAIL_NAME}" bash "${DL_FLAGS}" < /tmp/getcaddy.com
then
    echo "Failed to download/install Caddy"
    exit 1
fi

with $DL_FLAGS set to "-s personal tls.dns.cloudflare", it fails with this result:
Code:
/tmp/getcaddy.com                             100% of 7380  B   11 MBps 00m00s
jexec: dflare: no such user

The script is not run, Caddy is not downloaded, and somewhat disturbingly the script does not stop. The obvious solution is simply to not use quotes there, but I'm a little puzzled at this behavior.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well that's because the usage there intends for ${DL_FLAGS} to be expanded into multiple arguments. For that, yes, you omit the quotes. You're expecting that behaviour. Hopefully it's safe.

Consider this. You have some shell script that takes as an argument something that you want to process in a temp directory. So you make it, based on ${1}.

Code:
#! /bin/sh

DIR="/tmp/${1}"

mkdir ${DIR}

....yourstuff here...

if [ -d ${DIR} ]; then
        rm -fr ${DIR}
fi


That seems kinda reasonable, right? Make a temporary directory and clean up afterwards.

So consider:

# ./myscript "foo bar"

and suddenly you have executed

mkdir /tmp/foo bar

-i.e. the ability to make unintended directories.

Or consider:

# ./myscript "bar -o -d baz"

The ability to remove arbitrary directories... imagine if you substituted / for baz...

There's endless discussion of the hazards of unquoted shell variables out there on the 'net. Some will doubtless be much more informative than mine.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
You're expecting that behaviour.
Correct.
Hopefully it's safe.
I think it is under the circumstances. I guess I could re-work it to use a separate variable for the plugin to download, and have that variable default to "" (because if the user is using HTTP validation, they wouldn't be using a Caddy plugin at all).
 
Top