Shell Handling of special characters in variables

[Basil Hendroff]

Letcl is a Linux shell script that checks Let's Encrypt certificates issued for a domain. I've managed to modify it to run on FreeBSD and FreeBSD embedded systems like FreeNAS. The commands the script uses that are syntactically different between Linux and FreeBSD platforms are date and sed. For instance, consider the following Linux GNU sed command:

sed ':a;N;$!ba;s/\n/ /g' file

The FreeBSD equivalent is:

sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' file

Or, to change a date format from %b %d %H:%M:%S %Y %Z to %Y-%b-%d %H:%M %Z, the Linux date command will auto-detect the input format.

date -d "indate" +'%Y-%b-%d %H:%M %Z'

...while the input format has to be specified for the FreeBSD date command:

date -j -f '%b %d %H:%M:%S %Y %Z' "indate" +'%Y-%b-%d %H:%M %Z'[/CMD]

So, sprinkled throughout the code (the pull request is here if you'd like to take a look at it), I've got case statements that chose the appropriate command format based on $OSTYPE e.g.

case ${OSTYPE,,} in
  freebsd*)
    command using FreeBSD syntax
    ;;
  *)
    command using Linux syntax
    ;;
esac

The modified script works well (albeit slow). I then thought the command format is very similar for both platforms except for some switches. Could I pass those switches through variables? If I could, all the case constructs would disappear except for a single case construct early on in the code for setting up the switch variables, so, something like:

case ${OSTYPE,,} in
  freebsd*)
    sflag = "-e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g'"
    dflag = "-j -f '%b %d %H:%M:%S %Y %Z'"
    ;;
  *)
    sflag = "':a;N;$!ba;s/\n/ /g'"
    dflag = "-d"
    ;;
esac

Within the body of the code, the sed and date commands normalised for both Linux and FreeBSD platforms would look something like:

sed ${sflag} file
date ${dflag} "indate" +'%Y-%b-%d %H:%M %Z'

I've been researching and struggling for a couple of days to try to get this working and have not made any real headway. The problem arises because of the special characters in the flags being interpreted rather than being treated literally. I'm hoping the brains trust here can help me move forward as I'm stuck.

 

[ralphbsz]

Debugging this for you would take me hours, so I'm not volunteering to do that. But I'll give you a few pointers.

First: What you call a program is technically a shell script, which is interpreted by a shell. What shells are you using? On Linux, it is probably bash. Even when called as /bin/sh, bash has "bashisms", and does things differently. On FreeBSD, what shell are you using? Most likely /bin/sh, which is a very conservative/old-fashioned/simple/strict/... shell. Let me give you three pieces of advice, but they will be contradictory: (a) For the simple interpretations (parsing, syntax, semantics) of shell variables, bash versus sh should make very little difference, as the rules for that are ancient. (b) To debug your problems, use bash on both systems, simply install the bash package, and start your script with "#!/bin/bash" on Linux, and "#!/usr/local/bin/bash" on FreeBSD. (c) To debug your problems, stop using bash (which in typical GNU/Linux fashion is overly complex and not compatible), and switch to using a traditional shell on both. For example install pdksh on both, and use it for compatibility.

Second: Your basic problem is that you are using different versions of commands such as sed and date, and that on Linux you are using GNU-specific syntax. Just stop doing that. Instead of using Linux- or BSD-specific man pages, go to the Posix standard (which both sets of utilites 99% support), and use only options and parameters that are standardized. For concreteness: The standard is called IEEE 1003.2 a.k.a. POSIX.2, and it documents the API for command-line utilities such as sed and date (and many others). To figure out what the POSIX standard usage is, you can either search the web and read the actual standards documents (very tedious and long), or you can just use the FreeBSD man pages: they typically explain exactly how the commands are standard-conforming, and what part of the BSD usage are extensions to the standard.

Another version of this is: Just install all the Gnu version of utilities on FreeBSD, and use them; then you don't have to worry about the difference any longer.

Once you use the same shell, and use either the POSIX common subset or the same utilities, the whole need for all your if statements and variables goes away. But you may not like those options.

So failling that, third: You need to learn the rules for how a shell expands variables, and how quoting and escaping interacts with variable expansion. The rules are complicated, and I can never remember them. Read the man page for the shells involved several times, and perhaps even get a shell textbook and read it. Once you understand when the various quotes/escapes are stripped, it will become debuggable.

This is one of the reasons why complex programs should not be written in shell, but in higher-level programming languages (Perl, Python, ...), where variable naming, content and interpolation are clearly defined and separated.

 

[Alain De Vos]

You can install gsed and coreutils which contains gdate.

 

[Basil Hendroff]

ralphbsz And here I'm thinking I must be missing something straightforward! Thanks for the big picture. I was on the verge of losing my sanity.

Another version of this is: Just install all the Gnu version of utilities on FreeBSD, and use them; then you don't have to worry about the difference any longer.

You can install gsed and coreutils which contains gdate.

This is an option that I hadn't considered. It will definitely keep code changes to a minimum. Having a look at the code for Lectl, this is the only snippet of code that might need to be modified:

# macOS compatibility
if [ "$(uname -s)" = "Darwin" ]; then
    _date=gdate
    _sed=gsed
fi

I'll do some testing around your suggestion. If it works, I'll submit a pull request to the developer suggesting a modification to this snippet to accommodate FreeBSD.

I knew I could rely on the brains trust to clear things up for me. With the help I've received, I'm not only better informed, but I'll also end up with a much tidier solution as well. Thank you!