Week 09 — Expansions, conditionals

This week we will study while loops and if statements, several ways to test variable values and file properties, and some useful ways to manipulate the values stored in variables.

Evaluation

Up to 10 points can be gained towards your final score by completing the in-class assignment on Friday.

Preparation

1. Complete the self-preparation assignment at home before next class

This week's self-preparation assignment is mostly practice with some familiar shell commands and some new ones. The new commands are explained in the Notes section below, which also contains the practice exercises. (If you are already familiar with the command line, please at least skim the notes to make sure there are no small details that are new to you.) These commands and concepts will be further explored in the in-class assignment on Friday.

2. Check your understanding of the concepts using the self-assessment questionnaire

1. Answer each question in the self-assessment questionnaire as honestly as you can.
2. Revise the topics having the lowest scores, update your scores.
3. Repeat the previous step until you feel comfortable with most (or all) of the topics.

On Thursday evening, press 'submit'. In class on Friday we will check which topics were difficult for everyone.

To succeed at the in-class assignment for this class you should understand the topics outlined in the “Notes” section.

What you will learn from this class

• How to remove prefixes and suffixes when you fetch a variable's value.
• How to replace a pattern with some other text when you fetch a variable's value.
• How to replace part of a word with the output from a command.
• How to replace part of a word with the result of an arithmetic expression.
• How environment variables control your interaction with the command line.
• How to to write words (including filenames) that contain spaces.
• How to use while loops to repeat commands while a condition is true.
• How to use if statements to optionally run commands based on a condition.
• How to use the test command to test the properties of files, strings, and numbers.
• How to get help: using the man and help commands, and using the --help option with most commands.

Notes

The notes below include several exercises with answers that introduce new concepts. Many of these concepts will be used in this week's in-class assignment.

Read the notes and try to complete each of the exercises without looking at the sample answer. If you cannot complete an exercise using a few short commands then read the sample answer, practice it, and make sure you understand it before continuing.

Review

Make sure you understand the topics from last week. Click on the link below to expand a brief review.

Review

Exercises

As well as the indicated exercises, try typing in all the examples for yourself. If you can think of ways to modify the example to change the behaviour, try them. Exploration is the best way to learn.

Variables

Variables are used to store data. Variable names must begin with a letter which can be followed by any number of letters or digits. (The underscore “_” is treated as a letter.) Names that conform to these rules are legal (allowed by the rules); names that break these rules are illegal (not allowed by the rules).

Some examples of legal variable names:

Some examples of illegal variable names:

You create or set a variable using the = assignment operator. The syntax (general form) of assignment is:

where variableName follows the rules explained above and value is a single word (such as a filename), number, etc., with no spaces. There must not be any space either side of the = symbol.

You get the value of a variable by writing a $ before the variable's name. For example:

Again, there must be no space between the $ and the variable name.

Quoting

What if you want to put a space inside a value stored in a variable? You can protect spaces using quotation marks.

Single quotes around a value like this 'value' will protect everything inside the value. Wildcards (*, ?, etc.), dollar signs ($), and other special characters will be completely ignored. Spaces inside the value will be considered part of the value.

Double quotes around a value like this "value" will protect everything inside the value except for expansions (see below) introduced by the $ character. One such expansion is getting the value of a variable using $name.

If you want a value with spaces inside, use '…'. If you want a value with spaces inside and variables to be expanded, use "…".

Expansions

The $ character is used to transform variables and other values in the command line by a process called expansion. There are several kinds of expansion:

• variable expansion
• parameter expansion
• arithmetic expansion
• command substitution

Variable expansion

A $ followed by a variable name expands to the value stored in the variable. (If the variable is not set to any value then the result is blank.) Braces { and } around the variable name are optional but are necessary when a variable expansion is followed immediately by a letter or digit that is not part of the variable name, as in the last example below.

The brace syntax {variable} also provides several mechanisms that modify the value retrieved from variable. Within a variable expansion with braces, a suffix (such as a file extension) can be removed by following the variable name with %suffix.

A prefix can be removed by following the variable name with #prefix.

In both cases (${name%pattern} and ${name#pattern}) you can use wildcards such as ? in the prefix or suffix.

You can also replace a pattern anywhere in a value with some other text using /pattern/replacement after the variable name:

The variable expansions above should be all you need in most cases, but there are several more that you might need to use occasionally. If you are interested, here is a table showing most of them. (Click on the 'link' to toggle the table.)

String operators available during ''${...}'' expansion

Imagine that you are running out of disk space on your computer. You have a lot of 'lossless' music stored in .wav (Microsoft 'wave') files. You could halve the amount of space they use by converting them to .flac (free lossless audio codec) files. The program ffmpeg can do this for you. The syntax is:

First, make some 'fake' .wav files like this:

for i in {1..9}; do echo $i > track-$i.wav; done

1. Write a “for wav in …” loop that echos the names of all the *.wav files in the current directory, one at a time.

track-1.wav
track-2.wav
  ...
track-9.wav

2. Change the echo command so that for every file it prints two things: the original name ($wav) as well as the name with the original .wav suffix removed.

track-1.wav track-1
track-2.wav track-2
  ...
track-9.wav track-9

3. Change the echo command so that for every file it prints two things: the original name ($wav) as well as the name with the original .wav suffix removed and a new .flac suffix added.

track-1.wav track-1.flac
track-2.wav track-2.flac
  ...
track-9.wav track-9.flac

4. Change the echo command so that for every .wav file in the current directory your loop prints: ffmpeg -i filename.wav filename.flac

The output of your loop should look like this:

ffmpeg -i track-1.wav track-1.flac
ffmpeg -i track-2.wav track-2.flac
  ...
ffmpeg -i track-8.wav track-8.flac
ffmpeg -i track-9.wav track-9.flac

(If you had some genuine .wav files, and a copy of the ffmpeg program, you could remove the echo from your loop body and it really would convert them all the .wav files to .flac for you.)

Answer

for wav in *.wav; do
  echo ffmpeg -i $wav ${wav%.wav}.flac
done

Parameter expansion

Parameters are the values passed to a shell script on the command line. Whereas variables are named, parameters are numbered starting at 1. (If you ever happen to need it, $0 is the name of the shell script exactly as it appeared on the command line.)

There are three other special variables that are useful inside shell scripts. $# expands to the number of command-line arguments, and both $@ and $* expand to a sequence containing all of the command-line arguments separated by spaces.

Write a shell script that prints a single number showing how many command-line arguments it is run with. (Don't forget you have to make it executable using chmod +x filename before you can run it.)

Answer

#!/bin/sh
echo $#

The variables $@ and $* behave differently when quoted. To illustrate the difference, consider the following script:

#!/bin/sh

echo 'using "$@":'
for argument in "$@"; do
  echo "$argument"
done

echo 'using "$*":'
for argument in "$*"; do
  echo "$argument"
done

Running this script with three command-line arguments one, "two too", and three produces this result:

Create the script shown above. Run it with arguments one "two too" three. Run it with other arguments, including no arguments.

You can see that "$*" expands to a list of command-line arguments all inside one pair of double quotes ("). In other words, "$*" is one single value containing all of the command line arguments.

On the other hand, "$@" expands to a list of command-line arguments where each separate argument is inside a pair of double quotes ("). In other words, "$@" is one value per argument, each value containing a quoted version of the corresponding argument.

In a for loop you should almost always use "$@" (to repeat the loop for each argument).

for argument in "$@"; do some_operation_on "$argument"; done

When assigning to a variable you should probably always use "$*", however most shells are clever enough to let you use either.

all_arguments="$*"
all_arguments="$@"

Arithmetic substitution

You can evaluate arithmetic expressions by enclosing them in double parentheses preceded by a $ character: $((expression)) Within the expression you can use the normal arithmetic operators and the names of variables (without a $ in front of them).

Modify your shell script from the previous exercise so that it prints each command-line argument preceded with its number, starting at 1. For example:

Answer

#!/bin/sh
n=1
for arg in "$@"; do
  echo $n $arg
  n=$((n+1))
done

Write a shell script called factorial that calculates the factorial of its command line argument. Recall that factorial(n) = n * (n-1) * (n-2) * … * 1.

Answer

#!/bin/sh
num=$1
fac=1
while test $num -gt 0; do
  fac=$((fac*num))
  num=$((num-1))
done
echo $fac

Command substitution

Sometimes you will need to store the output of a command in a variable, or use the output of one command as an argument to another command. Command substitution provides a way to do this.

The pattern $(command) is replaced with the output from running command. Note that command can include command-line options and arguments, and can even be a pipeline made from several commands. The result can be used to set the value of a variable. In the following examples, note the use of double quotation marks around the command substitutions to protect any spaces in the output from the commands.

Another way of doing the same thing, without variables, is to use the command substitutions directly where their output is needed:

Write a shell script called nfiles.sh that prints the number of files in each of the directories written on the command line followed by the name of the directory.

(Of course, your results will differ.)

Answer

#!/bin/sh
for arg in "$@"; do
    echo $(ls "$arg" | wc -w) "$arg"
done

Control structures

A for loop is executed once for each member of a list of items. Other control structures include the while loop that executes until a condition becomes false, the until loop that executes until a condition becomes true, and the if statement that conditionally executes (or not) a sequence of commands.

While loop

The syntax (general form) of a while loop is

or on a single line like this:

The COMMANDS part works exactly like it does in a for loop. The TEST part should be a command that can either succeed or fail. The while loop will continue to run its TEST and the COMMANDS until the TEST fails.

The ''test'' command

A useful command to use for the TEST part of a while loop is test, which can do many things. One thing test can do is compare two numbers.

Combining a while loop with test and arithmetic expansion to update a counter:

If statement

The if statement conditionally executes a sequence of commands. The syntax of if statements is:

or on a single line like this:

The COMMANDS will be run only if the TEST succeeds. Using the test command again for the TEST:

Another form of the if statement provides a second set of commands to be run if the TEST fails.

or on a single line like this:

First the TEST command is run. If TEST succeeds then COMMANDS1 are run. If TEST fails then COMMANDS2 are run.

The test command can also check the properties of a file or directory, the size of a string, or the relationship between two strings.

Combining the test for a directory with the if statement:

Modify your nfiles.sh script so that it checks each command-line argument. If the argument is a directory, the script prints the number of files in the directory followed by the argument (as before). If the argument is not a directory, the script prints '?' and then the argument.

Hint: instead of using two echo commands, set a variable (e.g., n) to either the number of files in the directory or the value '?'. At the end of your loop use a single echo command to print n and then the argument.

Answer

#!/bin/sh
for arg in "$@"; do
    n=0
    if test -d "$arg"; then
        n=$(ls "$arg" | wc -w)
    else
        n='?'
    fi
    echo $n "$arg"
done

Modify your nfiles.sh script so that it checks each command-line argument. If the argument is a directory, the script prints the number of files in the directory followed by the argument (as before). If the argument is a regular file, the script prints 'F' and then the argument. If the argument is neither a directory nor a file (e.g., it does not exist) then the script prints '?' followed by the argument.

Hint: the commands in the else part of your if statement should include another if statement that tests whether the non-directory argument is a regular file (test -f). This second if selects between 'F' for a file or '?' for everything else.

Answer

#!/bin/sh
for arg in "$@"; do
    n=0
    if test -d "$arg"; then
        n=$(ls "$arg" | wc -w)
    else
        if test -f "$arg"; then
            n='F'
        else
            n='?'
        fi
    fi
    echo $n "$arg"
done

The meanings of the above test forms can be inverted by placing a ! (“not”) in front of them.

Combining if with the test for a directory (-d) and inverting it (!) to mean “the directory does not exist”:

You can combine two or more test forms with logical “and” or logical “or”:

To check if your log file exists as a regular file (-f) and (-a) is writable (-w):

Shorthand for ''test''

Many shells have an alternative version of test called [ (open square bracket). Instead of test expression you can write [ expression ] which looks quite a lot nicer. Note that you must put spaces on both sides of the opening “[” and another before the final “]”.

Modify your nfiles.sh script so that it prints the fist item on each line (the number of files, or an 'F' or a '?') right-justified in a field 5 characters wide. Use spaces to pad the number (or 'F' or '?') on the left to the required width.

Answer

#!/bin/sh
for arg in "$@"; do
    n=0
    if [ -d "$arg" ]; then
        n=$(ls "$arg" | wc -w)
    else
        if [ -f "$arg" ]; then
            n='F'
        else
            n='?'
        fi
    fi
    while [ ${#n} -lt 5 ]; do n=" $n"; done
    echo "$n" "$arg"
done

Other commands as loop/conditional tests

Many commands can be used as the test or condition in a loop or if statement. For example, grep succeeds if it finds a match and fails if it cannot find a match.

(-q tells grep not to print any output, and -s tells grep not to complain about missing files.)

See Finding information about commands and programs below for different ways to look for information about success/failure of commands and their other options that help when using them as tests in loops and if statements.

Infinite loops

Two built-in commands help with infinite loops.

The following while loop will never stop. (If you try it then to make it stop type Control+C.)

The following while loop will stop immediately and never execute the echo command.

One use of true and false is to set a flag in a shell script to affect an if statement later on.

Stopping or restarting loops: ''break'', and ''continue''

You can break out of a while or for loop using the break command. You can jump back to the test at the start of a while loop using the continue command. Inside a for loop, the continue command restarts the loop body with the loop variable set to the next item in the list of items.

Modify your nfiles.sh script so that it uses a flag to remember whether any arguments were non-directories. If at least one argument was not a directory (it was a regular file, or did not exist) then print a message at the end of the script saying: Warning: non-directories were encountered.

Answer

#!/bin/sh

nonDirs=false

for arg in "$@"; do
    n=0
    if [ -d "$arg" ]; then
        n=$(ls "$arg" | wc -w)
    else
        nonDirs=true
        if [ -f "$arg" ]; then
            n='F'
        else
            n='?'
        fi
    fi
    while [ ${#n} -lt 5 ]; do n=" $n"; done
    echo "$n" "$arg"
done

if $nonDirs; then
    echo Warning: non-directories were encountered
fi

Stopping a script or shell: ''exit''

You can terminate a shell script (or your interactive shell session) using exit.

if test ! -d data; then
  echo "data directory does not exist: giving up"
  exit 1
fi

The argument to exit is optional and should be a number. 0 is success and non-zero is failure. This allows scripts to control loops and conditionals, as part of their TEST, by returning success or failure from the entire script.

Write a short script called exit0.sh that immediately uses exit 0 to terminate its own execution.

Answer

#!/bin/sh
exit 0

Write another short script called exit1.sh that immediately uses exit 1 to terminate its own execution.

Answer

#!/bin/sh
exit 1

Use an if statement to verify which script 'succeeds' and which script 'fails'.

Which exit value represents 'success'?

Which exit value represents 'failure'?

Answer

Modify nfiles.sh so that it succeeds if all arguments were directories and fails if any arguments were non-directories. Test whether it works using an if statement on the command line.

Answer

if $nonDirs; then
    echo Warning: non-directories were encountered
    exit 1
fi

Command and filename completion

You can save a lot of time by typing the first few characters of a filename and then pressing the Tab key. The shell will try to find a file matching what you typed, and then 'complete' the part of the filename that you did not type. If there is more than one matching file, the shell will complete up to the point where the file names diverge. If there is only one matching file, the shell will complete the entire filename and than add a space at the end.

Finding information about commands and programs

Programs such as test (and many others) have a large number of command line options. Don't bother trying to memorise more than two or three of the most useful options. Instead, know where to look up information when you need it. There are several ways to find information about a command, depending on the kind of command it is.

Use ''help'' to learn about built-in commands

(Note: MobaXterm has its own non-standard help command that does not work as shown below.)

Using help you can find information about the syntax of loops and conditionals, the options understood by echo and other commands, and even obtain a list of all the builtin commands by typing help with no arguments.

Notice the last section, “Exit Status”. This tells you when the command will 'succeed' and when it will 'fail'. You can use the command as a TEST in a loop or if statement to check its “exit status” and therefore to test for whatever situation affects that status, according to the description of the command.

Use ''man'' to read the manual page for most programs

Commands that are not builtin to the shell usually have a manual page. Use man command to read the manual page describing command. Use man -k keyword to see a list of manual pages related to the given keyword. (Note that the version of man used by MobaXterm does not provide the -k keyword option.)

Note that the manual page for a command that can 'succeed' or 'fail' (and which is therefore useful in loop and if statement tests) will almost always include an “Exit Status” section describing what situations you can test for using the command.

Asking programs for help

Many programs respond to the option -h or -help or --help by printing brief instructions about how to use that program.

Commands that are useful as TESTs will generally tell you about their “exit status” too. For example, on my computer, the output from grep --help includes the following two lines:

Exit status is 0 if any line is selected, 1 otherwise;
if any error occurs and -q is not given, the exit status is 2.

Summary

• A variable name begins with a letter, followed by zero or more letters or digits.
• The underscore _ is a letter.
• Assign a value to a variables using name=value.
• To protect spaces inside a word or value, use 'single quotes' or "double quotes".
• Inside 'single quotes' neither “$” expansions nor wildcards work.
• Inside “double quotes” variables and other “$” expansions are performed, but wildcards are ignored.
• Variable expansion: get a variable's value using $name or ${name}.
• Remove prefixes with ${name#pattern} and suffixes with ${name%pattern}.
• In a script, access command-line arguments using $1 for the first argument, $2 for the second, and so on.
• In a script, use “$@” to obtain a list of all the arguments (with spaces inside individual arguments preserved).
• Arithmetic substitution: $((expression)) expands to the result of evaluating the given arithmetic expression.
• Command substitution: $(command) expands to the result of running command (which can be a pipeline).
• A while loop performs some commands until a 'test' program or command fails.
• The if statement conditionally runs some commands based on the success of a 'test' or 'condition' command.
• The test command implements many kinds of tests and then succeeds or fails in a way useful for while and if.
• [ expression ] is shorthand for test expression.
• Many commands can be used as the 'test' or 'condition' in an if statement or loop.
• In a loop, break exits from the loop immediately and continue restarts the loop immediately.
• In a script, exit terminates the entire script immediately. (Use it to terminate your shell, too.)
• Press Tab to complete a filename. Press it again to see a list of possible completions.
• Use help command, man program, and program --help to learn about commands and programs.