Command line interface

Engineers often use command line tools to help them manage files and directories, analyse data, or develop software tools. A popular command line interface available on all major operating systems is the shell. Examples include bash, which is used in the explanations below.

Shells

The command line interface is implemented by a program called a shell. The appearance and behaviour of the command line interface will differ slightly depending on which shell you are using. You can find out what shell you are using by the command:

echo $SHELL

The default shell on Linux and Windows (MobaXterm or WSL) is bash. The default shell on MacOS recently changed from bash to zsh, but they are very similar to each other. (If you want to temporarily switch from zsh to bash in MacOS, just type bash and press Enter.)

Why ‘shell’?

Why ‘bash’?

Basic command line (shell) usage

The shell prints a prompt to let you know it is waiting for your next command. Prompts are personal and configurable, so they will look different depending on the default configuration used by your installation.

As you are typing the command you can use Backspace to correct errors. If you want to discard the entire command, type Ctrl-u (hold down Control while typing u). When you have finished typing the command do not forget to press Enter so that the shell knows you have finished typing.

In the rest of this document fixed-width text indicates the name of something (usually a file or directory) or output that the computer prints, fixed-width font in a box indicates something you would type, and:

$ a large box with several lines indicates a dialogue with the shell, with your input $ in bold and the computer's output in non-bold

Note that the Enter pressed at the end of every command is not shown explicitly.

Exercise

Interrupting programs

To interrupt (terminate) a running program, hold down the Control key while typing C. (This is usually written Ctrl+C.)

Exercise

Directories

File and directory paths use the forward slash '/' to separate directory names in a path. '/' (the directory with no name) is the 'root' directory, the top-most directory in the filesystem.

Every computer user has a 'home' directory where the files belonging to that user are stored. If your account name is fred then on Mac and Windows your home directory is /Users/fred and on Linux your home directory is /home/fred.

Excercises

How the shell interprets what you type

Most lines you type at the prompt have the same form:

prompt$ command arg1 arg2 … argN

The first thing on the line is a command. It tells the shell what you want to do. A few of the most popular commands are built-in to the shell and interpreted by it directly. Most of the commands are programs located in standard places, such as /bin and /usr/bin.

Following the command are zero or more arguments. These convey additional information to the command, such as the name of files/directories or options (starting with a dash `-') that modify how the command behaves.

The first thing the shell does when you press Enter is to split the line you typed into command and arguments. It does this by looking for white space (space and tab characters). After they have done their job of separating command and arguments, the white space characters are discarded. This is one reason why it is a bad idea to put spaces in file names. If you really want to put spaces in file names or other arguments, we will see ways of quoting words.

A surprisingly useful built-in command is echo which prints the arguments (if any) that you pass to it, followed by a newline character. In addition to printing messages (and blank lines) echo is useful because it lets you see exactly how the shell is manipulating what you type before running a command.

Exercise

Moving around the file system

Every running program, including the shell, has a current working directory. In the shell this is where you appear 'to be' within the filesystem. When you start the shell, the current working directory is set to your home directory.

If you use a file or directory name that does not begin with / then the search for that file/directory begins in the current working directory (instead of at the root of the filesystem). These are called relative paths because the file/directory they refer to changes as you move your working directory around in the filesystem. File/directory names that begin with / are called absolute paths.

The pwd command prints the current working directory. The cd command changes the current working directory.

Excercises

Listing directory contents

The ls command lists the details of files and directories (including their contents). By default it lists the details (and contents) of the current directory.

For example:

For regular files, execute permission means they can be run as a program. For directories, execute permission means they can be searched (listed by ls, etc.). Each permission has its own dedicated position in the mode display, and is either a letter (r/w/x) indicating ‘allowed’ or a dash (-) indicating ‘denied’. Here is a summary of the modes:

Other options understood by ls include:

To understand the -d option, try ls -l . and ls -ld ..

Exercises

Changing file permissions and attributes

Every file 'belongs' to exactly one user. This user is the 'owner' of the file. Every file also 'belongs' to exactly one group. A group is a collection of users, and each user can belong to several groups. The idea is that a team of users working together can create a group and then share access to files and directories among members of the group. Access to files is therefore controlled for three sets of people: the user (owner) of the file, the members of the group to which the file belongs, and others (everyone else).

The chmod command changes the access mode of a file. The access mode of a file is a collection of nine permissions that control reading, writing, and executing the file. Each of these activities can be controlled independently for the file's user (its owner), members of the file's group, and others (anyone who is neither the owner of the file nor a member of the same group as the file).

To specify how to modify the permissions say who the change applies to (u, g, o, or a which means 'all of them'), then what kind of change (+ to add, - to remove, or = to set), and then which activities are being modified (r, w, x, or a meaning 'all of them').

The option -R applies the specified mode recursively to a directory and everything below it in the hierarchy.

You must be the owner of the file/directory, or be logged in as the user root (the 'superuser'), before you can do any of these things.

Exercises

Moving, renaming, and copying files

The following commands copy files, move files, remove files, make directories, and remove directories.

Exercises

Viewing and editing files

cat files...	Concatenate one or more files and send the result to the screen.
less file	Show the contents of file one page at a time. Within less you can use the following commands: Up or Down scroll up or down one line SPACE or f or PageDown scroll one page forward b or PageUp scroll one page backwards g or Home scroll to the start of the file G or End scroll to the end of the file q quits the program.
nano file	Edit a file using the nano editor. (Install it with apt-get install nano.)
head file	Show the first ten lines of file.
head -n num file	Show the first num lines of file.
tail file	Show the last ten lines of file.
tail -n num file	Show the last num lines of file.

Exercises

Searching for strings in files: grep

Grep searches the contents of one or more files for particular strings (sequences of characters).

Exercises

Why 'grep'?

Searching for files: find

Search patterns

The grep and find program match strings and filenames using patterns. Most characters in a pattern match themselves literally (as in the command.log example above). Some characters are wildcards and do not match themselves literally (as in the *.txt example above). The most common examples are:

Hence 'a*.t?[rp]' will match 'about.tmp' and 'archive.tar' but will not match 'about.txt' or 'about.temp' or 'other.tar'.

Exercises

Counting characters, words, and lines

The wc program prints character, line, and word count for text files.

Features of the shell

Environment variables

Environment variables store values for later use. For example:

When assigning to a variable you just use its name on the left of the assignment operator '='. (Note that there must not be any spaces either side of the '='.) To get the value of the variable you put $ in front of its name. (The $ tells the shell to replace the next thing on the line with something else. If the 'next thing' is the name of a variable then it is replaced with the value of that variable, or nothing if the variable is not defined. This is called variable substitution.) You can use variables this way in any command. Assuming the above assignments:

bash-3.2$ cd $TEMPDIR bash-3.2$ pwd /mnt/bigdisk/tmp bash-3.2$ echo $foo bash-3.2$

Environment variables are often used to control the behaviour of programs. Several environment variables affect the behaviour of the shell itself.

bash-3.2$ OLDPS1="$PS1" bash-3.2$ PS1="what should I do next? " what should I do next? pwd /mnt/bigdisk/tmp what should I do next? echo $foo 42 what should I do next? PS1="$OLDPS1" bash-3.2$

Exercises

Path name expansion

The same patterns that are used by grep and find are also used by the shell to match file names that you type on the command line. If a pattern on the command line matches at least one file then it is replaced by the matching filename(s). If a pattern matches no files then it is not replaced. Assuming your current directory contains only two files called 'data.txt' and 'results.txt' then:

bash-3.2$ echo *.log *.log bash-3.2$ echo *.txt data.txt results.txt bash-3.2$

Hidden files and directories (whose names begin with '.') are not included in the results of path name expansion.

Quoting

When the shell sees a wildcard in a word it will try to replace it with as many matching filenames as it can find. The command find . -name *.txt will normally find all .txt files under the current directory. However, if the current directory contains a file called data.txt then the shell will replace the pattern *.txt with data.txt before running the find command. The result will be is as if you had typed find . -name data.txt which is probably not what you wanted. To prevent the shell from expanding wildcards in path names, surround the name with single quotes: find . -name '*.txt'

Double quotes "…" work almost the same way, except that $ continues to perform variable substitution and command substitution (see below) within the quoted text.

bash-3.2$ echo '$HOME *.txt' $HOME *.txt bash-3.2$ echo "$HOME *.txt" /home/piumarta *.txt bash-3.2$ echo $HOME *.txt /home/piumarta data.txt bash-3.2$

Interactive history

A feature of bash is that you can use the up-arrow keys to access your previous commands, edit them (if desired) by typing, and re-execute them by pressing Enter.

To see a list of recently used commands, type: fc -l

Command and filename completion

Another feature of bash is that you can use the TAB key to complete a partially typed filename. For example, if you have a file called 'shopping-list-for-pizza-party-on-july-27.txt' in your directory and want to edit it you can type nano shop, press the TAB key, and bash will fill in the rest of the name for you (assuming that only one file starts with 'shop' in your current directory).

If there are multiple completions, if pressing TAB more than once will show you a list of all the possible completions.

Redirection

The redirection operator > file writes a command's output to the given file (the original contents of file will be deleted first). The redirection operator >> file appends a command's output to the given file.

One way to see a very long list of files one page at a time is to redirect the output of ls to a file, then look at the file one page at a time using less:

bash-3.2$ ls -l > /tmp/listing.txt bash-3.2$ less /tmp/listing.txt contents of file displayed bash-3.2$

A similar redirection < file exists to connect a program's input to a file instead of reading from the keyboard.

Exercises

Pipes

The pipe operator '|' (vertical bar) is used to direct the output of one command to the input of another command. For example:

The last example demonstrates the philosophy that underlies many of the command-line tools, which emphasizes composability (instead of monolithic design). Each program does one thing (well) and is written to work together with other programs by creating pipelines that connect the output of one command to the input of the next command in the pipeline, as demonstrated in the example above.

Exercises

Command substitution

You can use the output of one command as an input to another command in another way called command substitution. Command substitution replaces part of a command with the output from another command. The part of the command to be substituted is written inside $( and ).

Exercises

Arithmetic substitution

Arithmetic substitution replaces part of a command with the result of an arithmetic expression. The part of the command to be substituted is written inside $(( and )) and should be a valid arithmetic expression involving integers and variables. Within the expression, variable names will be substituted even if they are not preceded with the usual '$' character.

bash-3.2$ foo=20 bash-3.2$ echo $foo 20 bash-3.2$ echo $((foo*2)) 40 bash-3.2$ foo=$(((foo+1)*2)) bash-3.2$ echo $foo 42 bash-3.2$

Looking for help: man and help

Most commands have a manual page which gives useful (often detailed and sometimes cryptic) descriptions of their usage. Example:

The last of these commands explains that the man command has a -k option that means 'keyword', and prints a list of all manual pages mentioning that keyword.

bash-3.2$ man -k editor nano (1) - Nano's ANOther editor, an enhanced free Pico clone sed (1) - stream editor for filtering and transforming text bash-3.2$

(Your list, expecially on Linux, is probably a lot longer than this.)

For help with built-in commands (such as echo) type help. On its own it shows you a summary of the built-in commands. With the name of a command it explains in detail how that command works. Try both help on its own and help help.

Exercises

Editing text files with nano

Most command line environments have access to nano, a small and simple editor for plain text files. To edit file.txt with nano, type: nano file.txt

The table below lists many of the commands for editing text with nano. (Ctrl+X means hold down the Control key while typing X. Depending on your computer and operating system, Alt+X could mean type X while holding down Alt or Command or Option or Windows. You can also simulate Alt+X by typing Escape before typing X.)

To see the built-in guide to the commands, press Ctrl-G.

Shell scripts

Useful sequences of commands can be stored in a file and executed as a single command. For example, when working on a project it might be useful to make a snapshot of all the files in a directory. The snapshot directory should be named after the current date and time. Here is a simple script that does exactly that.

#!/bin/bash
DIR=".checkpoint-$(date +%Y%m%d-%H%M%S)"
mkdir    "$DIR"
cp -pr * "$DIR"
ls -ld   "$DIR"

The only unfamiliar command is date. This is what each line of the script does:

#!/bin/bash

All scripts should begin with a line similar to this one. It tells the operating system that this is a script (#!) and then the name of an interpreter that can understand the contents of the script (in this case the shell /bin/bash).

DIR=".checkpoint-$(date +%Y%m%d-%H%M%S)"

The date command prints the date and time in the specified format: four-digit year (%Y) two-digit month (%m) two-digit day (%d) a literal dash (-) two-digit hour (%H) two-digit minute (%M) two-digit seconds (%S). The output from date is by substituted ($(…)) into the enclosing command, which stores the path name of the new checkpoint directory in a variable (DIR="…").

mkdir "$DIR"

The checkpoint directory is created (mkdir) using the value of the previously-stored variable ("$DIR").

cp -pr * "$DIR"

All non-hidden file and directory path names (*) are copied (cp) recursively (-r) while preserving timestamps (-p) from the current directory into the checkpoint directory ($"DIR"). (Because the checkpoint directories all begin with '.' they are hidden, they will not be included in the expansion of *, and will therefore not be included in the copy operation.)

ls -ld "$DIR"

As feedback to the user, list (ls) in long format (-l) the details of the checkpoint directory (-d) that was created.

Running a script has the same effect as typing the commands it contains, one at a time, at the prompt. (One way to develop a part of a script is to do exactly that, until the desired effect is achieved, then copy the commands into the script being written.)

Assuming the script is in a file called 'checkpoint.sh' and is executable (chmod +x checkpoint.sh) then running it will produce something like this:

bash-3.2$ ls 01-cursor-hmove-bb-1.png 01-cursor-hmove.obj 01-cursor-hmove.png 01-example-email.txt 01-cursor-hmove-bb.pdf 01-cursor-hmove.pdf 01-example-email-attachment.txt bash-3.2$ ./checkpoint.sh drwxrwxr-x 2 piumarta staff 374 2020-10-10 15:26 .checkpoint-20201010-152935 bash-3.2$ ls .checkpoint-20201010-152935 01-cursor-hmove-bb-1.png 01-cursor-hmove.obj 01-cursor-hmove.png 01-example-email.txt 01-cursor-hmove-bb.pdf 01-cursor-hmove.pdf 01-example-email-attachment.txt bash-3.2$