~~NOCACHE~~ ====== Command line interface ====== Engineers often use command line tools to help them manage files and directories, analyse data, or develop software tools. A popular command line interface available on all major operating systems is the //shell//. Examples include //bash//, which is used in the explanations below. ===== Shells ===== The command line interface is implemented by a program called a //shell//. The appearance and behaviour of the command line interface will differ slightly depending on which shell you are using. You can find out what shell you are using by the command: echo $SHELL The default shell on Linux and Windows (MobaXterm or WSL) is bash. The default shell on MacOS recently changed from bash to zsh, but they are very similar to each other. (If you want to temporarily switch from zsh to bash in MacOS, just type ''bash'' and press ''Enter''.) ++++ Why ‘shell’? | The operating system is actually a relatively small program, that has no user interface at all, called the //kernel//. The kernel manages resources, enforces security, and controls the creation, execution, and termination of programs. One of those programs connects to the user through the screen and keyboard, interprets what they type, runs other programs on their behalf, and provides a text-based method to visualise and control the files and directories in the filesystem. Making an analogy to nuts, this wrapper around the kernel is called the shell.++++ ++++ Why ‘bash’? | One of the first and most popular shells on Unix systems was written by Steve Bourne. It was re-implemented and extended as an open-source program, and given the name ‘Bourne again shell’, or ‘bash’ for short. (‘Bourne again’ is a play on the original author’s name and the phrase ‘born again’, when someone makes a radical change in their life and becomes a new person.)++++ /* You can create a file with a list of shell commands and execute it like a program to perform a task. This kind of file is called a //shell script//. Executing scripts is in fact the primary purpose of most shells since only a few shells have the features needed for interactive command line use. */ ===== Basic command line (shell) usage ===== The shell prints a //prompt// to let you know it is waiting for your next command. Prompts are personal and configurable, so they will look different depending on the default configuration used by your installation. ^ **MobaXterm on Windows with prompt style 'modern'** | {{06-prompt-1.png?direct|MobaXterm modern}} | | date, time, current directory > | ::: | | ::: || | | ::: | ^ **MobaXterm on Windows with prompt style 'standard'** | {{06-prompt-2.png?direct|MobaXterm standard}} | | [date, time] current directory | ::: | | [account name . computer name] > | ::: | | | ::: | ^ **WSL running Debian** | {{06-prompt-3.png?direct|WSL running Debian}} | | account name @ computer name : current directory $ | ::: | | ::: || | | ::: | ^ **Xubuntu** | {{06-prompt-4.png?direct|Xubuntu}} | | account name @ computer name : current directory $ | ::: | | ::: || As you are typing the command you can use ''Backspace'' to correct errors. If you want to discard the entire command, type ''Ctrl-u'' (hold down ''Control'' while typing ''u''). When you have finished typing the command //do not forget// to press ''Enter'' so that the shell knows you have finished typing. In the rest of this document fixed-width text indicates the name of something (usually a file or directory) or output that the computer prints, ''fixed-width font in a box'' indicates something you would type, and: $ **a large box** with several lines indicates a dialogue with the shell, with your input $ **in bold** and the computer's output in non-bold Note that the ''Enter'' pressed at the end of every command is not shown explicitly. ++++ Exercise | - Type ''ls'' after the prompt but do not press ''Enter''. No matter how long you wait the shell will not 'get bored' and decide to run the ''ls'' command for you. * When you get bored, press ''Enter'' to run the ''ls'' command. ++++ ==== Interrupting programs ==== To interrupt (terminate) a running program, hold down the ''Control'' key while typing ''C''. (This is usually written ''Ctrl''+''C''.) ++++ Exercise | - Type ''cat'' and press ''Enter''. This command will accept any line of text that you type and immediately print it back at you. It will not terminate until you explicitly interrupt it. * Terminate the ''cat'' command now by typing ''Ctrl''+''C''. ++++ ==== Directories ==== File and directory paths use the forward slash '/' to separate directory names in a path. '/' (the directory with no name) is the 'root' directory, the top-most directory in the filesystem. Every computer user has a 'home' directory where the files belonging to that user are stored. If your account name is fred then on Mac and Windows your home directory is /Users/fred and on Linux your home directory is /home/fred. | / | the 'root' directory, at the top of the filesystem hierarchy | | /usr | the directory usr, a sub-directory of / the 'root' directory | | /usr/bin | the directory bin, a subdirectory of /usr | | . | the current directory | | .. | the parent directory (except at the root, where .. points back to the root) | | ~ | a compact notation for your home directory | | ~/Pictures | a compact notation for the Pictures directory in your home directory | | ~bob | a compact notation for bob's home directory | | ~bob/tmp | a compact notation for bob's temporary directory in his home directory | ++++ Excercises | - You have a file called data.txt inside a directory called Projects which is located in your home directory. * What is the full path name of the data.txt file, starting from the root directory '/'? * What is the path name of the data.txt file, starting with your home directory '~'? \\ \\ - There is nothing special about '.' and '..' --- they behave exactly like any other directory names. * What directory does the path '~/../bob' refer to? \\ \\ - Home directories are stored in either /Users or /home, depending on the operating system. * What path name refers to this 'directory of home directories', regardless of the operating system you are using? ++++ ==== How the shell interprets what you type ==== Most lines you type at the prompt have the same form: prompt$ **command //arg1// //arg2// ... //argN//** The first thing on the line is a //command//. It tells the shell what you want to do. A few of the most popular commands are built-in to the shell and interpreted by it directly. Most of the commands are programs located in standard places, such as /bin and /usr/bin. Following the command are zero or more //arguments//. These convey additional information to the command, such as the name of files/directories or //options// (starting with a dash `-') that modify how the command behaves. The first thing the shell does when you press ''Enter'' is to split the line you typed into command and arguments. It does this by looking for white space (space and tab characters). After they have done their job of separating command and arguments, the white space characters are discarded. This is one reason why it is a bad idea to put spaces in file names. If you really want to put spaces in file names or other arguments, we will see ways of [[#quoting|quoting]] words. A surprisingly useful built-in command is ''echo'' which prints the arguments (if any) that you pass to it, followed by a newline character. In addition to printing messages (and blank lines) ''echo'' is useful because it lets you see exactly how the shell is manipulating what you type before running a command. ++++ Exercise | - Run the echo command with different arguments. * ''echo'' * ''echo $SHELL'' * ''echo $USER'' * ''echo ~'' * ''echo $SHELL says hello to $USER who lives in ~'' ++++ ==== Moving around the file system ==== Every running program, including the shell, has a current working directory. In the shell this is where you appear 'to be' within the filesystem. When you start the shell, the current working directory is set to your home directory. If you use a file or directory name that does not begin with / then the search for that file/directory begins in the current working directory (instead of at the root of the filesystem). These are called //relative// paths because the file/directory they refer to changes as you move your working directory around in the filesystem. File/directory names that begin with / are called //absolute// paths. The ''pwd'' command **p**rints the current **w**orking **d**irectory. The ''cd'' command **c**hanges the current working **d**irectory. | ''pwd'' | **P**rint the current **w**orking **d**irectory name. | | ''cd'' | Change current directory to your //home// directory. | | ''cd /usr/local'' | Change current directory to /usr/local. | | ''cd bin'' | Change current directory to bin which is a sub-directory of the current directory. | | ''cd ..'' | Change current directory to the parent directory of the current directory. | | ''cd $TMPDIR'' | Change current directory to the directory defined by the [[#Environment variables|environment variable]] TMPDIR. | | ''cd ~'' | Change the current directory to your home directory. | | ''cd ~bob'' | Change the current directory to the user bob's home directory (if you have permission). | | ''cd -'' | Change back to the previous working directory (the - is interpreted specially). | ++++ Excercises | - Write down the simplest sequence of commands that you could use to perform each of these tasks: * Change to the /etc directory and verify that the change was successful? * Change to your home directory verify that the change was successful? \\ \\ - I just executed these commands: ''cd ~'' then ''cd -'' then ''cd -''. * What is my current working directory? ++++ ==== Listing directory contents ==== The ''ls'' command **l**i**s**ts the details of files and directories (including their contents). By default it lists the details (and contents) of the current directory. | ''ls'' || List the current directory. | | ''ls -l'' || List the current directory using //long// format (showing details read from the file's inode in the listing). | For example: {{06-ls-l-bb.png?750x|Long format listing}} For regular files, execute permission means they can be run as a program. For directories, execute permission means they can be searched (listed by ''ls'', etc.). Each permission has its own dedicated position in the mode display, and is either a letter (r/w/x) indicating ‘allowed’ or a dash (-) indicating ‘denied’. Here is a summary of the modes: | r-------- | Allow read by owner. | | -w------- | Allow write by owner. | | --x------ | For files, allow execution by owner; for directories, allow the owner to search in the directory. | | ---r----- | Allow read by group members. | | ----w---- | Allow write by group members. | | -----x--- | For files, allow execution by group members; for directories, allow group members to search in the directory. | | ------r-- | Allow read by others. | | -------w- | Allow write by others. | | --------x | For files, allow execution by others; for directories allow others to search in the directory. | Other options understood by ''ls'' include: | ''ls -a'' | List the current directory including hidden files. (Hidden files start with '.', therefore '.' and ..' are usually hidden.) | | ''ls -R'' | List recursively (''-R'') the current directory and all files/directories below it in the directory hierarchy. | | ''ls -ld *'' | List all the file and directory names in the current directory using long format (''-l''), showing details about directory inodes (''-d'') rather than about the files they contain. | | ''ls -F'' | Place a character after each file name as a visual clue indicating its type. For example directories are followed by a slash (/) and executables by an asterisk (*). | To understand the ''-d'' option, try ''ls -l .'' and ''ls -ld .''. ++++ Exercises | - What command would you use to perform the following tasks: * Print a long-format listing of your home directory, regardless of what your current working directory might be? ++++ ==== Changing file permissions and attributes ==== Every file 'belongs' to exactly one //user//. This user is the 'owner' of the file. Every file also 'belongs' to exactly one //group//. A group is a collection of users, and each user can belong to several groups. The idea is that a team of users working together can create a group and then share access to files and directories among members of the group. Access to files is therefore controlled for three sets of people: the **user** (owner) of the file, the members of the **group** to which the file belongs, and **others** (everyone else). The ''chmod'' command **ch**anges the access **mod**e of a file. The //access mode// of a file is a collection of nine //permissions// that control **r**eading, **w**riting, and e**x**ecuting the file. Each of these activities can be controlled independently for the file's **u**ser (its owner), members of the file's **g**roup, and **o**thers (anyone who is neither the owner of the file nor a member of the same group as the file). To specify how to modify the permissions say who the change applies to (**u**, **g**, **o**, or **a** which means 'all of them'), then what kind of change (**+** to add, **-** to remove, or **=** to set), and then which activities are being modified (**r**, **w**, **x**, or **a** meaning 'all of them'). The option ''-R'' applies the specified mode recursively to a directory and everything below it in the hierarchy. | ''chmod a+x file'' | Add e**x**ecute permission to file for **a**ll (user, group, and others). | | ''chmod go-w file'' | Remove **w**rite permission from file for members of the **g**roup and for **o**thers. | | ''chmod u=rwx,go=rx file'' | Set the permissions of file to be rwx for the **u**ser, and r-x for **g**roup members and all **o**thers. | | ''chmod 755 file'' | Set the permissions of file to be rwx for the **u**ser, and r-x for the **g**roup and **o**thers. (7 = 111 in binary = rwx, and 5 = 101 in binary = r-x). Note that ''chmod 0 file'' removes //all// permissions for //everyone//.) | | ''chgrp www-data file'' | **Ch**ange the **gr**ou**p** of file to www-data. | | ''chown piumarta file'' | **Ch**ange the **own**er (user) of file to piumarta. | | ''chown -R piumarta dir'' | **Ch**ange the **own**er of dir, and //everything below it// in the directory tree, to piumarta. | You must be the owner of the file/directory, or be logged in as the user root (the 'superuser'), before you can do any of these things. ++++ Exercises | - Assume you have a file called data.txt. What commands would you use to change its permissions in the following ways: * Set the permissions so that everyone can read it but only the user (owner) can write to it? * Ensure that the members of the file's group can read and write it? * Ensure that others (not owner, not group members) cannot access it at all? \\ \\ - Assume you have a hidden directory called '~/.ssh'. * What command would you use to make sure only you can read, write, and search it? * What would its //numeric// access mode (as a three-digit number) be after executing that command? ++++ ==== Moving, renaming, and copying files ==== The following commands **c**o**p**y files, **m**o**v**e files, **r**e**m**ove files, **m**a**k**e **dir**ectories, and **r**e**m**ove **dir**ectories. | ''cp file1 file2'' | **C**o**p**y file1 to file2. If file2 exists, it will be over-written. | | ''cp -p file1 file2'' | Copy file1 to file2 preserving (''-p'') inode information (permissions, timestamps, etc.). | | ''cp -r dir1 dir2'' | Copy recursively (''-r'') dir1 and its contents to dir2. | | ''cp -pr dir1 dir2'' | Copy dir1 and its contents to dir2 preserving permissions, timestamps, etc. | | ''mv file1 file2'' | **M**o**v**e file1 to file2. If file2 exists, it will be deleted before file1 is renamed. If file1 and file2 are in the same filesystem, only directory entries are modified (the inode, and the contents of the file1, are not copied or even modified at all). | | ''mv file1 ~/tmp/'' | Move file1 into sub-directory tmp in your home directory. | | ''rm file ...'' | **R**e**m**ove (delete) one or more files. When a file is deleted, its directory entry is deleted and the link count on its inode is reduced by 1. The inode and the contents of the file are not actually deleted unless the link count drops to 0. | | ''rm -r dir ...'' | Recursively (''-r'') remove a directory and its contents. | | ''rm -f file ...'' | Forcibly (''-f'') remove a file (with no warning if the file is write-protected or not found). | | ''rm -rf dir ...'' | Forcibly remove a directory and its contents. | | ''mkdir dir1 [dir2...]'' | **M**a**k**e (create) new **dir**ectories. | | ''mkdir -p path/to/dir'' | Create a directory including any missing parent directories that dir requires. | | ''rmdir dir1 [dir2...]'' | **R**e**m**ove an empty **dir**ectory. (If the directory is not empty, the command will fail.) | The command line tools always assume that you know exactly what you are doing and they will try to do what you ask without complaint or warning, even if what you are asking them to do will cause great damage. Be //very// careful with ''rm -f'', ''rm -r'', and //__especially__// ''rm -rf''. If you accidentally type ''rm -rf / tmp/junk'' (with an unintended space between '/' and 'tmp') then you will (1) erase the //entire disk// (''rm -rf /'' erases recursively, without warning, the root directory and everything below it) and (2) spend the next several hours reinstalling your operating system and recovering all your files from the last backup that you made. A safer way to perform the ''rm -rf /tmp/junk'' command is: | cd /tmp | | | pwd | verify //twice// that this command actually printed '/tmp' | | ls junk | verify //twice// that you see only the files you want to delete | | rm -rf junk | | Remember: the command line does not have a 'trash can'. You cannot undo the ''rm'' command. **Pro tip**: Instead of removing files, first ''mkdir old'' and then ''mv'' the unwanted files into old. When you have finished your session, if everything still looks and works as expected, finish by deleting the contents of the old directory and then ''rmdir old''. ++++ Exercises | - What commands would you use to perform the following tasks? * Create a directory called 'Assignments' in your home directory? * Create a directory called 'data' under the '/tmp' directory, copy 'data.txt' to it, and finally remove the 'data' directory and everything in it? ++++ ==== Viewing and editing files ====
cat files... Concatenate one or more files and send the result to the screen.
less file Show the contents of file one page at a time. Within less you can use the following commands:

Up or Down scroll up or down one line
SPACE or f or PageDown scroll one page forward
b or PageUp scroll one page backwards
g or Home scroll to the start of the file
G or End scroll to the end of the file
q quits the program.
nano file Edit a file using the nano editor. (Install it with apt-get install nano.)
head file Show the first ten lines of file.
head -n num file Show the first num lines of file.
tail file Show the last ten lines of file.
tail -n num file Show the last num lines of file.
++++ Exercises | - What commands would you use to: * Print the entire contents of the file /etc/passwd to the screen? * Print the first 4 lines of the same file? * Print the last 2 lines of the same file? ++++ ==== Searching for strings in files: grep ==== Grep searches the contents of one or more files for particular strings (sequences of characters). | ''grep string files..'' | Print all the lines in one or more files that contain the given string. | ++++ Exercises | - Assume you have a file called 'contacts.txt' containing contact information for your friends, colleagues, and professors. The entry for each contact begins with a line like this: 'NAME: //last-name first-name//'. For example, the entry for me might start with 'NAME: Piumarta Ian'. * What command would you use to see a list of the names of the contacts in your file? ++++ ++++ Why 'grep'? | The first editor on Unix systems was ''ed''. Ed works on one line of a file at a time and uses text commands to select and manipulate those lines. Lines are selected by patterns called //regular expressions// that are written between slash (/) characters. So ''/NAME:/'' selects the next line that includes the string 'NAME:'. The ''p'' command prints the selected line, so ''/NAME:/p'' selects the next line containing 'NAME:' prints it. Writing ''g'' (for 'global') before a selection makes it apply to all matching lines in the file. The command ''g/NAME:/p'' therefore selects every line containing 'NAME:' in the file and prints all of them. The general format of an ''ed'' command that will 'print all lines containing some regular expression //re//' is therefore: 'g\///re//\/p'. That operation was so useful in ''ed'' that it was turned into a standalone program that **g**lobally matches **r**egular **e**xpressions in files and **p**rints them, or ''grep'' for short. ''ed'' is still available on computers that support a standard command line interface. Type ''ed //filename//.txt'' to run it on a text file, then type ''g/.../p'' (replace ''...'' with some string that you know occurs in the file) and press ''Enter'', and ''ed'' should print all the matching lines for you. Type ''q'' and press ''Enter'' to quit.++++ ==== Searching for files: find ==== | ''find directory -name pattern'' | Find all the files in directory whose name matches pattern. | | ''find . -name command.log'' | Find files named command.log in the current directory, or in any subdirectory below it in the directory tree. | | ''find / -name '*.txt' '' | Find all files whose names end in .txt anywhere on the disk. | | ''find .. -type d'' | Find all directories (''-type d'') that are below the parent directory. | ==== Search patterns ==== The ''grep'' and ''find'' program match strings and filenames using patterns. Most characters in a pattern match themselves literally (as in the command.log example above). Some characters are //wildcards// and do not match themselves literally (as in the *.txt example above). The most common examples are: | ? | Match any single character, no matter what it is. | | * | Match zero or more characters, no matter what they are. | | [//chars//] | Match one character from the given list of //chars//. | Hence 'a*.t?[rp]' will match 'about.tmp' and 'archive.tar' but will not match 'about.txt' or 'about.temp' or 'other.tar'. ++++ Exercises | - One way to keep notes very well organised is to put them in files called '//year-month-date-topic//.txt'. For example, '2020-10-10-assignments.txt'. What command would you use to search your home directory for: * All notes on any topic created in the year 2020? * All notes on any topic in any year created on the first day of the month? * All notes on any topic in any year created in November or December? * All notes on the topic of 'recipes'? ++++ ==== Counting characters, words, and lines ==== The ''wc'' program prints character, line, and **w**ord **c**ount for text files. | ''wc essay.txt'' | Print the number of characters, words, and lines in the file essay.txt. | | ''wc -l *.txt'' | Print the number of lines in all the text files (whose names end in .txt) in the current directory. | ===== Features of the shell ===== ==== Environment variables ==== Environment variables store values for later use. For example: | ''foo=42'' | Set the variable foo to the value 42. | | ''export TEMPDIR=/mnt/bigdisk/tmp'' | Set the variable TEMPDIR to the value /mnt/bigdisk/tmp. | When assigning to a variable you just use its name on the left of the assignment operator '='. (Note that there must not be any spaces either side of the '='.) To get the value of the variable you put $ in front of its name. (The $ tells the shell to replace the next thing on the line with something else. If the 'next thing' is the name of a variable then it is replaced with the value of that variable, or nothing if the variable is not defined. This is called //variable substitution//.) You can use variables this way in any command. Assuming the above assignments: bash-3.2$ **cd $TEMPDIR** bash-3.2$ **pwd** /mnt/bigdisk/tmp bash-3.2$ **echo $foo** bash-3.2$ Environment variables are often used to control the behaviour of programs. Several environment variables affect the behaviour of the shell itself. bash-3.2$ **OLDPS1=\"$PS1\"** bash-3.2$ **PS1=\"what should I do next? \"** what should I do next? **pwd** /mnt/bigdisk/tmp what should I do next? **echo $foo** 42 what should I do next? **PS1=\"$OLDPS1\"** bash-3.2$ ++++ Exercises | - What commands would you use to perform the following tasks: * Set the environment variable NAME to be your first name (e.g., 'Bob')? * Print a greeting to yourself (e.g., 'Hello Bob) using ''echo'' and the value of the NAME variable? ++++ ==== Path name expansion ==== The same patterns that are used by ''grep'' and ''find'' are also used by the shell to match file names that you type on the command line. If a pattern on the command line matches at least one file then it is replaced by the matching filename(s). If a pattern matches no files then it is not replaced. Assuming your current directory contains only two files called 'data.txt' and 'results.txt' then: bash-3.2$ **echo *.log** *.log bash-3.2$ **echo *.txt** data.txt results.txt bash-3.2$ Hidden files and directories (whose names begin with '.') are not included in the results of path name expansion. ==== Quoting ==== When the shell sees a wildcard in a word it will try to replace it with as many matching filenames as it can find. The command ''find . -name *.txt'' will normally find all .txt files under the current directory. However, if the current directory contains a file called data.txt then the shell will replace the pattern *.txt with data.txt //before// running the find command. The result will be is as if you had typed ''find . -name data.txt'' which is probably not what you wanted. To prevent the shell from expanding wildcards in path names, surround the name with single quotes: find . -name '*.txt' Double quotes \"...\" work almost the same way, except that $ continues to perform [[#Environment variables|variable substitution]] and [[#Command substitution|command substitution]] (see below) within the quoted text. bash-3.2$ **echo '$HOME *.txt'** $HOME *.txt bash-3.2$ **echo \"$HOME *.txt\"** /home/piumarta *.txt bash-3.2$ **echo $HOME *.txt** /home/piumarta data.txt bash-3.2$ ==== Interactive history ==== A feature of bash is that you can use the up-arrow keys to access your previous commands, edit them (if desired) by typing, and re-execute them by pressing ''Enter''. To see a list of recently used commands, type: ''fc -l'' ==== Command and filename completion ==== Another feature of bash is that you can use the ''TAB'' key to complete a partially typed filename. For example, if you have a file called 'shopping-list-for-pizza-party-on-july-27.txt' in your directory and want to edit it you can type ''nano shop'', press the ''TAB'' key, and bash will fill in the rest of the name for you (assuming that only one file starts with 'shop' in your current directory). If there are multiple completions, if pressing ''TAB'' more than once will show you a list of all the possible completions. ==== Redirection ==== The redirection operator ''> file'' writes a command's output to the given file (the original contents of file will be deleted first). The redirection operator ''>> file'' appends a command's output to the given file. | ''grep string filename > newfile'' | Redirect the output of the grep command to the file 'newfile'. | | ''grep string filename >> existfile'' | Append the output of the grep command to the end of 'existfile'. | One way to see a very long list of files one page at a time is to redirect the output of ''ls'' to a file, then look at the file one page at a time using ''less'': bash-3.2$ **ls -l > /tmp/listing.txt** bash-3.2$ **less /tmp/listing.txt** // contents of file displayed // bash-3.2$ A similar redirection ''< file'' exists to connect a program's input to a file instead of reading from the keyboard. | ''grep string'' | Print lines typed at the keyboard that match string. | | ''grep string < filename'' | Print lines read from filename that match string. | ++++ Exercises | - What command would you use to perform the following tasks: * Find all the text files called '*.txt' in or below the current directory, storing the list of files in another file called '/tmp/files.txt'? ++++ ==== Pipes ==== The pipe operator '|' (vertical bar) is used to direct the output of one command to the input of another command. For example: | ''ls -l \| less'' | This runs the long (''-l'') format directory **l**i**s**ting command ''ls -l'' and redirects its output (''\|'') to the input of the ''less'' command. Compared to the example above, this is a much better way to view a very long list of files one page at a time. | | ''du -s * \| sort -n \| tail'' | The command ''du -s'' lists the byte size (''-s'') of the **d**isk **u**sage of all files and directories in the current working directory. The output is piped into ''sort'' which **sort**s the lines on its input numerically (''-n'') from smallest to largest and prints the result. Finally the output from ''sort -n'' is piped into the ''tail'' command, which displays only the last few results (which will be the largest, because of the sorting). | The last example demonstrates the [[https://en.wikipedia.org/wiki/Unix_philosophy|philosophy]] that underlies many of the command-line tools, which emphasizes //composability// (instead of //monolithic// design). Each program does one thing (well) and is written to work together with other programs by creating //pipelines// that connect the output of one command to the input of the next command in the pipeline, as demonstrated in the example above. ++++ Exercises | - What pipeline of commands would you use to perform these tasks: * Print the names of the first five files in the /etc directory? \\ \\ - I have a file called numbers.txt that is 7 lines long and includes the following words, one per line: one two three four five six seven. What is printed by each of these commands: * ''head -n 5 numbers.txt | tail -n 2'' ? * ''head -n 5 numbers.txt | tail -n 2 | sort'' ? * ''head -n 5 numbers.txt | sort | tail -n 2'' ? ++++ ==== Command substitution ==== You can use the output of one command as an input to another command in another way called //command substitution//. Command substitution replaces part of a command with the output from another command. The part of the command to be substituted is written inside ''$('' and '')''. | ''cat *.txt'' | Con**cat**enate the contents of all text files in the current directory and print the result on the screen. | | ''find . -name '*.txt' '' | Print the path names of all .txt files in the current directory or anywhere below it in the directory hierarchy. | | ''cat $(find . -name '*.txt')'' | Concatenate the contents of all text files in the current directory //or anywhere below it// in the directory hierarchy. | ++++ Exercises | - Assuming you have a list of path names, one per line, in a text file called 'files.txt'. What command would you use to print: * Just the first and last path name in the file, on the same line? (Hint: one way to do this uses ''echo'' followed by two command substitutions.) ++++ ==== Arithmetic substitution ==== Arithmetic substitution replaces part of a command with the result of an arithmetic expression. The part of the command to be substituted is written inside ''$\(('' and ''\))'' and should be a valid arithmetic expression involving integers and variables. Within the expression, variable names will be substituted even if they are not preceded with the usual '$' character. bash-3.2$ **foo=20** bash-3.2$ **echo $foo** 20 bash-3.2$ **echo $\((foo*2\))** 40 bash-3.2$ **foo=$\(\((foo+1)*2\))** bash-3.2$ **echo $foo** 42 bash-3.2$ ===== Looking for help: man and help ===== Most commands have a manual page which gives useful (often detailed and sometimes cryptic) descriptions of their usage. Example: | ''man ls'' | Show the **man**ual page for the ls command. | | ''man bash'' | Show the manual page for the bash command. | | ''man man'' | Show the manual page for the man command. | The last of these commands explains that the ''man'' command has a ''-k'' option that means 'keyword', and prints a list of all manual pages mentioning that keyword. bash-3.2$ **man -k editor** nano (1) - Nano's ANOther editor, an enhanced free Pico clone sed (1) - stream editor for filtering and transforming text bash-3.2$ (Your list, expecially on Linux, is probably a lot longer than this.) For help with built-in commands (such as ''echo'') type ''help''. On its own it shows you a summary of the built-in commands. With the name of a command it explains in detail how that command works. Try both ''help'' on its own and ''help help''. ++++ Exercises | - You can use the ''help'' command to find out more about the ''echo'' command. In particular, ''echo'' understands several options that modify its behaviour. * What command will print 'hello' on the terminal //without// a newline at the end? ++++ ===== Editing text files with nano ===== {{06-nano.png?direct|Nano}} {{06-nano-help.png?direct|Nano}} Most command line environments have access to ''nano'', a small and simple editor for plain text files. To edit file.txt with nano, type: ''nano file.txt'' The table below lists many of the commands for editing text with nano. (''Ctrl''+''X'' means hold down the ''Control'' key while typing ''X''. Depending on your computer and operating system, ''Alt''+''X'' could mean type ''X'' while holding down ''Alt'' or ''Command'' or ''Option'' or ''Windows''. You can also simulate ''Alt''+''X'' by typing ''Escape'' before typing ''X''.) To see the built-in guide to the commands, press ''Ctrl-G''. ^ File handling ^^ Moving around ^^ | ''Ctrl''+''S'' | Save current file | ''Ctrl''+''B'' | One character backward | | ''Ctrl''+''O'' | Offer to write file ("Save as") | ''Ctrl''+''F'' | One character forward | | ''Ctrl''+''R'' | Read (insert) a file into current one | ''Ctrl''+''←'' | One word backward | | ''Ctrl''+''X'' | Exit from nano | ''Ctrl''+''→'' | One word forward | ^ Editing ^| ''Ctrl''+''A'' | To start of line | | ''Ctrl''+''K'' | Cut current line | ''Ctrl''+''E'' | To end of line | | ''Alt''+''6'' | Copy current line | ''Ctrl''+''P'' | One line up | | ''Ctrl''+''U'' | Paste | ''Ctrl''+''N'' | One line down | | ''Alt''+''T'' | Cut until end of file | ''Ctrl''+''↑'' | To previous block | | ''Ctrl''+'']'' | Complete current word | ''Ctrl''+''↓'' | To next block | | ''Alt''+''3'' | Comment/uncomment line/region | ''Ctrl''+''Y'' | One page up | | ''Alt''+''U'' | Undo last action | ''Ctrl''+''V'' | One page down | | ''Alt''+''E'' | Redo last undone action | ''Alt''+''\ '' | To top of file | ^ Search and replace ^| ''Alt''+''/ '' | To end of file | | ''Ctrl''+''Q'' | Start backward search ^ Special movement ^^ | ''Ctrl''+''W'' | Start forward search | ''Alt''+''G'' | Go to specified line | | ''Alt''+''Q'' | Find next occurrence backward | ''Alt''+'']'' | Go to complementary bracket | | ''Alt''+''W'' | Find next occurrence forward | ''Alt''+''↑'' | Scroll viewport up | | ''Alt''+''R'' | Start a replacing session | ''Alt''+''↓'' | Scroll viewport down | ^ Deletion ^| ''Alt''+''<'' | Switch to previous file | | ''Ctrl''+''H'' | Delete character before cursor | ''Alt''+''>'' | Switch to next file | | ''Ctrl''+''D'' | Delete character under cursor ^ Information ^^ | ''Alt''+''Bsp'' | Delete word to the left | ''Ctrl''+''C'' | Report cursor position | | ''Ctrl''+''Del'' | Delete word to the right | ''Alt''+''D'' | Report word/line/char count | | ''Alt''+''Del'' | Delete current line | ''Ctrl''+''G'' | Display help text | ^ Operations ^^ Miscellaneous ^^ | ''Ctrl''+''T'' | Run spell checker (if available) | ''Alt''+''A'' | Turn the mark on/off | | ''Ctrl''+''J'' | Justify paragraph or region | ''Tab'' | Indent marked region | | ''Alt''+''J'' | Justify entire file | ''Shift''+''Tab'' | Unindent marked region | | ''Alt''+''B'' | Run a syntax check | ''Alt''+''N'' | Turn line numbers on/off | | ''Alt''+''F'' | Run a formatter/fixer/arranger | ''Alt''+''P'' | Turn visible whitespace on/off | | ''Alt''+'':'' | Start/stop recording of macro | ''Alt''+''V'' | Enter next keystroke verbatim | | ''Alt''+'';'' | Replay macro | ''Ctrl''+''L'' | Refresh the screen | | || ''Ctrl''+''Z'' | Suspend nano | ===== Shell scripts ===== Useful sequences of commands can be stored in a file and executed as a single command. For example, when working on a project it might be useful to make a //snapshot// of all the files in a directory. The snapshot directory should be named after the current date and time. Here is a simple script that does exactly that. #!/bin/bash DIR=".checkpoint-$(date +%Y%m%d-%H%M%S)" mkdir "$DIR" cp -pr * "$DIR" ls -ld "$DIR" The only unfamiliar command is ''date''. This is what each line of the script does: ; #!/bin/bash : All scripts should begin with a line similar to this one. It tells the operating system that this is a script (''#!'') and then the name of an //interpreter// that can understand the contents of the script (in this case the shell ''/bin/bash''). \\ \\ ; DIR=\".checkpoint-$(date +%Y%m%d-%H%M%S)\" : The ''date'' command prints the date and time in the specified format: four-digit year (''%Y'') two-digit month (''%m'') two-digit day (''%d'') a literal dash (''-'') two-digit hour (''%H'') two-digit minute (''%M'') two-digit seconds (''%S''). The output from ''date'' is by [[#Command substitution|substituted]] (''$(...)'') into the enclosing command, which stores the path name of the new checkpoint directory in a [[#environment_variables|variable]] (''DIR=\"...\"''). \\ \\ ; mkdir \"$DIR\" : The checkpoint directory is [[#moving_renaming_and_copying_files|created]] (''mkdir'') using the value of the previously-stored [[#environment_variables|variable]] (''\"$DIR\"''). \\ \\ ; cp -pr * \"$DIR\" : All non-hidden file and directory [[#path_name_expansion|path names]] (''*'') are [[#moving_renaming_and_copying_files|copied]] (''cp'') recursively (''-r'') while preserving timestamps (''-p'') from the current directory into the checkpoint directory (''$\"DIR\"''). (Because the checkpoint directories all begin with '.' they are hidden, they will not be included in the expansion of ''*'', and will therefore not be included in the copy operation.) \\ \\ ; ls -ld \"$DIR\" : As feedback to the user, [[#listing_directory_contents|list]] (''ls'') in long format (''-l'') the details of the checkpoint directory (''-d'') that was created. Running a script has the same effect as typing the commands it contains, one at a time, at the prompt. (One way to develop a part of a script is to do exactly that, until the desired effect is achieved, then copy the commands into the script being written.) Assuming the script is in a file called 'checkpoint.sh' and is executable (''chmod +x checkpoint.sh'') then running it will produce something like this: bash-3.2$ **ls** 01-cursor-hmove-bb-1.png 01-cursor-hmove.obj 01-cursor-hmove.png 01-example-email.txt 01-cursor-hmove-bb.pdf 01-cursor-hmove.pdf 01-example-email-attachment.txt bash-3.2$ **./checkpoint.sh** drwxrwxr-x 2 piumarta staff 374 2020-10-10 15:26 .checkpoint-20201010-152935 bash-3.2$ **ls .checkpoint-20201010-152935** 01-cursor-hmove-bb-1.png 01-cursor-hmove.obj 01-cursor-hmove.png 01-example-email.txt 01-cursor-hmove-bb.pdf 01-cursor-hmove.pdf 01-example-email-attachment.txt bash-3.2$ /*** The individual pieces needed are few and simple: * the current date and time in a numeric format, such as '20201010-154213' (just after 15:42 on October 10th, 2020) to name the snapshot directory * a command to create the snapshot directory * a command to recursively copy all non-hidden files from the current directory into the snapshot directory * a command to show the name and long-format information about the snapshot directory Of these, only obtaining the date has not been described above. The ''date'' command prints the date and time. bash-3.2$ **date** Sat 10 Oct 2020 03:10:38 PM JST bash-3.2$ The information can be //formatted// according to a specification that begins with + and then uses the following specifiers: | %Y | print the four-digit year (2020) | | %m | the two-digit month number (01 to 12) | | %d | the two-digit date within the month (01 to 31) | | %H | the two-digit hour (00 to 24) | | %M | the two-digit minute (00 to 59) | | %S | the two-digit second (00 to 59) | bash-3.2$ **date +%Y%m%d-%H%M%S** 20201015-152510 bash-3.2$ ***/ /* * Local Variables: * eval: (flyspell-mode) * eval: (ispell-change-dictionary "british") * eval: (flyspell-buffer) * End: */