Unix Shell Programming

Part of 22C:169, Computer Security Notes
by Douglas W. Jones
THE UNIVERSITY OF IOWA Department of Computer Science

Introduction

The Unix command line interface is known as the shell. The original Unix shell, written by Ken Thompson, was rewritten by Steven Bourne. This version, known as the Bourne Shell, has become the standard for Unix systems. A shell compatable with the Bourne shell is always installed as the sh command on Unix compatable systems.

On Microsoft systems, the command line interface bears a striking similarity to the conventions of Unix shells. The reason is simple. The original model for the Microsoft DOS command line interface was taken from the RT-11 operating system. It appears that Microsoft may have used Unix systems internally for development of DOS, and in any case, many Unix users came to work for the company. Whatever the case may be, as the DOS system was expanded, the general pattern was to copy features of Unix such as the tree-structured file system and many syntactic elements of the shell.

In the context of Unix, the shell is not an integral part of the operating system. If you don't like the shell, feel free to write your own. This has led to a proliferation of Unix shells. Here are just some of them:

For the exercises here, we'll use tcsh.

Why does the shell survive?

The shell continues to survive in the era of sophisticated GUIs for a very simple reason. Here is how Jeff Arnholt put it:

... While many fine GUI program builders are available (such as Tcl/Tk), it is still simpler to program shell scripts for command line input and output. Complex networking, file management, searches, or other system activities are most easily performed in this manner.

... terminal windows usually provide much faster interaction than GUI alternatives. It is much easier and faster to type ``date'' at the command line than to open an Xclock (which requires more keystrokes), move it and/or minimize it, read it, and then close it.

Fially, the shell is a scripting tool. In this role, it can be considered to be the ancestor of specialized languages such as Perl.

Using the Shell from Applications Programs

Many system calls aren't obvious, and programmers that know the Unix shell frequently find it easier to issue a shell command than to write code that does exactly the same using far more obscure system calls or standard library routines. In fact, this is the source of some serious security problems, but it is so easy. If you know that some shell command will do the job, just call that shell command from a user program.

For example, most Unix users know that, to delete the file named fff, you type the shell command rm fff. The Unix standard library includes a library function system that allows any program to execute a shell command. Thus system("rm fff") can be used to delete the file fff. In fact, had the programmer known enough of the Unix kernel, the programmer could have written unlink("fff") to do the same thing.

There are many reasons to use the Unix shell to do a job instead of coding the kernel calls directly. Among other things, the shell understands "globbing" -- the use of wildcards in file names. If a program operated by generating a C program in a temporary file, then compiled it and ran the result, it is much easier to clean up afterward, deleting temp.c, temp.h, and temp.o with the one command system("rm temp.*") than it is to use individual unlink calls for each of the temporary files.

How does a command line interpreter work

The basic structure of all command shells can be described as:

for (;;) {
        buf = fgets(stdin); /* get a string from standard input */
        argv = parse( buf ); /* break buf into its component strings */
        execute( argv[0], argv ); /* execute one command */
}

This is just a funny fetch-execute cycle, where argv serves as an instruction register, where the file pointer on the standard input stream serves as a program counter, and where the opcode field of each instruction is argv[0].

The execute step, involves checking to see if the command is one of the built-in commands in the shell, and if it is, doing that operation directly. If it is not, the shell simply does an exec system call to launch a program to do the job, but it must fork in order to retain control. In more detail, therefore, the code of a shell will look like this:

for (;;) {
        buf = fgets(stdin); /* get a string from standard input */
        argv = parse( buf ); /* break buf into its component strings */
        if (argv[0] is not a builtin command) {
                if (fork()) { /* parent */
                        wait();
                } else { /* child */
                        execute( argv[0], argv );
                }
        } else {
                /* execute builtin command */
        }
}

How does the System Exec a Shell Script

When the exec system call under Unix is given a file to execute, it looks at the first bytes in the file to see what interpreter to use. On other systems, it might look at the filename extension, or it might look at some attribute in the resource fork of the file. The concept remains the same in any of these cases. In the case of Unix variants, each CPU architecture has a "magic number" that is embedded in the first bytes of an executable object file to indicate what CPU type it was compiled for. The exec system call, in that case, will create a process that will directly run that program using that kind of CPU.

If the first byte of the file is #, the exec call takes the remainder of the first line of the file as the file name of an interpreter to launch. Thus, if the file begins with tcsh, the system launches a program called tcsh and gives it the remainder of the file as input. Here is a simple "hello world" shell script:

#tcsh
#  hello
# hello world shell script written for the tcsh shell

echo hello world

Some Shell Script Examples

Here is a shell script that echoes its first argument, demonstrating how a shell script can access its arguments.

#tcsh
#  echo arg
# shell script to echo its first argument

echo $argv[1]

Here is a script that demonstrates (in a rather dull way) sequential execution in the shell, by enclosing the echoed argument in a frame:

#tcsh
#  echo arg
# shell script to echo its first argument in a frame

echo -----------
echo $argv[1]
echo -----------

We can produce exactly the same output as the above script by redirecting the output from the echo commands to a file, outputting that file, and then deleting it.

#tcsh
#  echo arg
# shell script to echo its first argument in a frame

# create a temporary file with the desired content
echo -----------  > tempfile
echo $argv[1]    >> tempfile
echo ----------- >> tempfile

# output the temporary file
cat tempfile

# delete the temporary file
rm -f tempfile

With regard to the above script, note that the > operator in all Unix shells causes the program being run to redirect its output to the indicated file, overwriting any previous content in that file. In contrast, the >> operator causes the program to append its output to the indicated file. This is actually where the C++ and Java languages get their notation for directing data to or from streams.

In the above, the -f (force) option on the rm command tells that command not to bother asking permission to remove the indicated file.

Here is one to echo the first argument but only if there is one. This script demonstrates the conditional features of csh and tcsh. It also demonstrates how to access the argument count.

#tcsh
#  echo arg
# shell script to echo its first argument or complain 

if ($#argv == 1) then
        echo 'argv[1] =' $argv[1]
else
        echo wrong number of arguments
endif

Here is a script to echo all of its arguments. This script demonstrates doing arithmetic, using shell variables, and writing loops in csh.

#tcsh
#  echo arg1 arg2 arg3
# shell script to echo all of its arguments, one per line

@ count = 1
set argc = $#argv
while   ($count <= $argc)
        echo argv[ ${count} ] = $argv[$count] = $value
        @ count = $count + 1
end

References

The Wikipedia provides good coverage here:

The Unix Shell writeup is a good place to start.

Then, take a look at the Bourne Shell writeup and the writeup on the Bourne Shell's descendant, Bash the GNU Bourne Again Shell.

Finally, focus on the C shell since this is the ancestor of tcsh, the shell we are using.

Finally, on whatever system you are using, try the man sh, man bash, man csh, and man tcsh commands. This might also be a good time to review the man command, since most versions of man include tools for searching and navigating around in large manual pages.