24. Too Many Source Files!

Too Many Source Files

As we have noted several times, the directory containing our road-network simulation project is getting congested. Here is the current directory listing:

dwjones@fastx05 ~/project]$ ls
 allclasses-frame.html                'RoadNetwork$1MyEvent.class'
 allclasses-noframe.html              'RoadNetwork$2MyEvent.class'
 constant-values.html                  RoadNetwork.AAA
 deprecated-list.html                  RoadNetwork.bak
 Error.class                           RoadNetwork.class
 Error.java                            RoadNetwork.html
 help-doc.html                         RoadNetwork.java
 index-all.html                        RoadNetwork.shar
 index.html                            script.js
'Intersection$ConstructorFail.class'   serialized-form.html
 Intersection.class                   'Simulator$Event.class'
 Intersection.ConstructorFail.html     Simulator.class
 Intersection.html                     Simulator.Event.html
 Intersection.java                     Simulator.html
 MyRandom.class                        Simulator.java
 MyRandom.html                         Sink.class
 MyRandom.java                         Sink.html
'MyScanner$ErrorMessage.class'         Sink.java
 MyScanner.class                      'Source$1MyEvent.class'
 MyScanner.html                       'Source$2MyEvent.class'
 MyScanner.java                        Source.class
'NoStop$1MyEvent.class'                Source.html
'NoStop$2MyEvent.class'                Source.java
 NoStop.class                         'StopLight$1MyEvent.class'
 NoStop.html                          'StopLight$2MyEvent.class'
 NoStop.java                          'StopLight$3MyEvent.class'
 overview-tree.html                   'StopLight$4MyEvent.class'
 package-frame.html                   'StopLight$5MyEvent.class'
 package-list                          StopLight.class
 package-summary.html                  StopLight.html
 package-tree.html                     StopLight.java
 pathtest                              stylesheet.css
 README.md                             test1
'Road$1MyEvent.class'                  test2
'Road$ConstructorFail.class'           test3
 Road.class                            test4
 Road.ConstructorFail.html             testfile
 RoadFiles                             tests
 Road.html                             testscript
 Road.java
[dwjones@fastx05 ~/project]$ 

Makefiles

The problem of building large applications is complex enough that people began to build tools to automate this in the 1970s. Perhaps the most general of these tools is the make utility that grew out of the C and C++ communities. The thing that makes Make special is that it is not tied to C and C++, but can be applied to any programming language or mix of languages. While make was born in the Unix world, the DOS command line used on Windows systems has sufficient compatability with the Unix shell that make is also used for development under Windows.

Traditionally, the makefile for a project should be called Makefile, assuming that there is only one project in the current directory. Makefiles allow comments with a # (pound sign) prefix, so all of the usual rules for commenting apply. Comments should name the file, its authors, its purpose, and comments and whitespace should be used in a way that explains what is going on to a reader who knows make but does not know anything about this project.

Here is a useful makefile for our project:

# Makefile
# tools for maintaining the road network simulator

# example uses
#   make clean  -- deletes automatically generated files from the directory

clean:
        rm -f *.class *.html package-list script.js stylesheet.css

The basic structure of a make command is illustrated in one of the comments above: Each make command begins with the name of a target. To make the target clean, the make command will execute the shell command indented on the following line or lines. Classical versions of make require that all indenting in the makefile be done with tabs, spaces do not work. This is certainly a design error, but we are stuck with it.

By convention, most large projects have a make clean mechanism to delete everything that is automatically generated so that you can package up the contents of the directory for export, and so that you can clean out all the secondary files before you make a backup, and so that you can clean up the directory before you list it at the start of a day's work doing development.

Our project is intended to be used with Javadoc, so we can use the makefile to document how this is done as well:

# Makefile
#   input to make command to make things related to road network simulator
#   author Douglas W. Jones

# example uses
#   make clean -- deletes automatically generated files from the directory
#   make html  -- makes all the HTML documentation using javadoc

html: classes
        javadoc @RoadFiles

clean:
        rm -f *.class *.html package-list script.js stylesheet.css

We've added a new make target to our example uses, so if you type make html, it will run the command javadoc @classes. The added reference to RoadFiles after the colon on the target line tells make that, in order to make the target html, there must be a file called RoadFiles.

We can also tell make that it needs all the source files to make the html target by listing them on the same line. Such file lists get long, but we can use make variables to clean this up and add additional documentation:

# Makefile
#   input to make command to make things related to road network simulator
#   author Douglas W. Jones

# example uses
#   make clean -- deletes automatically generated files from the directory
#   make html  -- makes all the HTML documentation using javadoc

# all the source files, by category

support = Error.java MyRand.java MyScanner.java Simulator.java

inters = Intersection.java StopLight.java NoStop.java Source.java Sink.java
model = Roads.java $(inters)

main = RoadNetwork.java

RoadFiles = $(support) $(model) $(main)

# make targets

html: classes $(RoadFiles)
        javadoc @classes

clean:
        rm -f *.class *.html package-list script.js stylesheet.css

Note that the variable names we've used for each group of file names documents what the names in that group do. Note also that make uses a really miserable notation for references to named variables, a $ (dollar sign) prefix on a parenthesized variable name. This makes variable names stand out where they are used, but it is hardly convenient or intuitive.

Also note that make has no clean way to wrap over-length lines, so we used named variables to avoid long lines. This is another design error.

All make variables are character strings, typically just lists of file names separated by blanks. Variables must be defined before use.

Now, we can add a make target for the whole application:

RoadNetwork.class: RoadFiles
        javac @classes

Since this is the primary make target for our application, we should list it first. By default, if you just type make, it will make the first target, so by listing this one first, typing make is equivalent to typing make RoadNetwork.class.

Whicever one you use, make is smart enough to check if any of the files on which the target depends have been modified after the date of last modification of the target. If they have been modified, it will run the shell command. If not, it will just output a message saying that the target is already up to date.

Java and Inter-file Dependencies

Let's do a little experiment:

[dwjones@fastx05 ~/project]$ rm -f Error.class
[dwjones@fastx05 ~/project]$ javac Error.java
[dwjones@fastx05 ~/project]$ rm -f Error.java
[dwjones@fastx05 ~/project]$ javac MyScanner.java
[dwjones@fastx05 ~/project]$ sh < RoadNetwork.shar

What did we do above. First, we made certain that the object file Error.class was gone from our project directory. Then we compiled it, creating that file anew. Then we deleted the source file Errror.java before compiling MyScanner.java a file that we know uses class Error, the class defined in Error.java. There were no error messages. MyScanner.java was compiled successfully. Finally, we re-unpacked the .shar file to recover from our deletions.

Why did this work? When you compile any Java source file, the compiler looks for the classes your program uses but does not define as follows:

• First, it imports the .class files named on any import lines, searching the directories on the compiler's search path for those files. The search path leads into the Java library.
• Second, it looks for a .class file in the current directory.
• Third, if it does not find the .class file it needs, it looks for a .java file that names that class and recursively calls the compiler to compile that file.

We've oversimplified things a bit in the above. If there is both a .class file and a .java file for a class the compiler needs, it will recompile the .java file if that file has been edited since the time the .class file was last updated.

In summary, the Java compiler must find all of the inter-file dependencies in a large project, but it does this by reading the source files and hunting around in the directories is has access to. It does not provide any help in documenting these dependencies.

Make and Inter-file Dependencies

At this point, we aren't really using make to document or exploit the inter-file dependencies that exist in our collection of files. To start using these, note that our makefile already provides better documentation of the source files than the RoadFiles file because we've grouped the files by category and used variable names that identify the category. As a result, we can use this as our primary list of file names and let make automatically generate the RoadFiles file:

RoadFiles:
        echo $(RoadFiles) > RoadFiles

This says that to make the file RoadFiles, if this file does not already exist, generate it using the a shell command that creates the file from the list of java source files inside this makefile.

Note that if anyone edits the makefile, they might have edited the code that constructs the variable RoadFiles, so we should really write this:

RoadNetwork.class: RoadFiles
        javac @RoadFiles

RoadFiles: Makefile
        echo $(RoadFiles) > RoadFiles

For each make target, make includes a check to see if the target is up to date, so if the date of last modification of Makefile is more recent than the date of last modification of the file RoadFiles, it will rebuild RoadFiles, and then, if the date of last modification of RoadFiles is after the date of last modification of RoadNetwork.class, then we have to recompile things. On the other hand, if you type make RoadNetwork.class and there have been no edites to any of the prerequisite files, it will not do any work and will announce that everything is up to date.

Since the contents of the file RoadFiles was created from the We can now include RoadFiles in the list of files deleted by make clean, giving us this makefile:

RoadFiles = $(support) $(model) $(main)

# make targets

RoadFiles: Makefile
        echo $(RoadFiles) > RoadFiles

html: RoadFiles
        javadoc @RoadFiles

clean:
        rm -f *.class *.html package-list script.js stylesheet.css RoadFiles

We are still missing something: Make can also do the primary job of deciding which source files need to be recompiled after any change. When we run the program, we run it with java RoadNetwork, which is taken to be a reference to the file RoadNetwork.class, so that is the make target for the application. Make treats the first target as the default, so if you put RoadNetwork.class as the first make target in your list of targets, just typing make becomes equivalent to typing make RoadNetwork.

It is useful to separate the make targets into categories: The primary target, subsidiary make targets needed in order to make the primary target, and secondary targets you may never need to use. That is done in the following version of the makefile:

# Makefile
#   input to make command to make things related to road network simulator
#   author Douglas W. Jones

##########
# all the source files, by category

support = Error.java MyRand.java MyScanner.java Simulator.java

inters = Intersection.java StopLight.java NoStop.java Source.java Sink.java
model = Roads.java $(inters)

main = RoadNetwork.java

RoadFiles = $(support) $(model) $(main)

##########
# primary make target

RoadNetwork.class: RoadFiles
        javac @RoadFiles

# subsidiary make targets

RoadFiles: Makefile
        echo $(RoadFiles) > RoadFiles

##########
# secondary make targets

html: RoadFiles
        javadoc @RoadFiles

clean:
        rm -f *.class *.html RoadFiles package-list script.js stylesheet.css

The first time you type make, it will begin by creating the RoadFiles file and then it will run javac @RoadFiles. If you do nothing but type make again, it will tell you that RoadNetwork.java is up to date and not remake it. If you edit any of the source files and then type make, it will notice that you changed something on which RoadNetwork.java depends and it will recompile everything.

The behavior is not as nice for regenerating the HTML files. The reason is, the make command for target html doesn't create a file with that name, so every time you ask it to make html, it will repeat the job. We can fix this as follows:

index.html: RoadFiles $(RoadFiles)
        javadoc @classes
html:index.html

This explicitly names the primary output of Javadoc so that make can compare the date of last modification of index.html with the dates on all the files it depends on. If you just type make html, there are no shell commands to execute but because it depends on index.html, you asked make to make sure that is up to date.

Makefiles are just as worthy of comments as any other part of a software project. Makefiles have authors, versions and ownership, and they are worth documenting with suggested.

Relationships between Java classes

The Java compiler is designed to be used on an entire project, but we can use a makefile to document the internal structure of our project, using the compiler in a piecemeal way that, as a side effect, provides considerable documentation of the internal structure of our project:

##########
# primary make target

supportClasses = Error.class MyScanner.class Simulator.class

RoadNetwork.class: RoadNetwork.java Road.class $(supportClasses)
        javac RoadNetwork.java

##########
# subsidiary make targets -- support classes

Error.class: Error.java
        javac Error.java

MyScanner.class: MyScanner.java Error.class
        javac MyScanner.java

MyRandom.class: MyRandom.java
        javac MyRandom.java

Simulator.class: Simulator.java
        javac Simulator.java

# subsidiary make targets -- model classes
#    note:  this is a knot of interdependent classes and must be made as a lump

InterSubclasJava = StopLight.java NoStop.java Source.java Sink.java
modelSource = Road.java Intersection.java $(InterSubclasJava)

Road.class: $(modelSource)
        javac $(modelSource)

This explicitly names the primary output of Javadoc so that make can compare the date of last modification of index.html with the dates on all the files it depends on. If you just type make html, there are no shell commands to execute but because it depends on index.html,

With this code, the makefile will force recompilation of, for example, Error.java if the date and time of last change to Error.java was after the data and time of last change to Error.class, and it will not even try to compile Intersection.java until this is done.

The Java compiler duplicates much of this logic internally, but by bringing this out into a makefile, we explicitly document the dependencies between the different source files.

Make is smart enough to understand circular dependencies, so we can actually document groups of interdependent files and Make will not go into any nasty loops. For example, this makefile works:

##########
# primary make target

UtilityClasses = Error.class MyScanner.class Simulator.class
SimulationClasses = Road.class Intersection.class
RoadNetwork.class: RoadNetwork.java $(UtilityClasses) $(SimulationClasses)
        javac RoadNetwork.java

##########
# secondary make target -- simulation model

Road.class: Road.java Intersection.class $(UtilityClasses)
        javac Road.java

InterDepends = Road.class Source.class Sink.class StopLight.class NoStop.class
Intersection.class: Intersection.java $(InterDepends) $(UtilityClasses)

Source.class: Source.java NoStop.class $(UtilityClasses) MyRandom.class
        javac Source.java

Sink.class: Sink.java Intersection.class Road.class $(UtilityClasses)
        javac Sink.java

StopLight.class: StopLight.java Intersection.class Road.class $(UtilityClasses)
        javac StopLight.java

NoStop.class: NoStop.java Intersection.class Road.class $(UtilityClasses)
        javac NoStop.java

# secondary make target -- utility classes

MyScanner.class: MyScanner.java Error.class
        javac MyScanner.java

Error.class: Error.java
        javac Error.java

Random.class: Random.java
        javac Random.java

Simulator.class: Simulator.java
        javac Simulator.java

Testing

We can easily add test scripts to the makefile as secondary make targets:

test: RoadNetwork.java exampleAB example
        java RoadNetwork exampleAB
        java RoadNetwork example

Now, we can run all of our tests by typing make tests. If we'd followed through on our original development plan by creating and maintaining test scripts, this could launch those test scripts instead of directly running the tests.

A Manual in the makefile header

Makefiles can get large. When this happens, it's a good idea to write something of a manual in the header of the makefile, before any content. For example, for the file described above, we might begin with this:

# The following primary make commands are supported
#    make                    -- equivalent to make RoadNetwork.class
#    make RoadNetwork.class  -- makes RoadNetwork.class and all subsidiaries
#
# The following secondary make commands are supported
#    make test               -- run a demo showing how the program works
#    make html               -- makes web site of internal documentation
#    make clean              -- deletes all files created by any of the above

With this job completed, the README file can say something like: "For instructions on installing, building and testing this program, see the Makefile. The Makefile also documents the relationships between the different source files in this project." this suggestion may be a bit too terse. It may be better to have the README file explicitly state something like:

To compile and build this program, use this shell command:

```
        make
```

This implicitly reads Makefile to find the instructions for compilation.
For other make options, read the instructions at the head of the makefile.
Note that the makefile also documents the relationships between the vaious
components of this program.