Here is a program that reads a
collection of ten files, those that are specified
here , and extracts words
from these files. For our purposes, a word is defined as something
that is made up purely of characters. The program considers only words
that are more than four letters long. It then keeps track of the
number of occurances of each such word over the entire collection.
Finally, it prints the words that occur more than 20 times. This
program will be a useful starting point for this homework, to be
The basic task is to measure the similarity between any two files
in our collection. To do this, we will need a suitable universe
of words. This will consist of all words in the collection that
are (a) more than four letters long, (b) don't occur more than
20 times overall, and (c) don't occur in more than 7 files in the
collection. Now we constructor a vector (in the mathematical sense)
corresponding to each file. The vector will have as many coordinates
as words in the universe -- so there is one coordinate for each word
in the universe. If a word occurs in the file, the corresponding
coordinate is 1, otherwise it is 0.
Let us give an example: suppose the universe consists of the five
words: apple, grapes, banana, doctor, program. Suppose
file1 contains: apple, banana, program. Then the vector
for file1 is (1,0,1,0,1).
We need to normalize each of the vectors so that it has unit length.
So each coordinate in the above vector gets divided by the square
root of 3.
The similarity of two files is defined to be the scalar
product of the corresponding two vectors. The scalar product
of two vectors is obtained by multiplying corresponding components
and adding. For example, the scalar product of (2,1,3) and (0,5,6)
is 2 * 0 + 1 * 5 + 3 * 6.
Your task is to write a program that prints the names of the two
files with the highest similarity among the files in the collection, and
the names of the two files with the lowest similarity. The collection is
the same as the one processed in the above program, and consists of
the files named here . You'll need to
download the stuff in the directory, which is available in zipped
form here .