We, the authors of this book, have often heard our colleagues ask for a program to help them find duplicate files on their system. Let’s write something to do exactly that. We’ll pass the names of the directories we want to search on the command line, along with an optional switch to determine whether we want to recurse into subdirectories or not. In the first instance, we’ll do a very basic check for similarity based on filenames and sizes, as these are relatively cheap options. Example 11-1 shows our Main function.

static void Main(string[] args)
{
    bool recurseIntoSubdirectories = false;

    if (args.Length < 1)
    {
        ShowUsage();
        return;
    }

    int firstDirectoryIndex = 0;

    if (args.Length > 1)
    {
        // see if we're being asked to recurse
        if (args[0] == "/sub")
        {
            if (args.Length < 2)
            {
                ShowUsage();
                return;
            }
            recurseIntoSubdirectories = true;
            firstDirectoryIndex = 1;
        }
    }

    // Get list of directories from command line.
    var directoriesToSearch = args.Skip(firstDirectoryIndex);

    List<FileNameGroup> filesGroupedByName =
        InspectDirectories(recurseIntoSubdirectories, directoriesToSearch);

    DisplayMatches(filesGroupedByName);

    Console.ReadKey();
}

The basic structure is pretty straightforward. First we inspect the command-line arguments to work out which directories we’re searching. Then we call InspectDirectories (shown later) to build a list of all the files in those directories. This groups the files by filename (without the full path) because we do not consider two files to be duplicates if they have different names. Finally, we pass this list to DisplayMatches, which displays any potential matches in the files we have found. DisplayMatches refines our test for duplicates further—it considers two files with the same name to be duplicates only if they have the same size. (That’s not foolproof, of course, but it’s surprisingly effective, and we will refine it further later in the chapter.)

Let’s look at each of these steps in more detail.

The code that parses the command-line arguments does a quick check to see that we’ve provided at least one command-line argument (in addition to the /sub switch if present) and we print out some usage instructions if not, using the method shown in Example 11-2.

private static void ShowUsage()
{
    Console.WriteLine("Find duplicate files");
    Console.WriteLine("====================");
    Console.WriteLine(
        "Looks for possible duplicate files in one or more directories");
    Console.WriteLine();
    Console.WriteLine(
        "Usage: findduplicatefiles [/sub] DirectoryName [DirectoryName] ...");
    Console.WriteLine("/sub - recurse into subdirectories");
    Console.ReadKey();
}

The next step is to build a list of files grouped by name. We define a couple of classes for this, shown in Example 11-3. We create a FileNameGroup object for each distinct filename. Each FileNameGroup contains a nested list of FileDetails, providing the full path of each file that has that name, and also the size of that file.

class FileNameGroup
{
    public string FileNameWithoutPath { get; set; }
    public List<FileDetails> FilesWithThisName { get; set; }
}

class FileDetails
{
    public string FilePath { get; set; }
    public long FileSize { get; set; }
}

For example, suppose the program searches two folders, c:One and c:Two, and suppose both of those folders contain a file called Readme.txt. Our list will contain a FileNameGroup whose FileNameWithoutPath is Readme.txt. Its nested FilesWithThisName list will contain two FileDetails entries, one with a FilePath of c:OneReadme.txt and the other with c:TwoReadme.txt. (And each FileDetails will contain the size of the relevant file in FileSize. If these two files really are copies of the same file, their sizes will, of course, be the same.)

We build these lists in the InspectDirectories method, which is shown in Example 11-4. This contains the meat of the program, because this is where we search the specified directories for files. Quite a lot of the code is concerned with the logic of the program, but this is also where we start to use some of the file APIs.

private static List<FileNameGroup> InspectDirectories(
    bool recurseIntoSubdirectories,
    IEnumerable<string> directoriesToSearch)
{
    var searchOption = recurseIntoSubdirectories ?
        SearchOption.AllDirectories : SearchOption.TopDirectoryOnly;

    // Get the path of every file in every directory we're searching.
    var allFilePaths = from directory in directoriesToSearch
                       from file in Directory.GetFiles(directory, "*.*",
                                                        searchOption)
                       select file;

    // Group the files by local filename (i.e. the filename without the
    // containing path), and for each filename, build a list containing the
    // details for every file that has that filename.
    var fileNameGroups = from filePath in allFilePaths
                         let fileNameWithoutPath = Path.GetFileName(filePath)
                         group filePath by fileNameWithoutPath into nameGroup
                         select new FileNameGroup
                         {
                             FileNameWithoutPath = nameGroup.Key,
                             FilesWithThisName =
                              (from filePath in nameGroup
                               let info = new FileInfo(filePath)
                               select new FileDetails
                               {
                                   FilePath = filePath,
                                   FileSize = info.Length
                               }).ToList()
                         };

    return fileNameGroups.ToList();
}

To get it to compile, you’ll need to add:

using System.IO;

The parts of Example 11-4 that use the System.IO namespace to work with files and directories have been highlighted. We’ll start by looking at the use of the Directory class.

Our InspectDirectories method calls the static GetFiles method on the Directory class to find the files we’re interested in. Example 11-5 shows the relevant code.

var searchOption = recurseIntoSubdirectories ?
    SearchOption.AllDirectories : SearchOption.TopDirectoryOnly;

// Get the path of every file in every directory we're searching.
var allFilePaths = from directory in directoriesToSearch
                   from file in Directory.GetFiles(directory, "*.*",
                                                        searchOption)
                   select file;

The overload of GetFiles we’re calling takes the directory we’d like to search, a filter (in the standard command-line form), and a value from the SearchOption enumeration, which determines whether to recurse down through all the subfolders.

The GetFiles method returns the full path for each file concerned, but when it comes to finding matches, we just want the filename. We can use the Path class to get the filename from the full path.

The Path class provides methods for manipulating strings containing file paths. Imagine we have the path c:directory1directory2MyFile.txt. Table 11-1 shows you how you can slice that with various different Path methods.

What if we use a network path? Table 11-2 shows the results of the same methods when applied to this path:

\MyPCShare1directory2MyFile.txt

Notice how the path root includes the network hostname and the share name.

What happens if we don’t use a full path, but one relative to the current directory? And what’s the current directory anyway?

Directory.SetCurrentDirectory(@"c:");

var fileNameGroups = from filePath in allFilePaths
                     let fileNameWithoutPath = Path.GetFileName(filePath)
                     group filePath by fileNameWithoutPath into nameGroup
                     select ...

The various functions from the System.IO classes we’ve dealt with so far have all been static, but when it comes to retrieving information such as file size, we have to create an instance of a FileInfo object, passing its constructor the path of the file we’re interested in. That path can be either an absolute path like the ones we’ve seen already, or a path relative to the current working directory. FileInfo has a lot of overlapping functionality with other classes. For example, it provides a few helpers similar to Path to get details of the directory, filename, and extension.

However, the only method we’re really interested in for our example is its Length property, which tells us the size of the file. Every other member on FileInfo has a functional equivalent on other classes in the framework. Even Length is duplicated on the stream classes we’ll come to later, but it is simpler for us to use FileInfo if we don’t intend to open the file itself.

We use FileInfo in the final part of InspectDirectories, to put the file size into the per-file details. Example 11-8 shows the relevant excerpt from Example 11-4.

...
select new FileNameGroup
{
    FileNameWithoutPath = nameGroup.Key,
    FilesWithThisName =
     (from filePath in nameGroup
      let info = new FileInfo(filePath)
      select new FileDetails
      {
          FilePath = filePath,
          FileSize = info.Length
      }).ToList()
};

We’re now only one method short of a sort-of-useful program, and that’s the one that trawls through this information to find and display matches: DisplayMatches, which is shown in Example 11-9.

private static void DisplayMatches(
    IEnumerable<FileNameGroup> filesGroupedByName)
{
    var groupsWithMoreThanOneFile = from nameGroup in filesGroupedByName
                                    where nameGroup.FilesWithThisName.Count > 1
                                    select nameGroup;

    foreach (var fileNameGroup in groupsWithMoreThanOneFile)
    {
        // Group the matches by the file size, then select those
        // with more than 1 file of that size.
        var matchesBySize = from file in fileNameGroup.FilesWithThisName
                            group file by file.FileSize into sizeGroup
                            where sizeGroup.Count() > 1
                            select sizeGroup;

        foreach (var matchedBySize in matchesBySize)
        {
            string fileNameAndSize = string.Format("{0} ({1} bytes)",
                fileNameGroup.FileNameWithoutPath, matchedBySize.Key);
            WriteWithUnderlines(fileNameAndSize);
            // Show each of the directories containing this file
            foreach (var file in matchedBySize)
            {
                Console.WriteLine(Path.GetDirectoryName(file.FilePath));
            }
            Console.WriteLine();
        }
    }
}

private static void WriteWithUnderlines(string text)
{
    Console.WriteLine(text);
    Console.WriteLine(new string('-', text.Length));
}

We start with a LINQ query that looks for the filenames that crop up in more than one folder, because those are the only candidates for being duplicates. We iterate through each such name with a foreach loop. Inside that loop, we run another LINQ query that groups the files of that name by size—see the first emphasized lines in Example 11-9. If InspectDirectories discovered three files called Program.cs, for example, and two of them were 278 bytes long while the other was 894 bytes long, this group clause would separate those three files into two groups. The where clause in the same query removes any groups that contain only one file.

So the matchesBySize variable refers to a query that returns a group for each set of two or more files that have the same size (and because we’re inside a loop that iterates through the names, we already know they have the same name). Those are our duplicate candidates. We then write out the filename and size (and an underline separator of the same length). Finally, we write out each file location containing candidate matches using Path.GetDirectoryName.

If we compile and run that lot, we’ll see the following output:

Find duplicate files
====================
Looks for possible duplicate files in one or more directories

Usage: findduplicatefiles [/sub] DirectoryName [DirectoryName] ...
/sub - recurse into subdirectories

We haven’t given it anywhere to look! How are we going to test our application? Well, we could provide it with some command-line parameters. If you open the project properties and switch to the Debug tab, you’ll see a place where you can add command-line arguments (see Figure 11-1).

Figure 11-1. Setting command-line arguments

However, we could do a bit better for test purposes. Example 11-10 shows a modified Main that supports a new /test command-line switch, which we can use to create test files and exercise the function.

static void Main(string[] args)
{
    bool recurseIntoSubdirectories = false;

    if (args.Length < 1)
    {
        ShowUsage();
        return;
    }

    int firstDirectoryIndex = 0;
    IEnumerable<string> directoriesToSearch = null;
    bool testDirectoriesMade = false;

    try
    {
        // Check to see if we are running in test mode
        if (args.Length == 1 && args[0] == "/test")
        {
            directoriesToSearch = MakeTestDirectories();
            testDirectoriesMade = true;
            recurseIntoSubdirectories = true;
        }
        else
        {
            if (args.Length > 1)
            {
                // see if we're being asked to recurse
                if (args[0] == "/sub")
                {
                    if (args.Length < 2)
                    {
                        ShowUsage();
                        return;
                    }
                    recurseIntoSubdirectories = true;
                    firstDirectoryIndex = 1;
                }
            }

            // Get list of directories from command line.
            directoriesToSearch = args.Skip(firstDirectoryIndex);
        }

        List<FileNameGroup> filesGroupedByName =
            InspectDirectories(recurseIntoSubdirectories, directoriesToSearch);

        DisplayMatches(filesGroupedByName);
        Console.ReadKey();
    }
    finally
    {
        if( testDirectoriesMade )
        {
            CleanupTestDirectories(directoriesToSearch);
        }
    }

}

In order to operate in test mode, we’ve added an alternative way to initialize the variable that holds the list of directories (directoriesToSearch). The original code, which initializes it from the command-line arguments (skipping over the /sub switch if present), is still present. However, if we find the /test switch, we initialize it to point at some test directories we’re going to create (in the MakeTestDirectories method). The rest of the code can then be left as it was (to avoid running some completely different program in our test mode). Finally, we add a bit of cleanup code at the end to remove any test directories if we created them.

So, how are we going to implement MakeTestDirectories? We want to create some temporary files, and write some content into them to exercise the various matching possibilities.

A quick look at Path reveals the GetTempFileName method. This creates a file of zero length in a directory dedicated to temporary files, and returns the path to that file.

Let’s create another test console application, just to try out that method. We can do that by adding the following to our main function:

string fileName = Path.GetTempFileName();
// Display the filename
Console.WriteLine(fileName);
// And wait for some input
Console.ReadKey();

But wait! If we just compile and run that, we’ll leave the file we created behind on the system. We should make sure we delete it again when we’re done. There’s nothing special about a temporary file. We create it in an unusual way, and it ends up in a particular place, but once it has been created, it’s just like any other file in the filesystem. So, we can delete it the same way we’d delete any other file.

The System.IO namespace provides the File class, which offers various methods for doing things with files. Deleting is particularly simple: we just use the static Delete method, as Example 11-11 shows.

string fileName = Path.GetTempFileName();
try
{
    // Use the file
    // ...
    // Display the filename
    Console.WriteLine(fileName);
    // And wait for some input
    Console.ReadKey();
}
finally
{
    // Then clean it up
    File.Delete(fileName);
}

Notice that we’ve wrapped the code in which we (could) manipulate the file further in a try block, and deleted it in a finally block. This ensures that whatever happens, we’ll always attempt to clean up after ourselves.

If you compile and run this test project now, you’ll see some output like this:

C:UsersyourusernameAppDataLocalTemp	mpCA8F.tmp

The exact text will depend on your operating system version, your username, and (of course) the random filename that was created for you. If you browse to that path, you will see a zero-length file of that name.

If you then press a key, allowing Console.ReadKey to return, it will drop through to the finally block, where we delete the temporary file, using the static Delete method on the File class.

There are lots of scenarios where this sort of temporary file creation is just fine, but it doesn’t really suit our example application’s needs. We want to create multiple temporary files, in multiple different directories. GetTempFileName doesn’t really do the job for us.

If we look at Path again, though, there’s another likely looking method: GetRandomFileName. This returns a random string of characters that can be used as either a file or a directory name. It uses a cryptographically strong random number generator (which can be useful in some security-conscious scenarios), and is statistically likely to produce a unique name, thus avoiding clashes. Unlike GetTempFileName it doesn’t actually create the file (or directory); that’s up to us.

If you run the code in Example 11-12:

Console.WriteLine(Path.GetRandomFileName());

you’ll see output similar to this:

xnicz3rs.juc

(Obviously, the actual characters you see will, hopefully, be different, or the statistical uniqueness isn’t all that unique!)

So, we can use that method to produce our test file and directory names. But where are we going to put the files? Perhaps one of the various “well-known folders” Windows offers would suit our needs.

Most operating systems have a bunch of well-known filesystem locations, and Windows is no exception. There are designated folders for things like the current user’s documents, pictures, or desktop; the program files directory where applications are installed; and the system folder.

The .NET Framework provides a class called Environment that provides information about the world our program runs in. Its static method GetFolderPath is the one that interests us right now, because it will return the path of various well-known folders. We pass it one of the Environment.SpecialFolder enumeration values. Example 11-13 retrieves the location of one of the folders in which applications can store per-user data.

string path = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData);

Table 11-4 lists all of the well-known folders that GetFolderPath can return, and the location they give on the installed copy of Windows 7 (64-bit) belonging to one of the authors.

So, we need to choose a path in which our current user is likely to have permission to create/read/write and delete files and directories. It doesn’t have to be one that the user can see under normal circumstances. In fact, we’re going to create files with extensions that are not bound to any applications and we should not do that in a place that’s visible to the user if we want our application to be a good Windows citizen.

There are two candidates for this in Table 11-4: LocalApplicationData and ApplicationData. Both of these offer places for applications to store files that the user wouldn’t normally see. (Of course, users can find these folders if they look hard enough. The goal here is to avoid putting our temporary test files in the same folders as the user’s documents.)

The difference between these two folders is that if the user has a roaming profile, files in the latter folder will be copied around the network as they move from one machine to another, while files in the former folder remain on the machine on which they were created. We’re building temporary files for test purposes, so LocalApplicationData looks like the right choice.

So, let’s return to our demo application, and start to implement the MakeTestDirectories method. The first thing we need to do is to create a few test directories. Example 11-14 contains some code to do that.

private static string[] MakeTestDirectories()
{
    string localApplicationData = Path.Combine(
        Environment.GetFolderPath(
            Environment.SpecialFolder.LocalApplicationData),
        @"Programming CSharpFindDuplicates");

    // Let's make three test directories
    var directories = new string[3];
    for (int i = 0; i < directories.Length; ++i)
    {
        string directory = Path.GetRandomFileName();
        // Combine the local application data with the
        // new random file/directory name
        string fullPath = Path.Combine(localApplicationData, directory);
        // And create the directory
        Directory.CreateDirectory(fullPath);
        directories[i] = fullPath;
        Console.WriteLine(fullPath);
    }
    return directories;
}

First, we use the GetFolderPath method to get the LocalApplicationData path. But we don’t want to work directly in that folder—applications are meant to create their own folders underneath this. Normally you’d create a folder named either for your company or for your organization, and then an application-specific folder inside that—we’ve used Programming CSharp as the organization name here, and FindDuplicates as the application name. We then use a for loop to create three directories with random names inside that. To create these new directories, we’ve used a couple of new methods: Path.Combine and Directory.CreateDirectory.

If you’ve written any code that manipulates paths before, you’ll have come across the leading/trailing slash dilemma. Does your path fragment have one or not? You also need to know whether the path fragment you’re going to append really is a relative path—are there circumstances under which you might need to deal with a fully qualified path instead? Path.Combine does away with all that anxiety. Not only will it check all those things for you and do the right thing, but it will even check that your paths contain only valid path characters.

Table 11-5 contains some example paths, and the result of combining them with Path.Combine.

The last entry in that table is particularly interesting: notice that the second path is absolute, and so the combined path is “optimized” to just that second path.

In our case, Example 11-14 combines the well-known folder with a subfolder name to get a folder location specific to this example. And then it combines that with our new temporary folder names, ready for creation.

Directory.CreateDirectory is very straightforward: it does exactly what its name suggests. In fact, it will create any directories in the whole path that do not already exist, so you can create a deep hierarchy with a single call. (You’ll notice that Example 11-14 didn’t bother to create the Programming CSharpFindDuplicates folder—those will get created automatically the first time we run as a result of creating the temporary folders inside them.) A side effect of this is that it is safe to call it if all of the directories in the path already exist—it will just do nothing.

In addition to the overload we’ve used, there’s a second which also takes a DirectorySecurity parameter:

Directory.CreateDirectory(string path, DirectorySecurity directorySecurity)

The DirectorySecurity class allows you to specify filesystem access controls with a relatively simple programming model. If you’ve tried using the Win32 ACL APIs, you’ll know that it is a nightmare of GUIDs, SSIDs, and lists sensitive to item ordering. This model does away with much of the complexity.

Let’s extend our create function to make sure that only our current user has read/write/modify permissions on these directories. Example 11-15 modifies the previous example by explicitly granting the current user full control of the newly created folders. The new or changed lines are highlighted.

private static string[] MakeTestDirectories()
{
    string localApplicationData = Path.Combine(
        Environment.GetFolderPath(
            Environment.SpecialFolder.LocalApplicationData),
        @"Programming CSharpFindDuplicates");

    // Get the name of the logged in user
    string userName = WindowsIdentity.GetCurrent().Name;
    // Make the access control rule
    FileSystemAccessRule fsarAllow =
        new FileSystemAccessRule(
            userName,
            FileSystemRights.FullControl,
            AccessControlType.Allow);
    DirectorySecurity ds = new DirectorySecurity();
    ds.AddAccessRule(fsarAllow);

    // Let's make three test directories
    var directories = new string[3];
    for (int i = 0; i < directories.Length; ++i)
    {
        string directory = Path.GetRandomFileName();
        // Combine the local application data with the
        // new random file/directory name
        string fullPath = Path.Combine(localApplicationData, directory);

        // And create the directory
        Directory.CreateDirectory(fullPath, ds);

        directories[i] = fullPath;
        Console.WriteLine(fullPath);
    }
    return directories;
}

You’ll need to add a couple of using directives to the top of the file before you can compile this code:

using System.Security.AccessControl;
using System.Security.Principal;

What do these changes do? First, we make use of a type called WindowsIdentity to find the current user, and fish out its name. If you happen to want to specify the name explicitly, rather than get the current user programmatically, you can do so (e.g., MYDOMAINSomeUserId).

Then, we create a FileSystemAccessRule, passing it the username, the FileSystemRights we want to set, and a value from the AccessControlType enumeration which determines whether we are allowing or denying those rights.

If you take a look at the FileSystemRights enumeration in MSDN, you should recognize the options from the Windows security permissions dialog in the shell. You can combine the individual values (as it is a Flags enumeration), or use one of the precanned sets as we have here.

If you compile this application, and modify the debug settings to pass just the /test switch as the only command-line argument, when you run it you’ll see output similar to the following (but with your user ID, and some different random directory names):

C:UsersyourIdAppDataLocalProgramming CSharpFindDuplicatesyzw0iw3p.ysq
C:UsersyourIdAppDataLocalProgramming CSharpFindDuplicatesqke5k2ql.5et
C:UsersyourIdAppDataLocalProgramming CSharpFindDuplicates5hkhspqa.osc

If we take a look at the folder in Explorer, you should see your new directories (something like Figure 11-2).

If you right-click on one of these and choose Properties, then examine the Security tab, you should see something like Figure 11-3.

Notice how the only user with permissions on this directory is the currently logged on user (in this case ian, on a domain called idg.interact). All of the usual inherited permissions have been overridden. Rather than the regular read/modify/write checkboxes, we’ve apparently got special permissions. This is because we set them explicitly in the code.

Figure 11-2. Newly created folders

Figure 11-3. Permissions on the new directory

We can have a look at that in more detail if we click the Advanced button, and switch to the Effective Permissions tab. Click the Select button to pick a user (see Figure 11-4). First, let’s look at the effective permissions for the local administrator (this is probably MachineNameAdministrator, unless you’ve changed your default administrator name to try to make things slightly harder for an attacker).

Figure 11-4. Selecting a user

If you click OK, you’ll see the effective permissions for Administrator on that folder (Figure 11-5).

You can scroll the scroll bar to prove it for yourself, but you can see that even Administrator cannot actually access your folder! (This is not, of course, strictly true. Administrators can take ownership of the folder and mess with the permissions themselves, but they cannot access the folder without changing the permissions first.) Try again with your own user ID. You will see results similar to Figure 11-6—we have full control. Scroll the list and you’ll see that everything is ticked.

What if we wanted “not quite” full control? Say we wanted to deny the ability to write extended attributes to the file. Well, we can update our code and add a second FileSystemAccessRule. Example 11-16 shows the additional code required.

private static string[] MakeTestDirectories()
{
    // ...
    FileSystemAccessRule fsarAllow =
        new FileSystemAccessRule(
            userName,
            FileSystemRights.FullControl,
            AccessControlType.Allow);
    ds.AddAccessRule(fsarAllow);

    FileSystemAccessRule fsarDeny =
        new FileSystemAccessRule(
            userName,
            FileSystemRights.WriteExtendedAttributes,
            AccessControlType.Deny);
    ds.AddAccessRule(fsarDeny);

    // ...
}

Notice that we’re specifying AccessControlType.Deny.

Before you compile and run this, delete the folders you created with the last run, using Explorer—we’ll write some code to do that automatically in a minute, because it will get very boring very quickly!

You should see very similar output to last time (just with some new directory names):

C:UsersyourIdAppDataLocalProgramming CSharpFindDuplicatesslhwbtgo.sop
C:UsersyourIdAppDataLocalProgramming CSharpFindDuplicatessfndkgn.ucm
C:UsersyourIdAppDataLocalProgramming CSharpFindDuplicates	ayf1uvg.y4y

Figure 11-5. Effective permissions for Administrator on the new folder

Figure 11-6. Effective permissions for the current user on the new folder

If you look at the permissions, you will now see both the Allow and the new Deny entries (Figure 11-7).

Figure 11-7. Permissions now that we’ve denied write extended attributes

As a double-check, take a look at the effective permissions for your current user (see Figure 11-8).

Figure 11-8. Effective permissions with write extended attributes denied

In Figure 11-8 you can see that we’ve no longer got Full control, because we’ve been specifically denied Write extended attributes. Of course, we could always give that permission back to ourselves, because we’ve been allowed Change permissions, but that’s not the point!

OK, delete those new directories using Explorer, and we’ll write some code to clean up after ourselves. We need to delete the directories we’ve just created, by implementing our CleanupTestDirectories method.

You’re probably ahead of us by now. Yes, we can delete a directory using Directory.Delete, as Example 11-17 shows.

private static void CleanupTestDirectories(IEnumerable<string> directories)
{
    foreach (var directory in directories)
    {
        Directory.Delete(directory);
    }
}

We’re just iterating through the set of new directories we stashed away earlier, deleting them.

OK, we’ve got our test directories. We’d now like to create some test files to use. Just before we return from MakeTestDirectories, let’s add a call to a new method to create our files, as Example 11-18 shows.

...
CreateTestFiles(directories);
return directories;

Example 11-19 shows that method.

private static void CreateTestFiles(IEnumerable<string> directories)
{
    string fileForAllDirectories = "SameNameAndContent.txt";
    string fileSameInAllButDifferentSizes = "SameNameDifferentSize.txt";

    int directoryIndex = 0;
    // Let's create a distinct file that appears in each directory
    foreach (string directory in directories)
    {
        directoryIndex++;

        // Create the distinct file for this directory
        string filename = Path.GetRandomFileName();
        string fullPath = Path.Combine(directory, filename);
        CreateFile(fullPath, "Example content 1");

        // And now the one that is in all directories, with the same content
        fullPath = Path.Combine(directory, fileForAllDirectories);
        CreateFile(fullPath, "Found in all directories");

        // And now the one that has the same name in
        // all directories, but with different sizes
        fullPath = Path.Combine(directory, fileSameInAllButDifferentSizes);

        StringBuilder builder = new StringBuilder();
        builder.AppendLine("Now with");
        builder.AppendLine(new string('x', directoryIndex));
        CreateFile(fullPath, builder.ToString());
    }
}

As you can see, we’re running through the directories, and creating three files in each. The first has a different, randomly generated filename in each directory, and remember, our application only considers files with the same names as being possible duplicates, so we expect the first file we add to each directory to be considered unique. The second file has the same filename and content (so they will all be the same size) in every folder. The third file has the same name every time, but its content varies in length.

Well, we can’t put off the moment any longer; we’re going to have to create a file, and write some content into it. There are lots and lots and lots (and lots) of different ways of doing that with the .NET Framework, so how do we go about picking one?

Our first consideration should always be to “keep it simple,” and use the most convenient method for the job. So, what is the job? We need to create a file, and write some text into it. File.WriteAllText looks like a good place to start.

private static void CreateFile(string fullPath, string contents)
{
    File.WriteAllText(fullPath, contents);
}

private static void CreateFile(string fullPath, string p)
{
    using (StreamWriter writer = File.CreateText(fullPath))
    {
        // Use the stream writer here
    }
}

return new StreamWriter(fullPath, false);

private static void CreateFile(string fullPath, string p)
{
    using (StreamWriter writer = File.CreateText(fullPath))
    {
        writer.Write(p);
    }
}

C:UsersmwaAppDataLocalup022gsm.241
C:UsersmwaAppDataLocalgdovysqk.cqn
C:UsersmwaAppDataLocalxyhazu3n.4pw
SameNameAndContent.txt
----------------------
C:UsersmwaAppDataLocalup022gsm.241
C:UsersmwaAppDataLocalgdovysqk.cqn
C:UsersmwaAppDataLocalxyhazu3n.4pw

Directory.Delete(directory, true);

Exceptions related to file and stream operations fall into three broad categories:

The first category can, of course, be dealt with as normal—if they occur (as we discussed in Chapter 6) there is usually some bug or unexpected usage that you need to deal with.

The other two are slightly more interesting cases. We should expect problems with file I/O. Files and directories are (mostly) system-wide shared resources. This means that anyone can be doing something with them while you are trying to use them. As fast as you’re creating them, some other process might be deleting them. Or writing to them; or locking them so that you can’t touch them; or altering the permissions on them so that you can’t see them anymore. You might be working with files on a network share, in which case different computers may be messing with the files, or you might lose connectivity partway through working with a file.

This “global” nature of files also means that you have to deal with concurrency problems. Consider this piece of code, for example, that makes use of the (almost totally redundant) method File.Exists, shown in Example 11-23, which determines whether a file exists.

if (File.Exists("SomeFile.txt"))
{
    // Play with the file
}

Is it safe to play with the file in there, on the assumption that it exists?

No.

In another process, even from another machine if the directory is shared, someone could nip in and delete the file or lock it, or do something even more nefarious (like substitute it for something else). Or the user might have closed the lid of his laptop just after the method returns, and may well be in a different continent by the time he brings it out of sleep mode, at which point you won’t necessarily have access to the same network shares that seemed to be visible just one line of code ago.

So you have to code extremely defensively, and expect exceptions in your I/O code, even if you checked that everything looked OK before you started your work.

Unlike most exceptions, though, abandoning the operation is not always the best choice. You often see transient problems, like a USB drive being temporarily unavailable, for example, or a network glitch temporarily hiding a share from us, or aborting a file copy operation. (Transient network problems are particularly common after a laptop resumes from suspend—it can take a few seconds to get back on the network, or maybe even minutes if the user is in a hotel and has to sign up for an Internet connection before connecting back to the office VPN. Abandoning the user’s data is not a user-friendly response to this situation.)

When an I/O problem occurs, the framework throws one of several exceptions derived from IOException (or, as we’ve already seen, IOException itself) listed here:

If you are doing anything with I/O operations, you will need to think about most, if not all, of these exceptions, deciding where to catch them and what to do when they occur.

Let’s look back at our example again, and see what we want to do with any exceptions that might occur. As a first pass, we could just wrap our main loop in a try/catch block, as Example 11-24 does. Since our application’s only job is to report its findings, we’ll just display a message if we encounter a problem.

try
{
    List<FileNameGroup> filesGroupedByName =
        InspectDirectories(recurseIntoSubdirectories, directoriesToSearch);

    DisplayMatches(foundFiles);
    Console.ReadKey();
}
catch (PathTooLongException ptlx)
{
    Console.WriteLine("The specified path was too long");
    Console.WriteLine(ptlx.Message);
}
catch (DirectoryNotFoundException dnfx)
{
    Console.WriteLine("The specified directory was not found");
    Console.WriteLine(dnfx.Message);
}
catch (IOException iox)
{
    Console.WriteLine(iox.Message);
}
catch (UnauthorizedAccessException uax)
{
    Console.WriteLine("You do not have permission to access this directory.");
    Console.WriteLine(uax.Message);
}
catch (ArgumentException ax)
{
    Console.WriteLine("The path provided was not valid.");
    Console.WriteLine(ax.Message);
}
finally
{
    if (testDirectoriesMade)
    {
        CleanupTestDirectories(directoriesToSearch);
    }
}

We’ve decided to provide specialized handling for the PathTooLongException and DirectoryNotFoundException exceptions, as well as generic handling for IOException (which, of course, we have to catch after the exceptions derived from it).

In addition to those IOException-derived types, we’ve also caught UnauthorizedAccessException. This is a security exception, rather than an I/O exception, and so it derives from a different base (SystemException). It is thrown if the user does not have permission to access the directory concerned.

Let’s see that in operation, by creating an additional test directory and denying ourselves access to it. Example 11-25 shows a function to create a directory where we deny ourselves the ListDirectory permission.

private static string CreateDeniedDirectory(string parentPath)
{
    string deniedDirectory = Path.GetRandomFileName();
    string fullDeniedPath = Path.Combine(parentPath, deniedDirectory);
    string userName = WindowsIdentity.GetCurrent().Name;
    DirectorySecurity ds = new DirectorySecurity();
    FileSystemAccessRule fsarDeny =
        new FileSystemAccessRule(
            userName,
            FileSystemRights.ListDirectory,
            AccessControlType.Deny);
    ds.AddAccessRule(fsarDeny);

    Directory.CreateDirectory(fullDeniedPath, ds);
    return fullDeniedPath;
}

We can call it from our MakeTestDirectories method, as Example 11-26 shows (along with suitable modifications to the code to accommodate the extra directory).

private static string[] MakeTestDirectories()
{
    // ...
    // Let's make three test directories
    // and leave space for a fourth to test access denied behavior
    var directories = new string[4];
    for (int i = 0; i < directories.Length - 1; ++i)
    {
        ... as before ...
    }

    CreateTestFiles(directories.Take(3));

    directories[3] = CreateDeniedDirectory(localApplicationData);

    return directories;
}

But hold on a moment, before you build and run this. If we’ve denied ourselves permission to look at that directory, how are we going to delete it again in our cleanup code? Fortunately, because we own the directory that we created, we can modify the permissions again when we clean up.

private static void AllowAccess(string directory)
{
    DirectorySecurity ds = Directory.GetAccessControl(directory);

    string userName = WindowsIdentity.GetCurrent().Name;

    // Remove the deny rule
    FileSystemAccessRule fsarDeny =
        new FileSystemAccessRule(
            userName,
            FileSystemRights.ListDirectory,
            AccessControlType.Deny);
    ds.RemoveAccessRuleSpecific(fsarDeny);

    // And add an allow rule
    FileSystemAccessRule fsarAllow =
        new FileSystemAccessRule(
            userName,
            FileSystemRights.FullControl,
            AccessControlType.Allow);
    ds.AddAccessRule(fsarAllow);

    Directory.SetAccessControl(directory, ds);
}

C:UsersmwaAppDataLocalufmnho4z.h5p
C:UsersmwaAppDataLocal5chw4maf.xyu
C:UsersmwaAppDataLocals1ydovhu.0wk
You do not have permission to access this directory.
Access to the path 'C:UsersmwaAppDataLocalyjijkza.3cj' is denied.

var allFilePaths = from directory in directoriesToSearch
                   from file in Directory.GetFiles(directory, "*.*",
                                                   searchOption)
                   select file;

private static IEnumerable<string> GetDirectoryFiles(
    string directory, SearchOption searchOption)
{
    try
    {
        return Directory.GetFiles(directory, "*.*", searchOption);
    }
    catch (DirectoryNotFoundException dnfx)
    {
        Console.WriteLine("Warning: The specified directory was not found");
        Console.WriteLine(dnfx.Message);
    }
    catch (UnauthorizedAccessException uax)
    {
        Console.WriteLine(
            "Warning: You do not have permission to access this directory.");
        Console.WriteLine(uax.Message);
    }

    return Enumerable.Empty<string>();
}

var allFilePaths = from directory in directoriesToSearch
                   from file in GetDirectoryFiles(directory,
                                                  searchOption)
                   select file;

private static IEnumerable<FileDetails> GetDetails(IEnumerable<string> paths)
{
    foreach (string filePath in paths)
    {
        FileDetails details = null;
        try
        {
            FileInfo info = new FileInfo(filePath);
            details = new FileDetails
            {
                FilePath = filePath,
                FileSize = info.Length
            };
        }
        catch (FileNotFoundException fnfx)
        {
            Console.WriteLine("Warning: The specified file was not found");
            Console.WriteLine(fnfx.Message);
        }
        catch (IOException iox)
        {
            Console.Write("Warning: ");
            Console.WriteLine(iox.Message);
        }
        catch (UnauthorizedAccessException uax)
        {
            Console.WriteLine(
                "Warning: You do not have permission to access this file.");
            Console.WriteLine(uax.Message);
        }

        if (details != null)
        {
            yield return details;
        }
    }
}

var fileNameGroups = from filePath in allFilePaths
                     let fileNameWithoutPath = Path.GetFileName(filePath)
                     group filePath by fileNameWithoutPath into nameGroup
                     select new FileNameGroup
                     {
                         FileNameWithoutPath = nameGroup.Key,
                         FilesWithThisName = GetDetails(nameGroup).ToList()
                     };

C:UsersmwaAppDataLocaldcyx0fv1.hv3
C:UsersmwaAppDataLocalnf2wqwr.y3s
C:UsersmwaAppDataLocalkfilxte4.exy
Warning: You do not have permission to access this directory.
Access to the path 'C:UsersmwaAppDataLocal
2gl4q1a.ycp' is denied.
SameNameAndContent.txt
----------------------
C:UsersmwaAppDataLocaldcyx0fv1.hv3
C:UsersmwaAppDataLocalnf2wqwr.y3s
C:UsersmwaAppDataLocalkfilxte4.exy

To compare the candidate files, we could load them into memory. The File class offers three likely looking static methods: ReadAllBytes, which treats the file as binary, and loads it into a byte array; File.ReadAllText, which treats it as text, and reads it all into a string; and File.ReadLines, which again treats it as text, but loads each line into its own string, and returns an array of all the lines. We could even call File.OpenRead to obtain a StreamReader (equivalent to the StreamWriter, but for reading data—we’ll see this again later in the chapter).

Because we’re looking at all file types, not just text, we need to use one of the binary-based methods. File.ReadAllBytes returns a byte[] containing the entire contents of the file. We could then compare the files byte for byte, to see if they are the same. Here’s some code to do that.

First, let’s update our DisplayMatches function to do the load and compare, as shown by the highlighted lines in Example 11-33.

private static void DisplayMatches(
    IEnumerable<FileNameGroup> filesGroupedByName)
{
    var groupsWithMoreThanOneFile = from nameGroup in filesGroupedByName
                                    where nameGroup.FilesWithThisName.Count > 1
                                    select nameGroup;

    foreach (var fileNameGroup in groupsWithMoreThanOneFile)
    {
        // Group the matches by the file size, then select those
        // with more than 1 file of that size.
        var matchesBySize = from match in fileNameGroup.FilesWithThisName
                            group match by match.FileSize into sizeGroup
                            where sizeGroup.Count() > 1
                            select sizeGroup;

        foreach (var matchedBySize in matchesBySize)
        {
            List<FileContents> content = LoadFiles(matchedBySize);
            CompareFiles(content);
        }
    }
}

Notice that we want our LoadFiles function to return a List of FileContents objects. Example 11-34 shows the FileContents class.

internal class FileContents
{
    public string FilePath { get; set; }
    public byte[] Content { get; set; }
}

It just lets us associate the filename with the contents so that we can use it later to display the results. Example 11-35 shows the implementation of LoadFiles, which uses ReadAllBytes to load in the file content.

private static List<FileContents> LoadFiles(IEnumerable<FileDetails> fileList)
{
    var content = new List<FileContents>();
    foreach (FileDetails item in fileList)
    {
        byte[] contents = File.ReadAllBytes(item.FilePath);
        content.Add(new FileContents
        {
            FilePath = item.FilePath,
            Content = contents
        });
    }
    return content;
}

We now need an implementation for CompareFiles, which is shown in Example 11-36.

private static void CompareFiles(List<FileContents> files)
{
    Dictionary<FileContents, List<FileContents>> potentiallyMatched =
        BuildPotentialMatches(files);

    // Now, we're going to look at every byte in each
    CompareBytes(files, potentiallyMatched);

    DisplayResults(files, potentiallyMatched);
}

This isn’t exactly the most elegant way of comparing several files. We’re building a big dictionary of all of the potential matching combinations, and then weeding out the ones that don’t actually match. For large numbers of potential matches of the same size this could get quite inefficient, but we’ll not worry about that right now! Example 11-37 shows the function that builds those potential matches.

private static Dictionary<FileContents, List<FileContents>>
   BuildPotentialMatches(List<FileContents> files)
{
    // Builds a dictionary where the entries look like:
    //  { 0, { 1, 2, 3, 4, ... N } }
    //  { 1, { 2, 3, 4, ... N }
    // ...
    //  { N - 1, { N } }
    // where N is one less than the number of files.
    var allCombinations = Enumerable.Range(0, files.Count - 1).ToDictionary(
        x => files[x],
        x => files.Skip(x + 1).ToList());

    return allCombinations;
}

This set of potential matches will be whittled down to the files that really are the same by CompareBytes, which we’ll get to momentarily. The DisplayResults method, shown in Example 11-38, runs through the matches and displays their names and locations.

private static void DisplayResults(
    List<FileContents> files,
    Dictionary<FileContents, List<FileContents>> currentlyMatched)
{
    if (currentlyMatched.Count == 0) { return; }

    var alreadyMatched = new List<FileContents>();

    Console.WriteLine("Matches");

    foreach (var matched in currentlyMatched)
    {
        // Don't do it if we've already matched it previously
        if (alreadyMatched.Contains(matched.Key))
        {
            continue;
        }
        else
        {
            alreadyMatched.Add(matched.Key);
        }
        Console.WriteLine("-------");
        Console.WriteLine(matched.Key.FilePath);
        foreach (var file in matched.Value)
        {
            Console.WriteLine(file.FilePath);
            alreadyMatched.Add(file);
        }
    }
    Console.WriteLine("-------");
}

This leaves the method shown in Example 11-39 that does the bulk of the work, comparing the potentially matching files, byte for byte.

private static void CompareBytes(
    List<FileContents> files,
    Dictionary<FileContents, List<FileContents>> potentiallyMatched)
{
    // Remember, this only ever gets called with files of equal length.
    int fileLength = files[0].Content.Length;
    var sourceFilesWithNoMatches = new List<FileContents>();
    for (int fileByteOffset = 0; fileByteOffset < fileLength; ++fileByteOffset)
    {
        foreach (var sourceFileEntry in potentiallyMatched)
        {
            byte[] sourceContent = sourceFileEntry.Key.Content;
            for (int otherIndex = 0; otherIndex < sourceFileEntry.Value.Count;
                                                                 ++otherIndex)
            {
                // Check the byte at i in each of the two files, if they don't
                //  match, then we remove them from the collection
                byte[] otherContent =
                    sourceFileEntry.Value[otherIndex].Content;
                if (sourceContent[fileByteOffset] != otherContent[fileByteOffset])
                {
                    sourceFileEntry.Value.RemoveAt(otherIndex);
                    otherIndex -= 1;
                    if (sourceFileEntry.Value.Count == 0)
                    {
                        sourceFilesWithNoMatches.Add(sourceFileEntry.Key);
                    }
                }
            }
        }
        foreach (FileContents fileWithNoMatches in sourceFilesWithNoMatches)
        {
            potentiallyMatched.Remove(fileWithNoMatches);
        }
        // Don't bother with the rest of the file if
        // there are no further potential matches
        if (potentiallyMatched.Count == 0)
        {
            break;
        }
        sourceFilesWithNoMatches.Clear();
    }
}

We’re going to need to add a test file that differs only in the content. In CreateTestFiles add another filename that doesn’t change as we go round the loop:

string fileSameSizeInAllButDifferentContent =
    "SameNameAndSizeDifferentContent.txt";

Then, inside the loop (at the bottom), we’ll create a test file that will be the same length, but varying by only a single byte:

// And now one that is the same length, but with different content
fullPath = Path.Combine(directory, fileSameSizeInAllButDifferentContent);

builder = new StringBuilder();
builder.Append("Now with ");
builder.Append(directoryIndex);
builder.AppendLine(" extra");
CreateFile(fullPath, builder.ToString());

If you build and run, you should see some output like this, showing the one identical file we have in each file location:

C:UsersmwaAppDataLocale33yz4hg.mjp
C:UsersmwaAppDataLocalung2xdgo.k1c
C:UsersmwaAppDataLocaljcpagntt.ynd
Warning: You do not have permission to access this directory.
Access to the path 'C:UsersmwaAppDataLocalcmoof2kj.ekd' is denied.
Matches
-------
C:UsersmwaAppDataLocale33yz4hg.mjpSameNameAndContent.txt
C:UsersmwaAppDataLocalung2xdgo.k1cSameNameAndContent.txt
C:UsersmwaAppDataLocaljcpagntt.yndSameNameAndContent.txt
-------

Needless to say, this isn’t exactly very efficient; and it is unlikely to work so well when you get to those DVD rips and massive media repositories. Even your 64-bit machine probably doesn’t have quite that much memory available to it.^[24] There’s a way to make this more memory-efficient. Instead of loading the file completely into memory, we can take a streaming approach.

You can think of a stream like one of those old-fashioned news ticker tapes. To write data onto the tape, the bytes (or characters) in the file are typed out, one at a time, on the continuous stream of tape.

We can then wind the tape back to the beginning, and start reading it back, character by character, until either we stop or we run off the end of the tape. Or we could give the tape to someone else, and she could do the same. Or we could read, say, 1,000 characters off the tape, and copy them onto another tape which we give to someone to work on, then read the next 1,000, and so on, until we run out of characters.

To illustrate this, let’s write a method that’s equivalent to File.ReadAllBytes using a stream (see Example 11-40).

private static byte[] ReadAllBytes(string filename)
{
    using (FileStream stream = File.OpenRead(filename))
    {
        long streamLength = stream.Length;
        if (streamLength > 0x7fffffffL)
        {
            throw new InvalidOperationException(
               "Unable to allocate more than 0x7fffffffL bytes" +
               "of memory to read the file");
        }
        // Safe to cast to an int, because
        // we checked for overflow above
        int bytesToRead = (int) stream.Length;
        // This could be a big buffer!
        byte[] bufferToReturn = new byte[bytesToRead];
        // We're going to start at the beginning
        int offsetIntoBuffer = 0;
        while (bytesToRead > 0)
        {
            int bytesRead = stream.Read(bufferToReturn,
                                        offsetIntoBuffer,
                                        bytesToRead);
            if (bytesRead == 0)
            {
                throw new InvalidOperationException(
                    "We reached the end of file before we expected..." +
                    "Has someone changed the file while we weren't looking?");
            }
            // Read may return fewer bytes than we asked for, so be
            // ready to go round again.
            bytesToRead -= bytesRead;
            offsetIntoBuffer += bytesRead;
        }

        return bufferToReturn;
    }
}

The call to File.OpenRead creates us an instance of a FileStream. This class derives from the base Stream class, which defines most of the methods and properties we’re going to use.

First, we inspect the stream’s Length property to determine how many bytes we need to allocate in our result. This is a long, so it can support truly enormous files, even if we can allocate only 2 GB of memory.

Then (once we’ve set up a few variables) we call stream.Read and ask it for all of the data in the stream. It is entitled to give us any number of bytes it likes, up to the number we ask for. It returns the actual number of bytes read, or 0 if we’ve hit the end of the stream and there’s no more data.

Notice that it returns us an int. So even if .NET did let us allocate arrays larger than 2 GB (which it doesn’t) a stream can only tell us that it has read 2 GB worth of data at a time, and in fact, the third argument to Read, where we tell it how much we want, is also an int, so 2 GB is the most we can ask for. So while FileStream is able to work with larger files thanks to the 64-bit Length property, it will split the data into more modest chunks of 2 GB or less when we read. But then one of the main reasons for using streams in the first place is to avoid having to deal with all the content in one go, so in practice we tend to work with much smaller chunks in any case.

So we always call the Read method in a loop. The stream maintains the current read position for us, but we need to work out where to write it in the destination array (offsetIntoBuffer). We also need to work out how many more bytes we have to read (bytesToRead).

We can now update the call to ReadAllBytes in our LoadFile method so that it uses our new implementation:

byte[] contents = ReadAllBytes(item.Filename);

Build and run again, and you should see output with exactly the same form as before:

C:UsersmwaAppDataLocal1ssoimgj.wqg
C:UsersmwaAppDataLocalcjiymq5b.bfo
C:UsersmwaAppDataLocaldiss5tgl.zae
Warning: You do not have permission to access this directory.
Access to the path 'C:UsersmwaAppDataLocalu1w0rj0o.2xe' is denied.
Matches
-------
C:UsersmwaAppDataLocal1ssoimgj.wqgSameNameAndContent.txt
C:UsersmwaAppDataLocalcjiymq5b.bfoSameNameAndContent.txt
C:UsersmwaAppDataLocaldiss5tgl.zaeSameNameAndContent.txt
-------

That’s all very well, but we haven’t actually improved anything. We wanted to avoid loading all of those files into memory. Instead of loading the files, let’s update our FileContents class to hold a stream instead of a byte array, as Example 11-41 shows.

internal class FileContents
{
    public string FilePath { get; set; }
    public FileStream Content { get; set; }
}

We’ll have to update the code that creates the FileContents too, in our LoadFiles method from Example 11-35. Example 11-42 shows the change required.

content.Add(new FileContents
                {
                    FilePath = item.FilePath,
                    Content = File.OpenRead(item.FilePath)
                });

(You can now delete our ReadAllBytes implementation, if you want.)

Because we’re opening all of those files, we need to make sure that we always close them all. We can’t implement the using pattern, because we’re handing off the references outside the scope of the function that creates them, so we’ll have to find somewhere else to call Close.

DisplayMatches (Example 11-33) ultimately causes the streams to be created by calling LoadFiles, so DisplayMatches should close them too. We can add a try/finally block in that method’s innermost foreach loop, as Example 11-43 shows.

foreach (var matchedBySize in matchesBySize)
{
    List<FileContents> content = LoadFiles(matchedBySize);
    try
    {
        CompareFiles(content);
    }
    finally
    {
        foreach (var item in content)
        {
            item.Content.Close();
        }
    }
}

The last thing to update, then, is the CompareBytes method. The previous version, shown in Example 11-39, relied on loading all the files into memory upfront. The modified version in Example 11-44 uses streams.

private static void CompareBytes(
    List<FileContents> files,
    Dictionary<FileContents, List<FileContents>> potentiallyMatched)
{
    // Remember, this only ever gets called with files of equal length.
    long bytesToRead = files[0].Content.Length;
    // We work through all the files at once, so allocate a buffer for each.
    Dictionary<FileContents, byte[]> fileBuffers =
        files.ToDictionary(x => x, x => new byte[1024]);

    var sourceFilesWithNoMatches = new List<FileContents>();
    while (bytesToRead > 0)
    {
        // Read up to 1k from all the files.
        int bytesRead = 0;
        foreach (var bufferEntry in fileBuffers)
        {
            FileContents file = bufferEntry.Key;
            byte[] buffer = bufferEntry.Value;
            int bytesReadFromThisFile = 0;
            while (bytesReadFromThisFile < buffer.Length)
            {
                int bytesThisRead = file.Content.Read(
                    buffer, bytesReadFromThisFile,
                    buffer.Length - bytesReadFromThisFile);
                if (bytesThisRead == 0) { break; }
                bytesReadFromThisFile += bytesThisRead;
            }
            if (bytesReadFromThisFile < buffer.Length
             && bytesReadFromThisFile < bytesToRead)
            {
                throw new InvalidOperationException(
                    "Unexpected end of file - did a file change?");
            }
            bytesRead = bytesReadFromThisFile; // Will be same for all files
        }
        bytesToRead -= bytesRead;

        foreach (var sourceFileEntry in potentiallyMatched)
        {
            byte[] sourceFileContent = fileBuffers[sourceFileEntry.Key];

            for (int otherIndex = 0; otherIndex < sourceFileEntry.Value.Count;
                                                                 ++otherIndex)
            {
                byte[] otherFileContent =
                    fileBuffers[sourceFileEntry.Value[otherIndex]];
                for (int i = 0; i < bytesRead; ++i)
                {
                    if (sourceFileContent[i] != otherFileContent[i])
                    {
                        sourceFileEntry.Value.RemoveAt(otherIndex);
                        otherIndex -= 1;
                        if (sourceFileEntry.Value.Count == 0)
                        {
                            sourceFilesWithNoMatches.Add(sourceFileEntry.Key);
                        }
                        break;
                    }
                }
            }
        }
        foreach (FileContents fileWithNoMatches in sourceFilesWithNoMatches)
        {
            potentiallyMatched.Remove(fileWithNoMatches);
        }
        // Don't bother with the rest of the file if there are
        // not further potential matches
        if (potentiallyMatched.Count == 0)
        {
            break;
        }
        sourceFilesWithNoMatches.Clear();
    }
}

Rather than reading entire files at once, we allocate small buffers, and read in 1 KB at a time. As with the previous version, this new one works through all the files of a particular name and size simultaneously, so we allocate a buffer for each file.

We then loop round, reading in a buffer’s worth from each file, and perform comparisons against just that buffer (weeding out any nonmatches). We keep going round until we either determine that none of the files match or reach the end of the files.

Notice how each stream remembers its position for us, with each Read starting where the previous one left off. And since we ensure that we read exactly the same quantity from all the files for each chunk (either 1 KB, or however much is left when we get to the end of the file), all the streams advance in unison.

This code has a somewhat more complex structure than before. The all-in-memory version in Example 11-39 had three loops—the outer one advanced one byte at a time, and then the inner two worked through the various potential match combinations. But because the outer loop in Example 11-44 advances one chunk at a time, we end up needing an extra inner loop to compare all the bytes in a chunk. We could have simplified this by only ever reading a single byte at a time from the streams, but in fact, this chunking has delivered a significant performance improvement. Testing against a folder full of source code, media resources, and compilation output containing 4,500 files (totaling about 500 MB), the all-in-memory version took about 17 seconds to find all the duplicates, but the stream version took just 3.5 seconds! Profiling the code revealed that this performance improvement was entirely a result of the fact that we were comparing the bytes in chunks. So for this particular application, the additional complexity was well worth it. (Of course, you should always measure your own code against representative problems—techniques that work well in one scenario don’t necessarily perform well everywhere.)

if (sourceFileContent[i] != otherFileContent[i])
{
    sourceFileEntry.Value.RemoveAt(otherIndex);
    otherIndex -= 1;
    if (sourceFileEntry.Value.Count == 0)
    {
        sourceFilesWithNoMatches.Add(sourceFileEntry.Key);
    }
#if DEBUG
    // Remember where we got to
    long currentPosition = sourceFileEntry.Key.Content.Position;
    // Seek to 0 bytes from the beginning
    sourceFileEntry.Key.Content.Seek(0, SeekOrigin.Begin);
    // Read 100 bytes from
    for (int index = 0; index < 100; ++index)
    {
        var val = sourceFileEntry.Key.Content.ReadByte();
        if (val < 0) { break; }
        if (index != 0) { Console.Write(", "); }
        Console.Write(val);
    }
    Console.WriteLine();
    // Put it back where we found it
    sourceFileEntry.Key.Content.Seek(currentPosition, SeekOrigin.Begin);
#endif
    break;
}

private static void WriteTo(Stream source, Stream target, int bufferLength)
{
    bufferLength = Math.Max(100, bufferLength);
    var buffer = new byte[bufferLength];
    int bytesRead;

    do
    {
        bytesRead = source.Read(buffer, 0, buffer.Length);
        if (bytesRead != 0)
        {
            target.Write(buffer, 0, bytesRead);
        }
    } while (bytesRead > 0);
}

So, we’ve seen how to read and write data to and from streams, and how we can move the current position in the stream by seeking to some offset from a known position. Up until now, we’ve been using the File.OpenRead and File.OpenWrite methods to create our file streams. There is another method, File.Open, which gives us access to some extra features.

The simplest overload takes two parameters: a string which is the path for the file, and a value from the FileMode enumeration. What’s the FileMode? Well, it lets us specify exactly what we want done to the file when we open it. Table 11-6 shows the values available.

If you use this two-argument overload, the file will be opened in read/write mode. If that’s not what you want, another overload takes a third argument, allowing you to control the access mode with a value from the FileAccess enumeration. Table 11-7 shows the supported values.

All of the file-opening methods we’ve used so far have locked the file for our exclusive use until we close or Dispose the object—if any other program tries to open the file while we have it open, it’ll get an error. However, it is possible to play nicely with other users by opening the file in a shared mode. We do this by using the overload which specifies a value from the FileShare enumeration, which is shown in Table 11-8. This is a flags enumeration, so you can combine the values if you wish.

You have to be careful when opening files in a shared mode, particularly one that permits modifications. You are open to all sorts of potential exceptions that you could normally ignore (e.g., people deleting or truncating it from underneath you).

If you need even more control over the file when you open it, you can create a FileStream instance directly.

There are two types of FileStream constructors—those for interop scenarios, and the “normal” ones. The “normal” ones take a string for the file path, while the interop ones require either an IntPtr or a SafeFileHandle. These wrap a Win32 file handle that you have retrieved from somewhere. (If you’re not already using such a thing in your code, you don’t need to use these versions.) We’re not going to cover the interop scenarios here.

If you look at the list of constructors, the first thing you’ll notice is that quite a few of them duplicate the various permutations of FileShare, FileAccess, and FileMode overloads we had on File.Open.

You’ll also notice equivalents with one extra int parameter. This allows you to provide a hint for the system about the size of the internal buffer you’d like the stream to use. Let’s look at buffering in more detail.

Long-running file operations are a common bottleneck. How many times have you clicked the Save button, and seen the UI lock up while the disk operation takes place (especially if you’re saving a large file to a network location)?

Developers commonly resort to a background thread to push these long operations off the main thread so that they can display some kind of progress or “please wait” UI (or let the user carry on working). We’ll look at that approach in Chapter 16; but you don’t necessarily have to go that far. You can use the asynchronous mode built into the stream instead. To see how it works, look at Example 11-47.

static void Main(string[] args)
{
    string path = "mytestfile.txt";
    // Create a test file
    using (var file = File.Create(path, 4096, FileOptions.Asynchronous))
    {
        // Some bytes to write
        byte[] myBytes = new byte[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
        IAsyncResult asyncResult = file.BeginWrite(
            myBytes,
            0,
            myBytes.Length,
            // A callback function, written as an anonymous delegate
            delegate(IAsyncResult result)
            {
                // You *must* call EndWrite() exactly once
                file.EndWrite(result);
                // Then do what you like
                Console.WriteLine(
                    "Called back on thread {0} when the operation completed",
                    System.Threading.Thread.CurrentThread.ManagedThreadId);
            },
            null);

        // You could do something else while you waited...
        Console.WriteLine(
            "Waiting on thread {0}...",
            System.Threading.Thread.CurrentThread.ManagedThreadId);
        // Waiting on the main thread
        asyncResult.AsyncWaitHandle.WaitOne();
        Console.WriteLine(
            "Completed {0} on thread {1}...",
            asyncResult.CompletedSynchronously ?
                "synchronously" : "asynchronously",
            System.Threading.Thread.CurrentThread.ManagedThreadId);
        Console.ReadKey();
        return;
    }
}

If you put this code in a new console application, and then compile and run, you’ll get output similar to this (the actual thread IDs will vary from run to run):

Waiting on thread 10...
Completed asynchronously on thread 10...
Called back on thread 6 when the operation completed

So, what is happening?

When we create our file, we use an overload on File.Create that takes the FileOptions we discussed earlier. (Yes, back then we showed that by constructing the FileStream directly, but the File class supports this too.) This lets us open the file with asynchronous behavior enabled.

Then, instead of calling Write, we call BeginWrite. This takes two additional parameters. The first is a delegate to a callback function of type AsyncCallback, which the framework will call when it has finished the operation to let us know that it has completed. The second is an object that we can pass in, that will get passed back to us in the callback.

We’ve used an anonymous method to provide the callback delegate. The first thing we do in that method is to call file.EndWrite, passing it the IAsyncResult we’ve been provided in the callback. You must call EndWrite exactly once for every time you call BeginWrite, because it cleans up the resources used to carry out the operation asynchronously. It doesn’t matter whether you call it from the callback, or on the main application thread (or anywhere else, for that matter). If the operation has not completed, it will block the calling thread until it does complete, then do its cleanup. Should you call it twice with the same IAsyncResult for any reason the framework will throw an exception.

In a typical Windows Forms or WPF application, we’d probably put up some progress dialog of some kind, and just process messages until we got our callback. In a server-side application we’re more likely to want to kick off several pieces of work like this, and then wait for them to finish. To do this, the IAsyncResult provides us with an AsyncWaitHandle, which is an object we can use to block our thread until the work is complete.

So, when we run, our main thread happens to have the ID 10. It blocks until the operation is complete, and then prints out the message about being done. Notice that this was, as you’d expect, on the same thread with ID 10. But after that, we get a message printed out from our callback, which was called by the framework on another thread entirely.

It is important to note that your system may have behaved differently. It is possible that the callback might occur before execution continued on the main thread. You have to be extremely careful that your code doesn’t depend on these operations happening in a particular order.

Remember that we set the FileOptions.Asynchronous flag when we opened the file to get this asynchronous behavior? What happens if we don’t do that? Let’s tweak the code so that it opens with FileOptions.None instead, and see. Example 11-48 shows the statements from Example 11-47 that need to be modified

...
// Create a test file
using (var file = File.Create(path, 4096, FileOptions.None))
{
...

If you build and run that, you’ll see some output similar to this:

Waiting on thread 9...
Completed asynchronously on thread 9...
Called back on thread 10 when the operation completed

What’s going on? That all still seemed to be asynchronous!

Well yes, it was, but under the covers, the problem was solved in two different ways. The first one used the underlying support Windows provides for asynchronous I/O in the filesystem to handle the asynchronous file operation. In the second case, the .NET Framework had to do some work for us to grab a thread from the thread pool, and execute the read operation on that to deliver the asynchronous behavior.

So far, everything we’ve talked about has been related to files, but we can create streams over other things, too. If you’re a Silverlight developer, you’ve probably been skimming over all of this a bit—after all, if you’re running in the web browser you can’t actually read and write files in the filesystem. There is, however, another option that you can use (along with all the other .NET developers out there): isolated storage.

In the duplicate file detection application we built earlier in this chapter, we had to go to some lengths to find a location, and pick filenames for the datafiles we wished to create in test mode, in order to guarantee that we don’t collide with other applications. We also had to pick locations that we knew we would (probably) have permission to write to, and that we could then load again.

Isolated storage takes this one stage further and gives us a means of saving and loading data in a location unique to a particular piece of executing code. The physical location itself is abstracted away behind the API; we don’t need to know where the runtime is actually storing the data, just that the data is stored safely, and that we can retrieve it again. (Even if we want to know where the files are, the isolated storage API won’t tell us.) This helps to make the isolated storage framework a bit more operating-system-agnostic, and removes the need for full trust (unlike regular file I/O). Hence it can be used by Silverlight developers (who can target other operating systems such as Mac OS X) as well as those of us building server or desktop client applications for Windows.

This compartmentalization of the information by characteristics of the executing code gives us a slightly different security model from regular files. We can constrain access to particular assemblies, websites, and/or users, for instance, through an API that is much simpler (although much less sophisticated) than the regular file security.

static void Main(string[] args)
{
    IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForAssembly();
    // Create a directory - safe to call multiple times
    store.CreateDirectory("Settings");
    // Open or create the file
    using (IsolatedStorageFileStream stream = store.OpenFile(
                                "Settings\standardsettings.txt",
                                System.IO.FileMode.OpenOrCreate,
                                System.IO.FileAccess.ReadWrite))
    {
        UseStream(stream);
    }
    Console.ReadKey();
}

static void UseStream(Stream stream)
{
    if (stream.Length > 0)
    {
        using (StreamReader reader = new StreamReader(stream))
        {
            Console.WriteLine(reader.ReadToEnd());
        }
    }
    else
    {
        using (StreamWriter writer = new StreamWriter(stream))
        {
            writer.WriteLine(
                "Initialized settings at {0}", DateTime.Now.TimeOfDay);
            Console.WriteLine("Settings have been initialized");
        }
    }
}

Settings have been initialized

Initialized settings at 10:34:47.7014833

Defining “Isolated”

So, what makes isolated storage “isolated”? The .NET Framework partitions information written into isolated storage based on some characteristics of the executing code.

Several types of isolated store are available to you:

• Isolation by user and assembly (optionally supporting roaming)

• Isolation by user, domain, and assembly (optionally supporting roaming)

• Isolation by user and application (optionally supporting roaming)

• Isolation by user and site (only on Silverlight)

• Isolation by machine and assembly

• Isolation by machine, domain, and assembly

• Isolation by machine and application

Silverlight supports only two of these: by user and site, and by user and application.

Isolation by user and assembly

In Example 11-50, we acquired a store isolated by user and assembly, using the static method IsolatedStorageFile.GetUserStoreForAssembly. This store is unique to a particular user, and the assembly in which the calling code is executing. You can try this out for yourself. If you log in to your box as a user other than the one under which you’ve already run our example app, and run it again, you’ll see some output like this:

Settings have been initialized

That means our settings file doesn’t exist (for this user), so we must have been given a new store.

As you might expect, the user is identified by the authenticated principal for the current thread. Typically, this is the logged-on user that ran the process; but this could have been changed by impersonation (in a web application, for example, you might be running in the context of the web user, rather than that of the ASP.NET process that hosts the site).

Identifying the assembly is slightly more complex. If you have signed the assembly, it uses the information in that signature (be it a strong name signature, or a software publisher signature, with the software publishing signature winning if it has both).

If, on the other hand, the assembly is not signed, it will use the URL for the assembly. If it came from the Internet, it will be of the form:

http://some/path/to/myassembly.dll

If it came from the local filesystem, it will be of the form:

file:///C:/some/path/to/myassembly.dll

Figure 11-9 illustrates how multiple stores get involved when you have several users and several different assemblies. User 1 asks MyApp.exe to perform some task, which asks for user/assembly isolated storage. It gets Store 1. Imagine that User 1 then asks MyApp.exe to perform some other task that requires the application to call on MyAssembly.dll to carry out the work. If that in turn asks for user/assembly isolated storage, it will get a different store (labeled Store 2 in the diagram). We get a different store, because they are different assemblies.

When a different user, User 2, asks MyApp.exe to perform the first task, which then asks for user/assembly isolated storage, it gets a different store again—Store 3 in the diagram—because they are different users.

Figure 11-9. User and assembly isolation

OK, what happens if we make two copies of MyApp.exe in two different locations, and run them both under the same user account? The answer is that it depends....

If the applications are not signed the assembly identification rules mean that they don’t match, and so we get two different isolated stores.

If they are signed the assembly identification rules mean that they do match, so we get the same isolated store.

Our app isn’t signed, so if we try this experiment, we’ll see the standard “first run” output for our second copy.

Warning

Be very careful when using isolated storage with signed assemblies. The information used from the signature includes the Name, Strong Name Key, and Major Version part of the version info. So, if you rev your application from 1.x to 2.x, all of a sudden you’re getting a different isolated storage scope, and all your existing data will “vanish.” One way to deal with this is to use a distinct DLL to access the store, and keep its version numbers constant.

Isolation by user, domain, and assembly

Isolating by domain means that we look for some information about the application domain in which we are running. Typically, this is the full URL of the assembly if it was downloaded from the Web, or the local path of the file.

Notice that this is the same rule as for the assembly identity if we didn’t sign it! The purpose of this isolation model is to allow a single signed assembly to get different stores if it is run from different locations. You can see a diagram that illustrates this in Figure 11-10.

Figure 11-10. Assembly and domain isolation compared

To get a store with this isolation level, we can call the IsolatedStorageFile class’s GetUserStoreForDomain method.

Isolation by user and application

A third level of isolation is by user and application. What defines an “application”? Well, you have to sign the whole lot with a publisher’s (Authenticode) signature. A regular strong-name signature won’t do (as that will identify only an individual assembly).

Note

If you want to try this out quickly for yourself, you can run the ClickOnce Publication Wizard on the Publish tab of your example project settings. This will generate a suitable test certificate and sign the app.

To get a store with user and application isolation, we call the IsolatedStorageFile class’s GetUserStoreForApplication method.

Note

If you haven’t signed your application properly, this method will throw an exception.

So, it doesn’t matter which assembly you call from; as long as it is a part of the same application, it will get the same store. You can see this illustrated in Figure 11-11.

Figure 11-11. Application isolation

Note

This can be particularly useful for settings that might be shared between several different application components.

Machine isolation

What if your application or component has some data you want to make available to all users on the system? Maybe you want to cache common product information or imagery to avoid a download every time you start the app. For these scenarios you need machine isolation.

As you saw earlier, there is an isolation type for the machine which corresponds to each isolation type for the user. The same resolution rules apply in each case. The methods you need are:

GetMachineStoreForApplication

GetMachineStoreForDomain

GetMachineStoreForAssembly

Managing Isolated Storage

As a user, you might want to look at the data stored in isolated storage by applications running on your machine. It can be complicated to manage and debug isolated storage, but there are a few tools and techniques to help you.

First, there’s the storeadm.exe tool. This allows you to inspect isolated storage for the current user (by default), or the current machine (by specifying the /machine option) or current roaming user (by specifying /roaming).

So, if you try running this command:

storeadm /MACHINE /LIST

you will see output similar to this (listing the various stores for this machine, along with the evidence that identifies them):

Microsoft (R) .NET Framework Store Admin 4.0.30319.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Record #1
[Assembly]
<StrongName version="1"
Key="0024000004800000940000000602000000240000525341310004000001000100A5FE84898F
190EA6423A7D7FFB1AE778141753A6F8F8235CBC63A9C5D04143C7E0A2BE1FC61FA6EBB52E7FA9B
48D22BAF4027763A12046DB4A94FA3504835ED9F29CD031600D5115939066AABE59A4E61E932AEF
0C24178B54967DD33643FDE04AE50786076C1FB32F64915E8200729301EB912702A8FDD40F63DD5
A2DE218C7"
Name="ConsoleApplication7"
Version="1.0.0.0"/>

        Size : 0
Record #2
[Domain]
<StrongName version="1"
Key="0024000004800000940000000602000000240000525341310004000001000100A5FE84898F
190EA6423A7D7FFB1AE778141753A6F8F8235CBC63A9C5D04143C7E0A2BE1FC61FA6EBB52E7FA9B
48D22BAF4027763A12046DB4A94FA3504835ED9F29CD031600D5115939066AABE59A4E61E932AEF
0C24178B54967DD33643FDE04AE50786076C1FB32F64915E8200729301EB912702A8FDD40F63DD5
A2DE218C7"
Name="ConsoleApplication7"
Version="1.0.0.0"/>

[Assembly]
<StrongName version="1"
Key="0024000004800000940000000602000000240000525341310004000001000100A5FE84898F
190EA6423A7D7FFB1AE778141753A6F8F8235CBC63A9C5D04143C7E0A2BE1FC61FA6EBB52E7FA9B
48D22BAF4027763A12046DB4A94FA3504835ED9F29CD031600D5115939066AABE59A4E61E932AEF
0C24178B54967DD33643FDE04AE50786076C1FB32F64915E8200729301EB912702A8FDD40F63DD5
A2DE218C7"
Name="ConsoleApplication7"
Version="1.0.0.0"/>

        Size : 0

Notice that there are two stores in that example. One is identified by some assembly evidence (the strong name key, name, and major version info). The other is identified by both domain and assembly evidence. Because the sample application is in a single assembly, the assembly evidence for both stores happens to be identical!

Warning

You can also add the /REMOVE parameter which will delete all of the isolated storage in use at the specified scope. Be very careful if you do this, as you may well delete storage used by another application entirely.

That’s all very well, but you can’t see the place where those files are stored. That’s because the actual storage is intended to be abstracted away behind the API. Sometimes, however, it is useful to be able to go and pry into the actual storage itself.

Warning

Remember, this is an implementation detail, and it could change between versions. It has been consistent since the first version of the .NET Framework, but in the future, Microsoft could decide to store it all in one big file hidden away somewhere, or using some mystical API that we don’t have access to.

We can take advantage of the fact that the debugger can show us the private innards of the IsolatedStorageFile class. If we set a breakpoint on the store.CreateFile line in our sample application, we can inspect the IsolatedStorageFile object that was returned by GetUserStoreForApplication in the previous line. You will see that there is a private field called m_RootDir. This is the actual root directory (in the real filesystem) for the store. You can see an example of that as it is on my machine in Figure 11-12.

Figure 11-12. IsolatedStorageFile internals

If you copy that path and browse to it using Windows Explorer, you’ll see something like the folder in Figure 11-13.

There’s the Settings directory that we created! As you might expect, if you were to look inside, you’d see the standardsettings.txt file our program created.

Figure 11-13. An isolated storage folder

As you can see, this is a very useful debugging technique, allowing you to inspect and modify the contents of files in isolated storage, and identify exactly which store you have for a particular scope. It does rely on implementation details, but since you’d only ever do this while debugging, the code you ultimately ship won’t depend on any nonpublic features of isolated storage.

OK. So far, we’ve seen two different types of stream; a regular file, and an isolated storage file. We use our familiar stream tools and techniques (like StreamReader and StreamWriter), regardless of the underlying type.

So, what other kinds of stream exist? Well, there are lots; several subsystems in the .NET framework provide stream-based APIs. We’ll see some networking ones in Chapter 13, for example. Another example is from the .NET Framework’s security features: CryptoStream (which is used for encrypting and decrypting a stream of data). There’s also a MemoryStream in System.IO which uses memory to store the data in the stream.

In this final section, we’ll look at a stream that is not a file. We’ll use a stream from .NET’s cryptographic services to encrypt a string. This encrypted string can be decrypted later as long as we know the key. The test program in Example 11-51 illustrates this.

static void Main(string[] args)
{
    byte[] key;
    byte[] iv;

    // Get the appropriate key and initialization vector for the algorithm
    SelectKeyAndIV(out key, out iv);

    string superSecret = "This is super secret";

    Console.WriteLine(superSecret);

    string encryptedText = EncryptString(superSecret, key, iv);

    Console.WriteLine(encryptedText);

    string decryptedText = DecryptString(encryptedText, key, iv);

    Console.WriteLine(decryptedText);

    Console.ReadKey();
}

It is going to write a message to the console, encrypt it, write the encrypted text to the console, decrypt it, and write the result of that back to the console. All being well, the first line should be the same as the last, and the middle line should look like gibberish!

The first thing we do is get a suitable key and initialization vector for our cryptographic algorithm. These are the two parts of the secret key that are shared between whoever is encrypting and decrypting our sensitive data.

A detailed discussion of cryptography is somewhat beyond the scope of this book, but here are a few key points to get us going. Unenciphered data is known as the plain text, and the encrypted version is known as cipher text. We use those terms even if we’re dealing with nontextual data. The key and the initialization vector (IV) are used by a cryptographic algorithm to encrypt the unenciphered data. A cryptographic algorithm that uses the same key and IV for both encryption and decryption is called a symmetric algorithm (for obvious reasons). Asymmetric algorithms also exist, but we won’t be using them in this example.

Needless to say, if an unauthorized individual gets hold of the key and IV, he can happily decrypt any of your cipher text, and you no longer have a communications channel free from prying eyes. It is therefore extremely important that you take care when sharing these secrets with the people who need them, to ensure that no one else can intercept them. (This turns out to be the hardest part—key management and especially human factors turn out to be security weak points far more often than the technological details. This is a book about programming, so we won’t even attempt to solve that problem. We recommend the book Secrets and Lies: Digital Security in a Networked World by Bruce Schneier [John Wiley & Sons] for more information.)

We’re calling a method called SelectKeyAndIV to get hold of the key and IV. In real life, you’d likely be sharing this information between different processes, usually even on different machines; but for the sake of this demonstration, we’re just creating them on the fly, as you can see in Example 11-52.

private static void SelectKeyAndIV(out byte[] key, out byte[] iv)
{
    var algorithm = TripleDES.Create();
    algorithm.GenerateIV();
    algorithm.GenerateKey();

    key = algorithm.Key;
    iv = algorithm.IV;
}

TripleDES is an example of a symmetric algorithm, so it derives from a class called SymmetricAlgorithm. All such classes provide a couple of methods called GenerateIV and GenerateKey that create cryptographically strong random byte arrays to use as an initialization vector and a key. See the sidebar below for an explanation of why we need to use a particular kind of random number generator when cryptography is involved.

static void Main(string[] args)
{
    Random random = new Random(3);
    for (int i = 0; i < 5; ++i)
    {
        Console.WriteLine(random.Next());
    }
    Console.ReadKey();
}

630327709
1498044246
1857544709
426253993
1203643911

OK, with that done, we can now implement our EncryptString method. This takes the plain text string, the key, and the initialization vector, and returns us an encrypted string. Example 11-53 shows an implementation.

private static string EncryptString(string plainText, byte[] key, byte[] iv)
{
    // Create a crypto service provider for the TripleDES algorithm
    var serviceProvider = new TripleDESCryptoServiceProvider();

    using (MemoryStream memoryStream = new MemoryStream())
    using (var cryptoStream = new CryptoStream(
                                    memoryStream,
                                    serviceProvider.CreateEncryptor(key, iv),
                                    CryptoStreamMode.Write))
    using (StreamWriter writer = new StreamWriter(cryptoStream))
    {
        // Write some text to the crypto stream, encrypting it on the way
        writer.Write(plainText);
        // Make sure that the writer has flushed to the crypto stream
        writer.Flush();
        // We also need to tell the crypto stream to flush the final block out to
        // the underlying stream, or we'll
        // be missing some content...
        cryptoStream.FlushFinalBlock();

        // Now, we want to get back whatever the crypto stream wrote to our memory
        // stream.
        return GetCipherText(memoryStream);
    }
}

We’re going to write our plain text to a CryptoStream, using the standard StreamWriter adapter. This works just as well over a CryptoStream as any other, but instead of coming out as plain text, it will be enciphered for us. How does that work?

Representing Binary As Text with Base64 Encoding

Unfortunately, the cipher text is not actually text at all—it is just a stream of bytes. We can’t use the UTF8Encoding.UTF8.GetString technique we saw in Chapter 10 to turn the bytes into text, because these bytes don’t represent UTF-8 encoded characters.

Instead, we need some other sort of text-friendly representation if we’re going to be able to print the encrypted text to the console. We could write each byte out as hex digits. That would be a perfectly reasonable string representation.

However, that’s not very compact (each byte is taking five characters in the string!):

0x01 0x0F 0x03 0xFA 0xB3

A much more compact textual representation is Base64 encoding. This is a very popular textual encoding of arbitrary data. It’s often used to embed binary in XML, which is a fundamentally text-oriented format.

And even better, the framework provides us with a convenient static helper method to convert from a byte[] to a Base64 encoded string: Convert.ToBase64String.

Note

If you’re wondering why there’s no Encoding class for Base64 to correspond to the Unicode, ASCII, and UTF-8 encodings we saw in Chapter 10, it’s because Base64 is a completely different kind of thing. Those other encodings are mechanisms that define binary representations of textual information. Base64 does the opposite—it defines a textual representation for binary information.

Example 11-54 shows how we make use of that in our GetCipherText method.

Example 11-54. Converting to Base64

private static string GetCipherText(MemoryStream memoryStream)
{
    byte[] buffer = memoryStream.ToArray();
    return System.Convert.ToBase64String(buffer, 0, buffer.Length);
}

We use a method on MemoryStream called ToArray to get a byte[] array containing all the data written to the stream.

Warning

Don’t be caught out by the ToBuffer method, which also returns a byte[] array. ToBuffer returns the whole buffer including any “extra” bytes that have been allocated but not yet used.

Finally, we call Convert.ToBase64String to get a string representation of the underlying data, passing it the byte[], along with a start offset into that buffer of zero (so that we start with the first byte), and the length.

That takes care of encryption. How about decryption? That’s actually a little bit easier. Example 11-55 shows how.

Example 11-55. Decryption

private static string DecryptString(string cipherText, byte[] key, byte[] iv)
{
    // Create a crypto service provider for the TripleDES algorithm
    var serviceProvider = new TripleDESCryptoServiceProvider();

    // Decode the cipher-text bytes back from the base-64 encoded string
    byte[] cipherTextBytes = Convert.FromBase64String(cipherText);

    // Create a memory stream over those bytes
    using (MemoryStream memoryStream = new MemoryStream(cipherTextBytes))
    // And create a cryptographic stream over the memory stream,
    // using the specified algorithm
    // (with the provided key and initialization vector)
    using (var cryptoStream =
                  new CryptoStream(
                      memoryStream,
                      serviceProvider.CreateDecryptor(key, iv),
                      CryptoStreamMode.Read))
    // Finally, create a stream reader over the stream, and recover the
    // original text
    using (StreamReader reader = new StreamReader(cryptoStream))
    {
        return reader.ReadToEnd();
    }
}

First, we use Convert.FromBase64String to convert our Base64 encoded string back to an array of bytes. We then construct a MemoryStream over that byte[] by passing it to the appropriate constructor.

As before, we wrap the MemoryStream with a CryptoStream, this time passing it the ICryptoTransform created by a call to CreateDecryptor on our TripleDESCryptoServiceProvider, and putting it into CryptoStreamMode.Read.

Finally, we construct our old friend the StreamReader over the CryptoStream, and read the content back as a string.

So, what’s actually happening here?

CryptoStream uses the ICryptoTransform to take care of turning the cipher text in the MemoryStream back into plain text. If you remember, that plain text is actually the set of UTF-8 encoded bytes we originally wrote to the stream with the StreamWriter back in the encryption phase. So, the StreamReader takes those and converts them back into a string for us. You can see that illustrated in Figure 11-14.

This is a very powerful example of how we can plug together various components in a kind of pipeline to achieve quite complex processing, from simple, easily understood building blocks that conform to a common pattern, but which have no dependencies on each other’s implementation details. The Stream abstraction is the key to this flexibility.

Figure 11-14. Encryption and decryption pipeline using streams

In this chapter we looked at the classes in the System.IO namespace that relate to files and streams. We saw how we can use static methods on the File, Directory, and Path classes to manage and manipulate files and folders in the filesystem, including creating, deleting, appending, and truncating data, as well as managing their access permissions.

We saw how to use StreamReader and StreamWriter to deal with reading and writing text from files, and how we can also read and write binary data using the underlying Stream objects themselves, including the ability to Seek backward and forward in the file.

We then looked at a special type of file stream called isolated storage. This gives us the ability to manage the scope of file access to particular users, machines, applications, or even assemblies. We gain control over quotas (the maximum amount of space any particular store is allowed to use), and get to use local file storage in normally restricted security contexts like that of a Silverlight application, for example.

Finally, we looked at some streams that aren’t files, including MemoryStream, which uses memory as its underlying storage mechanism, and CryptoStream, which has no storage of its own, delegating that responsibility to another stream. We showed how these patterns can be used to plug streams together into a processing pipeline.