Testing with American Fuzzy Lop

In this book, we'll make use of American Fuzzy Lop, a best-of-breed fuzzer commonly used in other systems languages. AFL is an external tool that takes a corpus of inputs, an executable that reads inputs from STDIN, and mutates the corpus with a variety of heuristics to find crashing inputs. Our aim is to seek out crash bugs in naive HashMap, this implies that we're going to need some kind of program to run our HashMap in. In fact, if you'll recall back to the project's Cargo.toml, we already had the infrastructure for such in place:

[[bin]]
name = "naive_interpreter"
doc = false

The source for naive_interpreter is a little goofy looking but otherwise uneventful:

extern crate naive_hashmap;

use std::io;
use std::io::prelude::*;

fn main() {
    let mut hash_map = naive_hashmap::HashMap::new();

    let n = io::stdin();
    for line in n.lock().lines() {
        if let Ok(line) = line {
            let mut cmd = line.split(" ");
            match cmd.next() {
                Some("LOOKUP") => {
                    if let Some(key) = cmd.next() {
                        let _ = hash_map.get(key);
                    } else {
                        continue;
                    }
                }
                Some("INSERT") => {
                    if let Some(key) = cmd.next() {
                        if let Some(val) = cmd.next() {
                            let _ = hash_map.insert(key.to_string(), 
                            val.to_string());
                        }
                    } else {
                        continue;
                    }
                }
                _ => continue,
            }
        } else {
            break;
        }
    }
}

Lines are read from stdin, and these lines are split along spaces and interpreted as commands, either LOOKUP or INSERT and these are interpreted into actions on the naive HashMap. The corpus for an AFL run can live pretty much anywhere. By convention, in this book we'll store the corpus in-project in a top-level resources/directory. Here's resources/in/mixed_gets_puts:

LOOKUP 10
INSERT abs 10
LOOKUP abs

The larger your input corpus, the more material AFL has to mutate and the faster—maybe—you'll find crashing bugs. Even in a case such as naive HashMap where we can be reasonably certain that there will be no crashing bugs—owing to the lack of pointer manipulation and potentially fatal integer operations—it's worthwhile building up a good corpus to support future efforts. After building a release of the project, getting an AFL run going is a cargo command away:

> cargo afl build --release
> cargo afl fuzz -i resources/in/ -o resources/out/ target/release/naive_interpreter

This says to execute target/release/naive_interpreter under AFL with input corpus at resources/in and to output any failing cases under resources/out. Such crashes often make excellent unit tests. Now, the trick with fuzzing tools in general is they're not part of any kind of quick-cycle test-driven development loop. These tool runs are long and often get run on dedicated hardware overnight or over many days. Here, for example, is the AFL run I had going while writing this chapter:

There's a fair bit of information here but consider that the AFL runtime indicator is measured in days. One key advantage to the use of AFL compared to other fuzzers is the prominence of AFL in the security community. There are a good many papers describing its implementation and the interpretation of its, uh, comprehensive interface. You are warmly encouraged to scan the Further reading section of this chapter for more information.