Fuzzing EPICS with AFL

To start, let’s get some background on a few things to fill you in depending on where you are coming from. Fuzzing is an automated process of testing software for implementation bugs that can lead to crashes and exploits. This is typically done by pumping random data into the software system that then gets mutated into other random data in an attempt to see what the software system can and cannot handle. One of the state-of-the-art tools to use for this is the American Fuzzing Lop (AFL) written by Michal Zalewski. AFL allows a tester to provide some test input data which it then feeds to the software system, and then mutates that data in many different ways to feed it variations of the original input tests.

If you are coming from the EPICS community, you can skip this part as you already know what EPICS is. For those who don’t, EPICS, the Experimental Physics and Industrial Control System, is a set of software tools that provide a software infrastructure for distributed systems that operate major scientific devices like particle colliders and major telescopes.

Who is this for?

This article is for anyone interested at learning how to use AFL through example, current threat researchers looking for a new project to help out, or EPICS developers who might want to get some insight about a process that could help improve the integrity of their software.

Setup

To get started, we’ll need a few things. First, we’ll need to be running on a Linux box, in this case we’ll be working on Ubuntu, but you should be able to adapt the slight differences (mostly just repository items) for your specific distribution.

First, make sure you have these dependencies installed:
$ sudo apt-get install git build-essential curl libssl-dev sudo libtool libtool-bin libglib2.0-dev bison flex automake python3 python3-dev python3-setuptools libpixman-1-dev gcc-9-plugin-dev cgroup-tools \
clang-11 clang-tools-11 libc++1-11 libc++-11-dev libc++abi1-11 libc++abi-11-dev libclang1-11 libclang-11-dev libclang-common-11-dev libclang-cpp11 libclang-cpp11-dev liblld-11 liblld-11-dev liblldb-11 liblldb-11-dev libllvm11 libomp-11-dev libomp5-11 lld-11 lldb-11 python3-lldb-11 llvm-11 llvm-11-dev llvm-11-runtime llvm-11-tools

Next, clone the AFLplusplus (newer version of AFL) repository, change into the directory, and run make as such to build AFL:

$ git clone https://github.com/AFLplusplus/AFLplusplus
$ cd AFLplusplus
$ make distrib
$ sudo make install

Now that we have AFL built and ready to go, let’s move on to getting EPICS setup. First, setup a directory in which you would like to clone EPICS to and change to it, for now we’ll just assume you clone it to your home directory. Then clone the repository as such:

$ git clone --recursive https://github.com/epics-base/epics-base.git

Before we can run make and compile EPICS, we need to change a couple of things. In order to fuzz a binary, we have to compile it with AFL’s version of gcc/g++ for it to fuzz the binary. So open the EPICS config file located in

~/epics-base/configure/CONFIG.gnuCommon

In here you’ll find two lines at the top that specify the compiler to be used. On Ubuntu, it should look like this:

CC = $(GNU_BIN)/$(CMPLR_PREFIX)gcc$(CMPLR_SUFFIX)
CCC = $(GNU_BIN)/$(CMPLR_PREFIX)g++$(CMPLR_SUFFIX)

Comment out these two lines and add the path to the AFL gcc and g++ compilers. To find where these are located you can do:

$ whereis afl-cc
$ whereis afl-g++

Once you have these paths, edit the config file to include these two paths instead as such (and remember your AFL compiler paths might be different than what I have here):

#CC = $(GNU_BIN)/$(CMPLR_PREFIX)gcc$(CMPLR_SUFFIX) 
#CCC = $(GNU_BIN)/$(CMPLR_PREFIX)g++$(CMPLR_SUFFIX) 
CC = /usr/bin/afl-gcc 
CCC = /usr/bin/afl-g++

Now that we have made these changes, change directories so that you are in the epics-base directory and run make to compile and build EPICS.

Fuzzing

Now to actually get started with fuzzing, we’ll need one more thing which is some example test data to feed for the fuzzer to mutate off of. This is just one of the few ways on fuzzing a target, and in the case of the EPICS softIoc binary, we’ll be using .db files. The .db files used for testing were gathered from the EPICS repository. Here’s an example of one:

record(calc,"$(P)$(R)_P$(PORT)_A$(A)_calc") {
    field(DESC, "Counter")
    field(SCAN,"Passive")
    field(CALC, "(A<99)?(A+1):0")
    field(INPA,"$(P)$(R)_P$(PORT)_A$(A)_calc NPP NMS")
    field(FLNK,"$(P)$(R)_P$(PORT)_A$(A)_so")
    field(EGU, "Counts")
    field(HOPR, "10")
    field(FLNK,"$(P)$(R)_P$(PORT)_A$(A)_so")
}
record(stringout,"$(P)$(R)_P$(PORT)_A$(A)_so") {
    field(DOL,"$(P)$(R)_P$(PORT)_A$(A)_calc NPP NMS")
    field(OMSL,"closed_loop")
    field(FLNK,"$(P)$(R)_P$(PORT)_A$(A)_si")
}
record(stringin,"$(P)$(R)_P$(PORT)_A$(A)_si") {
    field(DTYP,"testBlock")
    field(INP,"@asyn($(PORT),$(A)) $(P)$(R)_P$(PORT)_A$(A)_so")
    field(VAL,"$(VAL)")
}

Once you have a collection of these, there is some work we can do to help AFL run a little faster. By running the afl-cmin command on a directory with these input data files, AFL will go through these and find which files might be redundant tests and reduces the input data file count and sends it to an output directory. These tests are checked and reduced based on the binary they will be used with, which in this case is ~/epics-base/bin/linux-x86_64/softIoc. This command also has options that you can look into but aren’t necessary.

$ afl-cmin -i <input test folder> -o <output tests folder> ~/epics-base/bin/linux-x86_64/softIoc

Optional:

There is also another AFL command than can help further fuzzing speed and that’s afl-tmin. This will reduce each individual test file down and must be done for each individual file. This wasn’t used in this testing, but here is an example of the use of the command with softIoc:

$ afl-tmin -i <test input file> -o <reduced file> ~/epics-base/bin/linux-x86_64/softIoc

Now that we have our test input, whether its the original list of files or the reduced list from afl-cmin, we can now begin to fuzz EPICS softIoc. We simply run the afl-fuzz command with some options specified. We specify the path to the test inputs, the path for the output of AFL, and then the target binary that we are fuzzing which for us is softIoc, and then any further options such as memory limits and others.

$ afl-fuzz -i <inputs folder path> -o <output folder path> ~/epics-base/bin/linux-x86_64/softIoc

Note:

On Ubuntu (and potentially on other distros as well), you will get this error message below. Once your run the echo command as root to fix it, you’ll be given a second error to which you’ll need to run the commands that it states to resolve it. You will need to do this to run AFL.

First AFL error with echo command to fix

Second AFL error with commands to fix

After those fixes, running AFL will then bring up a console information window like this:

Here, we get a lot of different information, but there’s few main things. In the top part of the console in the “process timing” box, we can see the total run time of the fuzzer, the time since the last new path has been explored, and the amount of time since the last unique crash and hang was found. The “overall results” section shows us the total paths and the unique crashes and hangs for the current fuzzer. The “stage process” section shows us the total executions done on the binary as well as the speed of executions per second. For a large binary like softIoc, this number will be pretty slow (~7 exec/sec), compared to others this is pretty slow, which AFL hints at with the added “zzzz…”. The “findings in depth” gives some more of the same information that is expressed in the “overall results” section. To stop the fuzzer, press ctrl-c.

To get good code coverage, it’s recommended that you run the fuzzer for at least 24 hours.

Faster Results

To get even faster results, depending on the hardware you are running the fuzzing on, we can speed things up a bit more. AFL supports the use of multiple fuzzers running together in parallel, which greatly speeds up the process of code coverage across a given binary, especially large ones like this. So that each fuzzer doesn’t cover the same branches of code as the each of the other fuzzers, there is a master fuzzer and then the rest of the worker fuzzers. Each of the fuzzers checks on the master fuzzer’s folder to determine what tests are queued. Each fuzzer runs on one CPU core/thread, so the more you have available to work with, the better. In this example, I’ll show how to setup four parallel fuzzers. To start, run the afl-gotscpu command to see what cores are available to run the fuzzers on and take note of them. Then we can start running each instance of the fuzzers in the same way as before but with a few added bits of information, such as whether the fuzzer is the master or worker(-M/-S) node and the name of the fuzzer and what core to run on (-b). It should look something like this:

$ afl-fuzz -i <inputs path> -o <outputs path> -M fuzzer1 -b 0 epics-base/bin/linux-x86_64/softIoc
...
$ afl-fuzz -i <inputs path> -o <outputs path> -S fuzzer2 -b 1 epics-base/bin/linux-x86_64/softIoc

After you have as many fuzzers as you want running, you can use the command afl-whatsup <path to output folder> with the main output folder to get an information overview of all the fuzzers that are running.

Analyzing Results

Now after running our fuzzing, let’s look at the outputs. The output folder has a few sub-folders, but the main one that we are interested in is the crashes folder. This contains all the input tests that resulted in crashes to the binary, which we can run against softIoc in gdb and see what kind of crash it was.

Next we need to recompile EPICS as it would be normally with gcc and g++, instead of the AFL versions. Just create a new directory and re-clone epics-base and run make in the directory to compile and build it.

Then we can run softIoc in gdb and run each of the crashes against it with the run command and the path to the crash case, like such:

$ gdb ~/epics-base/bin/linux-x86_64/softIoc
gdb> run ~/outputs/crashes/id:000000,sig:11,src:000250+000465,time:43744812,op:splice,rep:2

Gdb will then show the crash error type, such as SIGSEGV, or segmentation fault.

There is also a simple plugin tool written in python called “exploitable” that exists that will classify the crash as exploitable or not based on the type of crash that resulted from the crash case file. The author of this plugin has noted that this tool is an engineering tool and is only tested on some distros, so take it with a slight grain of salt and be sure to follow up on the crashes with your own analysis. To get the plugin, clone the repository and import it into gdb when you run it each time:

git clone https://github.com/jfoote/exploitable.git
...
gdb> source <clone directory>/exploitable/exploitable.py

After you run a crash case file in gdb with softIoc and crash it, type the command exploitable into gdb. This will take the crash info from gdb and use it to classify the likelihood of that crash being exploitable. To look at all the classifications, you can look in the rules.py file located in exploitable/libs/rules.py. The exploitable command will give you the crash type, the likelihood of exploitation, and a summary of what that crash type is.

Here is an example from a crash case from my time fuzzing EPICS:

gdb output from running a crash case file against softIoc, resulting in a segmentation fault

And here’s the example output when we use exploitable with that gdb output:

As you can see with the particular crash case, the crash resulted from a segmentation fault, and from the information that gdb provides, exploitable classifies it as “Probably Not Exploitable”.

If you do end up finding an exploitable crash, always be sure to properly disclose it to the developers of EPICS. Finally, to give credit where credit is due, this project was sponsored by the following NSF* Grant with the guidance of Dr. Gedare Bloom.

*Disclaimer: the opinions, findings, and conclusions or recommendations expressed are those of the author(s) and do not necessarily reflect the views of the National Science Foundation