Fuzz testing is a technique in software testing using random and unexpected inputs to identify vulnerabilities and data handling gaps in a program. Many of these detectable errors, like buffer overflow, can have serious security implications. These tests can be automated and run periodically as part of CI/CD. There is also a requirement from one of our customers to fuzz test our software.
There are different ways to fuzz test userspace programs and libraries and kernel modules. We investigated and tested two types of solutions to fuzz test userspace programs. There is also a kernel space fuzzer, but its more involved and we have not tested it. It also works by fuzzing systemcall/ sysfs etc. inputs:
Lib-fuzzer ([LibFuzzer](https://llvm.org/docs/LibFuzzer.html)): This is a built-in feature of LLVM. We would compile our program using clang with specific compiler options. The result is an executable which we run and the llvm runtime automatically passes fuzzed inputs to the function we are testing. This works like a Unittest, so we neeed to implement the functioality by writing fuzz-tests in C (or whatever the language our programs and libraries are written). The full coverage will take a long time and is considerable development effort.
AFL++ ([AFL++](https://aflplus.plus/)): AFL++ is more advanced but easier to use version of fuzzing engine. Here we don't have to write custom tests. Instead we just need to compile our userspace programs and libraries using their compiler wrapper. These compilers are compatible to gcc and clang. We can provide the engine with a large sets of inputs and the engine automatically fuzz tests it. We can run this periodically as part of our CI/CD and its not a big effort to start running it.
syzkaller ([Kernel Fuzzer](https://github.com/google/syzkaller)): This can be used to fuzz kernel modules like lnet drivers. Needs more investigation and test to figure out how to use it.
We can start with AFL++ as its very straightforward to get it up and running and its very good as finding issues.
We have run both Lib-Fuzzer and AFL test in lnetctl command and within a few minutes of running it has already identified issues like double free, buffer overflow etc. These could be potentially exploited by hackers. So it's a very good idea to run it periodically and identify issues and fix them.
We need a program that would run periodically in our CI/CD. it needs to.
* Clone and build AFL++ ([AFL++ Github repo](https://github.com/AFLplusplus/AFLplusplus))
* Clone Lustre Release
* Since this Compiler wrapper does not run with kernel modules, we need to run ./configure rormally.
* Make the utilities using AFL++ compiler wrapper.
`CC=afl-gcc-fast make utils`
* Before running the test, create an input folder and create files with input parapeters that we normally run.
eg: echo "ping --sources=192.168.64.5@tcp" testcases/ping.txt"
We can have as many of these test cases of the utility we are testing.
* run the test:
eg: `afl-fuzz -i ./testcases -o ./results lnetctl`
* It will run the tests feeding randomized inputs and it can run for hours. The results folder will contain all the crashes it encounters and there will also be core dumps we can analyze.