Purpose


Fuzz testing is a technique in software testing using random and unexpected inputs to identify vulnerabilities and data handling gaps in a program. Many of these detectable errors, like buffer overflow, can have serious security implications. These tests can be automated and run periodically as part of CI/CD. There is also a requirement from one of our customers to fuzz test our software.


Solutions


There are different ways to fuzz test userspace programs and libraries and kernel modules. We investigated and tested two types of solutions to fuzz test userspace programs. There is also a kernel space fuzzer, but its more involved and we have not tested it. It also works by fuzzing systemcall/ sysfs etc. inputs:

Lib-fuzzer ([LibFuzzer](https://llvm.org/docs/LibFuzzer.html)): This is a built-in feature of LLVM. We would compile our program using clang with specific compiler options. The result is an executable which we run and the llvm runtime automatically passes fuzzed inputs to the function we are testing. This works like a Unittest, so we neeed to implement the functioality by writing fuzz-tests in C (or whatever the language our programs and libraries are written). The full coverage will take a long time and is considerable development effort.

AFL++ ([AFL++](https://aflplus.plus/)): AFL++ is more advanced but easier to use version of fuzzing engine. Here we don't have to write custom tests. Instead we just need to compile our userspace programs and libraries using their compiler wrapper. These compilers are compatible to gcc and clang. We can provide the engine with a large sets of inputs and the engine automatically fuzz tests it. We can run this periodically as part of our CI/CD and its not a big effort to start running it.

syzkaller ([Kernel Fuzzer](https://github.com/google/syzkaller)): This can be used to fuzz kernel modules like lnet drivers. Needs more investigation and test to figure out how to use it.


We can start with AFL++ as its very straightforward to get it up and running and its very good as finding issues.


Initial Results


We have run both Lib-Fuzzer and AFL test in lnetctl command and within a few minutes of running it has already identified issues like double free, buffer overflow etc. These could be potentially exploited by hackers. So it's a very good idea to run it periodically and identify issues and fix them.

Setup


We need a program that would run periodically in our CI/CD. it needs to.
* Clone and build AFL++ ([AFL++ Github repo](https://github.com/AFLplusplus/AFLplusplus))
* Clone Lustre Release
* Since this Compiler wrapper does not run with kernel modules, we need to run ./configure rormally.
* Make the utilities using AFL++ compiler wrapper.
`CC=afl-gcc-fast make utils`
* Before running the test, create an input folder and create files with input parapeters that we normally run.
eg: echo "ping --sources=192.168.64.5@tcp" testcases/ping.txt"
We can have as many of these test cases of the utility we are testing.
* run the test:
eg: `afl-fuzz -i ./testcases -o ./results lnetctl`
* It will run the tests feeding randomized inputs and it can run for hours. The results folder will contain all the crashes it encounters and there will also be core dumps we can analyze.


  • No labels

6 Comments

  1. Andreas Dilger

    Manish, the Lustre/LNet testing is driven by the autotest framework, like lustre/tests/sanity-lnet.sh and lustre/tests/lnet-selftest.sh, so this is the right place to add the fuzz testing as well. 

    I think the right approach would be for normal Lustre builds to create a separate "fuzz testing version" of lnetctl-fuzz and lctl-fuzz and lfs-fuzz and lst-fuzz that are linked with the fuzzing libraries calling the syscalls and ioctls with fuzzed parameters during testing. These should be part of the lustre-tests RPMs, rather than having a wholly separate build for this testing. 

    I do think it will be important to have something like syzkaller calling the syscalls and ioctls with appropriately-fuzzed parameters (not just random garbage, since that will mostly be rejected).

     Also, we need to have fuzzing at the client RPC level, and not just the syscall level, since that is what will be exposed on the servers. 

    1. Manish Regmi

      Bundling the AFL compiled version of utils in a tests RPMs sounds like a good idea to me. But we don't want to run it as part of normal CI/CD which triggers at each PR/commit. The AFL tests takes hours and would be better to run periodically (like daily) to see new issues.

      Also we will need some kind of script to separate duplicate issues . A lot of the crashes has same signature.

      As for kernel level fuzz, i am still investigating.

  2. Serguei Smirnov

    If I remember correctly what was mentioned in earlier discussions, another way to get going with some form of LNet fuzz testing at kernel level may be harnessing the existing "fault injection" interface via lnetctl to instruct LNet to "fuzz" incoming data before handing it. 


  3. Andreas Dilger

    Serguei Smirnov I was thinking that the existing LNet delay/drop mechanism could also be used to inject intermittent data errors into the packets in strategic places.  This shouldn't necessarily be verified for "correctness" (though some fraction of errors will already be caught by checksums), but rather for detecting bad data handling errors/crashes (like racer).

  4. Cyril Bordage

    I took a quick look into AFL++ and I don't see how it can help in simulating bad inputs from network. Is it something we can do with it? Is it something we would like to handle in the same way / at the same time?

    In case on network errors, I think using existing fault injection is not enough, and could be very tedious. We need something more powerful to be able to test several defined inputs (and also random ones).
    For instance, in DDN-4691, various fields in the message are wrong and thus, some checks are skipped leading to the LASSERT error, and using current fault injection is not convenient.

  5. Serguei Smirnov

    As discussed with Frank, another option for lnet-level fuzz testing may be the introduction of a special "fuzz" mode for an lnet router, enabling the router to modify the traffic it is passing through to add the "fuzz"