Back in Summer 2019 we had a workweek full of tech talks and presentations, where we explored various topics from advanced DevOps practises to biometric engines. We had eeeverything. Of course testing-me had to live up to his urge and enrolled to all listed talks regarding his favorite IT-discipline: System Design Processes, Enterprise-Scale QA… and then there was that particular presentation about a simple yet effective test automation technique called fuzzing.
That one got me. I listened with an evil grin and decided to give it a shot. And that’s what we are going to do today!
Fuzzing – as explained in the talk – is a testing technique, that feeds the application a huge amount of random input data with different types and checks, which of them crashes the application. Simple enough. This can happen in a Black Box fashion by bombarding the public API or in a more White Box fashion by instrumenting the application code in order to get even more coverage and insides-related details.
In today’s tutorial, we will go through a Black Box Fuzzing setup written in Java. That’s because I’m more of a Black Box Testing person, and my main field of action is Java-based enterprise applications. More exactly, we will prepare a happy little Play Framework-based web application, that somehow got a commercial 3rd party conversion library called „Legacy“ imposed upon. Next we QAs want to have a first glance at Legacy’s state of quality to see, whether the purchase was at least somewhat worth it.
Prerequisites
This tutorial assumes that you have sbt and Maven installed. Since I wanted to try the Play Framework as a nice little side learning, we have to get along with sbt, but don’t worry: We need it only to compile the app. If you are curious, you can use it to run the app, too, but that’s 100% optional.
Maven on the other hand is used to operate the fuzz tests and thus will be our bread and butter tool.
Our Setup
Here’s the link to Happy Little Webapp’s source code repository. In ./app you can find the code of our Legacy-Module next to the web app’s controllers and (unused) views. Technically it’s not a blackbox, since I had to write the example code by myself, but let’s assume, we as the testers don’t know anything about it’s details, except for the public methods‘ signatures.
First, open the sbt shell: In your terminal of choice, enter the command sbt
. Next, in the sbt-shell we just opened, we enter compile
to compile the app’s code. Afterwards, if you are curious about what the app actually does, you can type run
to start it. Now you can perform a request in your browser like:
http://localhost:9000/dollar2euro/58
It should display 53.36
. Not as correct as we would expect it to be, because the used factor for the calculation is static and likely outdated. But for testing purposes, let’s assume, it is sufficient.
Next, we take care of our fuzz tests located in ./fuzztests. The pom.xml already knows about their location, so by using it, we can execute the tests right away. The fuzzing will be executed with a maven plugin called jqf-fuzz. Please see its github repository for the code and its well-elaborated documentation. With all that coming together, we are ready to fuzz.
Get the fuzzing started
First, we have to install the jqf-fuzz Maven plugin by doing a simple:
mvn clean test-compile
This downloads the jqf-fuzz plugin to our local maven repository and compiles the test sources. Now we have access to 2 new maven goals: jqf:fuzz
executes the fuzz tests, and jqf:repro
replays failed test cases to hunt down associated defects. Both goals expect several input parameter defined by JVM parameters (-D on the CLI) and/or by definition within the POM. This allows for a rich set of customization, that is both user- and CI-friendly. For demonstration purposes, I already configured the parameter time
in the POM so that the test runs for 10 seconds, that still provides us with lots of input. Further, I predefined the fuzz test class to be executed. Therefore, the only parameter we must provide from the terminal is our test method -Dmethod=dollar2euro
. We will do that in a minute, but first let’s have a look at the fuzz test class.
Let’s run the test
This is what we gonna unleash upon our web app:
@RunWith(JQF.class)
public class LegacyConverterFuzzer {
private static LegacyConverter legacyConverter;
@BeforeClass
public static void beforeClass(){
legacyConverter = new LegacyConverter();
}
@Fuzz
public void dollar2euro(Object input){ // this is where the fun things happen
try {
System.out.println("Input: " + input.toString());
System.out.println("Output: " + legacyConverter.dollar2euro(input));
} catch (Throwable e) {
System.out.println(e.getClass().getName() + ":" +
e.getMessage());
}
}
[... some more Fuzz-Tests, please see the repository linked above...]
}
Legacy’s executives promised, that any input works fine. Okay! Then we perform the test dynamic-typed by using an Object
-typed input parameter.
Alright, that’s the code. Let’s fire it up. On your terminal, please do:
mvn jqf:fuzz -Dmethod=dollar2euro
Here’s an excerpt from the results as seen in my terminal. The output will vary for each new test run, because, as we said earlier, the input values in fuzz tests are random.
java.lang.NumberFormatException: Character 텈 is neither a decimal digit number, decimal point, nor "e" notation exponential mark. Input: edu.berkeley.cs.jqf.fuzz.junit.quickcheck.InputStreamGenerator$1@4fc3c165 java.lang.NumberFormatException: Too many nonzero exponent digits. Input: 뤇皽 java.lang.NumberFormatException: Character 뤇 is neither a decimal digit number, decimal point, nor "e" notation exponential mark. Input: ky java.lang.NumberFormatException: Character k is neither a decimal digit number, decimal point, nor "e" notation exponential mark. Input: FixedClock[+898773291-08-05T17:23:55.165612278Z,UTC] java.lang.NumberFormatException: Character array is missing "e" notation exponential mark. Input: -8475850143961316955 Output: -7797782132444411598.60 Input: bn java.lang.NumberFormatException: Character b is neither a decimal digit number, decimal point, nor "e" notation exponential mark. Input: 16:19:25.242056065Z java.lang.NumberFormatException: Character array is missing "e" notation exponential mark. Input: -895394919-05-23T23:50:04.780324820 java.lang.NumberFormatException: Character array is missing "e" notation exponential mark. Input: 11:14:21.890848137Z
Phew! We got a lot of NumberFormatExceptions. So much about „any input works“. Our PO should know about that.
6 months full of arguments later, the supplier delivered API version v1.0.1 of his LegacyConverter ensuring a static-typed API. He changed dollar2euro to the following:
public String dollar2euro(BigDecimal input){
BigDecimal dollars = input.setScale(2, BigDecimal.ROUND_HALF_EVEN);
[...]
}
Of course, we have to adapt our controller, too. For playground reasons, we keep that change as simple as possible.
public Result dollar2euro(String dollars) {
return ok(importantConverter.dollar2euro(
BigDecimal.valueOf(Double.valueOf(dollars))));
}
When we enter non-numeric inputs, the app will still fail, but at least it’s on us now.
Alright, the fixes are applied. Now in our test class, we see a sweet little type check error: We have to change the test method’s input parameter’s type accordingly to BigDecimal
, too. This makes our fuzz test static-typed.
Afterwards, we recompile the tests and repeat the fuzz:
mvn clean test-compile
mvn jqf:fuzz -Dmethod=dollar2euro
giving us (excerpt):
Input: 152 Output: 139.84 Input: -1000 Output: -920.00 Input: -771298122 Output: -709594272.24 Input: 80372941329620235 Output: 73943106023250616.20 Input: 272536 Output: 250733.12 Input: -1000 Output: -920.00 Input: -2625164447481769740006272317 Output: -2415151291683228160805770531.64 Input: 9340202544 Output: 8592986340.48 Input: -34567 Output: -31801.64 Input: 17223398969630190416957297 Output: 15845527052059775183600713.24
Much better!
Conclusion – What did we achieve by fuzzing?
We have seen, how we can use fuzzing to create a vast storm of static or dynamic-typed test inputs and thus create hundreds of different test cases. From the output logs we can learn, what inputs can be handled by our application and – more interesting – what not. This provides us with an insightful first glance at the quality, a great starting point for further functional test cases, and, of course, with even more application bombing by using our favorite CI system.
From here, we can follow the functional testing track with even more elaborated automation or dive deeper into Java Fuzzing with the official JQF-Fuzz paper. And if you still need motivation to automate your tests, check out here why test automation is cool. Also there is a new tutorial about threads in Java fresh out of the oven. Enjoy it while it’s hot!
Last but not least a huge shoutout to the great people at X41 D-SEC, who held the exciting talk that inspired me and made me put fuzzing into my tool box.
Danke fuer die Info