Fuzzing in Java – How and Why

Back in Summer 2019 we had a workweek full of tech talks and presentations, where we explored various topics from advanced DevOps practises to biometric engines. We had eeeverything. Of course testing-me had to live up to his urge and enrolled to all listed talks regarding his favorite IT-discipline: System Design Processes, Enterprise-Scale QA… and then there was that particular presentation about a simple yet effective test automation technique called fuzzing.

That one got me. I listened with an evil grin and decided to give it a shot. And that’s what we are going to do today!

Fuzzing – as explained in the talk – is a testing technique, that feeds the application a huge amount of random input data with different types and checks, which of them crashes the application. Simple enough. This can happen in a Black Box fashion by bombarding the public API or in a more White Box fashion by instrumenting the application code in order to get even more coverage and insides-related details.

In today’s tutorial, we will go through a Black Box Fuzzing setup written in Java. That’s because I’m more of a Black Box Testing person, and my main field of action is Java-based enterprise applications. More exactly, we will prepare a happy little Play Framework-based web application, that somehow got a commercial 3rd party conversion library called “Legacy” imposed upon. Next we QAs want to have a first glance at Legacy’s state of quality to see, whether the purchase was at least somewhat worth it.

Prerequisites

This tutorial assumes that you have sbt and Maven installed. Since I wanted to try the Play Framework as a nice little side learning, we have to get along with sbt, but don’t worry: We need it only to compile the app. If you are curious, you can use it to run the app, too, but that’s 100% optional.

Maven on the other hand is used to operate the fuzz tests and thus will be our bread and butter tool.

Our Setup

Here’s the link to Happy Little Webapp’s source code repository. In ./app you can find the code of our Legacy-Module next to the web app’s controllers and (unused) views. Technically it’s not a blackbox, since I had to write the example code by myself, but let’s assume, we as the testers don’t know anything about it’s details, except for the public methods’ signatures.

First, open the sbt shell: In your terminal of choice, enter the command sbt. Next, in the sbt-shell we just opened, we enter compile to compile the app’s code. Afterwards, if you are curious about what the app actually does, you can type run to start it. Now you can perform a request in your browser like:

http://localhost:9000/dollar2euro/58

It should display 53.36. Not as correct as we would expect it to be, because the used factor for the calculation is static and likely outdated. But for testing purposes, let’s assume, it is sufficient.

Next, we take care of our fuzz tests located in ./fuzztests. The pom.xml already knows about their location, so by using it, we can execute the tests right away. The fuzzing will be executed with a maven plugin called jqf-fuzz. Please see its github repository for the code and its well-elaborated documentation. With all that coming together, we are ready to fuzz.

Get the fuzzing started

First, we have to install the jqf-fuzz Maven plugin by doing a simple:

mvn clean test-compile

This downloads the jqf-fuzz plugin to our local maven repository and compiles the test sources. Now we have access to 2 new maven goals: jqf:fuzz executes the fuzz tests, and jqf:repro replays failed test cases to hunt down associated defects. Both goals expect several input parameter defined by JVM parameters (-D on the CLI) and/or by definition within the POM. This allows for a rich set of customization, that is both user- and CI-friendly. For demonstration purposes, I already configured the parameter time in the POM so that the test runs for 10 seconds, that still provides us with lots of input. Further, I predefined the fuzz test class to be executed. Therefore, the only parameter we must provide from the terminal is our test method -Dmethod=dollar2euro. We will do that in a minute, but first let’s have a look at the fuzz test class.

Let’s run the test

This is what we gonna unleash upon our web app:

@RunWith(JQF.class)
public class LegacyConverterFuzzer {

private static LegacyConverter legacyConverter;

@BeforeClass
public static void beforeClass(){
    legacyConverter = new LegacyConverter();
}

@Fuzz
public void dollar2euro(Object input){ // this is where the fun things happen
   try {
       System.out.println("Input: " + input.toString());
       System.out.println("Output: " + legacyConverter.dollar2euro(input));
   } catch (Throwable e) {
       System.out.println(e.getClass().getName() + ":" + 
                          e.getMessage());
   }
}

[... some more Fuzz-Tests, please see the repository linked above...]
}

Legacy’s executives promised, that any input works fine. Okay! Then we perform the test dynamic-typed by using an Object-typed input parameter.

Alright, that’s the code. Let’s fire it up. On your terminal, please do:

mvn jqf:fuzz -Dmethod=dollar2euro

Here’s an excerpt from the results as seen in my terminal. The output will vary for each new test run, because, as we said earlier, the input values in fuzz tests are random.

java.lang.NumberFormatException: Character 텈 is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: edu.berkeley.cs.jqf.fuzz.junit.quickcheck.InputStreamGenerator$1@4fc3c165
java.lang.NumberFormatException: Too many nonzero exponent digits.
Input: 뤇皽
java.lang.NumberFormatException: Character 뤇 is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: ky
java.lang.NumberFormatException: Character k is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: FixedClock[+898773291-08-05T17:23:55.165612278Z,UTC]
java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
Input: -8475850143961316955
Output: -7797782132444411598.60
Input: bn
java.lang.NumberFormatException: Character b is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: 16:19:25.242056065Z
java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
Input: -895394919-05-23T23:50:04.780324820
java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
Input: 11:14:21.890848137Z

Phew! We got a lot of NumberFormatExceptions. So much about “any input works”. Our PO should know about that.

6 months full of arguments later, the supplier delivered API version v1.0.1 of his LegacyConverter ensuring a static-typed API. He changed dollar2euro to the following:

    public String dollar2euro(BigDecimal input){
        BigDecimal dollars = input.setScale(2, BigDecimal.ROUND_HALF_EVEN);
[...]
}

Of course, we have to adapt our controller, too. For playground reasons, we keep that change as simple as possible.

    public Result dollar2euro(String dollars) { 
        return ok(importantConverter.dollar2euro(
                  BigDecimal.valueOf(Double.valueOf(dollars))));
    }

When we enter non-numeric inputs, the app will still fail, but at least it’s on us now.

Alright, the fixes are applied. Now in our test class, we see a sweet little type check error: We have to change the test method’s input parameter’s type accordingly to BigDecimal, too. This makes our fuzz test static-typed.

Afterwards, we recompile the tests and repeat the fuzz:

mvn clean test-compile
mvn jqf:fuzz -Dmethod=dollar2euro

giving us (excerpt):

Input: 152
Output: 139.84
Input: -1000
Output: -920.00
Input: -771298122
Output: -709594272.24
Input: 80372941329620235
Output: 73943106023250616.20
Input: 272536
Output: 250733.12
Input: -1000
Output: -920.00
Input: -2625164447481769740006272317
Output: -2415151291683228160805770531.64
Input: 9340202544
Output: 8592986340.48
Input: -34567
Output: -31801.64
Input: 17223398969630190416957297
Output: 15845527052059775183600713.24

Much better!

Conclusion – What did we achieve by fuzzing?

We have seen, how we can use fuzzing to create a vast storm of static or dynamic-typed test inputs and thus create hundreds of different test cases. From the output logs we can learn, what inputs can be handled by our application and – more interesting – what not. This provides us with an insightful first glance at the quality, a great starting point for further functional test cases, and, of course, with even more application bombing by using our favorite CI system.

From here, we can follow the functional testing track with even more elaborated automation or dive deeper into Java Fuzzing with JQF-Fuzz by checking out its paper. And if you still need motivation to automate your tests, check this one out.

Last but not least a huge shoutout to the great people at X41 D-SEC, who held the exciting talk, that inspired me and made me put fuzzing into my tool box.

Playwright for Browser Automation

Last week I held a short & sweet presentation in the company about the usage, benefits and drawbacks of Browser Remote Debugging APIs. One unfortunate problem we discovered was the lack of a standard across the browsers; every major browser maintains its very own implementation. The RemoteDebug – Intitiative tried to solve this problem, but until now without noticeable success, as you can see here by the lack of activity. Therefore, the Test and Development – World needed to deal with that all by themselves. A great team of ex Puppeteer-developers, who moved from Google to Microsoft, did exactly that by bringing us Playwright, a framework for writing automated tests encapsulating and using the various Remote Debugging Interfaces. In today’s short example we write a quick example test with Playwright.

Installing Playwright

As a starting prerequisite, we need a NodeJS-Distribution with Version 10 or greater. Next, we go to our already well-filled project directory and create a new NodeJS-Project:

$ cd /path/to/your/project/directory
$ mkdir playwright_test && cd playwright_test
$ npm init
$ npm install --save-dev playwright

While the installation progresses, you will notice that Playwright brings its own browser binaries. Don’t worry about that, they are still perfectly valid, as the rendering engines are not modified at all. Only the debugging capabilities have been given a few extensions.

Alright, that’s all we need.

Time to dive into the code!

Let’s assume we want to buy red shoes on Amazon, because we need new shoes, and red is a nice color.

// 1. We start by initializing and launching a non-headless Firefox 
// for demo purposes.
// (How do you call them, "headful"? "headded"? Feel free to drop me 
// your best shots. :))
const {firefox} = require("playwright");

(async () => {
  const browser = await firefox.launch({headless: false, slowMo: 50});
  const context = await browser.newContext();

  // 2. Next, we head to the Amazon Landing Page...
  const page = await context.newPage();
  await page.goto("https://www.amazon.com");
  
  // 3. ...do the search for Red Shoes...
  await page.fill("#twotabsearchtextbox", "Red Shoes");
  const searchBox = await page.$("#twotabsearchtextbox");
  await searchBox.press("Enter");

  // 4. ...and take a nice deep look at the page 
  // by saving a screenshot.
  await page.waitFor("img[data-image-latency='s-product-image']");
  await page.screenshot({path: "./screenshot.jpg", type: "jpeg"});
  
  // 5. Afterwards, we leave the testrun with a clean state.
  await browser.close();
})();

That’s it for now. From here, we can extend the test by doing elaborate verification steps, check out a nice pair of red shoes and pay them with our hard-earned testing money.  Feel free to check out the example’s full source code from here and go ham.

Conclusion

With Playwright we got a means to write automated tests with ease against the many different Remote Debugging APIs. It copes nicely with the API differences while preserving an intuitive and familiar JS test automation syntax.

So if you are looking for a more lightweight and lower level alternative to Selenium, give it a go!

Zalenium in a minimal Docker Compose – Setup

What is Zalenium?

Zalenium, brought to us by German online fashion retailer Zalando, is a feature-enriched Selenium test platform based on the popular Selenium Grid. Besides the core features like scaling Selenium test execution nodes, it provides nice things like video recording, a video player directly in the management UI and integrations with popular browser test tools like Sauce Labs. For a more detailed overview, please check out the project page. As far as we are concerned here, we have all the good arguments we need to fire up a small test setup. 🙂

What are we going to do?

In the following miniworkshop, we temporarily slip into the shoes of a devops engineer and set up a minimal Zalenium Grid – environment in order to execute remote Selenium tests there. The goal is that we use no more than 2 files (of resonable size):

  • the docker-compose-file to build and start the Zalenium-container provided by Zalenium
  • a sample selenium-webdriver-test to be executed inside Zalenium, kindly provided by Felipe Almeida, thank you very much.

For our experiment, I modified the latter to enable remote driver execution instead of starting a local firefox. Therefore, I prepared everything in a small bitbucket-repo.

Prereqs for the Zalenium Setup

  • a recent version of Docker (should already include docker-compose)
  • Ruby > 2.3.1 (I recommend using RVM)
  • a recent Chrome-browser
    • Unfortunately, my Firefox (v67.0.4) does not support the video format of the test execution recordings. 🙁

Steps

  1. Open a terminal and clone the repo.
  2. cd inside the new directory and fire up the containers: $ docker-compose up -d
  3. Start the test: $ ruby selenium_minimal.rb
  4. After the test execution, open a Chromeand head to the Dashboard: http://localhost:4444/dashboard/
  5. You should see one test execution in the list on the left side. Click it.
  6. Play the video and enjoy the action of your test.

Conclusion

Now that you have the power to quickly fire up Zalenium and its grid nodes, you can go further. Host it on a remote machine serving your needs as a Test Automaton Engine, move it to the cloud and go to town. This should step up your Quality Assurance Game in a scalable and easily maintainable way. Have fun!

3 Perks of being a Technical QA

The IT world flourishes and blooms superfast. New products and services pop up on a daily basis. We all know and work with these trends: while product managers want to ship new product features blazing fast, their fellow QA department invests time to “step back” and make sure the shipping product increment does, what it is supposed to do. This of course creates a fragile balance in the release schedule. Now if we want to optimize the balance by applying upfront techniques like code reviews, TDD and test automation, the role of QA undergoes a slow but steady change. In my career, i saw a growing number of job offers in manual QA requiring more extensive technical knowledge. At least basic test automation skills and abilities to understand code are often required nowadays.

Following, I’d like to use my very first blog post to point out, why it is a good thing to have the mentioned technical skills in your QA career.

Almost 3 years ago, I joined the ranks of the QA as a Test Automation Engineer. Since then, I worked for many different projects & people in different roles. I met Lead Developers,  testers and CTOs of various companies. One of my most notable projects was the Logistics Department of a major food delivery company, which has a huge impact on the business, and therefore a great demand for quality. The team consisted almost only of developers and one (part-time) product manager. The logistics monitoring tool the team built is based on Ruby on Rails; a huge advantage for me, because my test automation stack is based on Ruby & Cucumber, that fits neatly into Rails. With this setup in mind, lets dive into the real questions: how can technical proficiency help you increase quality?

1. Static Code Analysis & Code Reviews

Highly underestimated in the testing departments, CR-techniques bring quality on a detailed level to the desk. Static analysing tools like Codeclimate or Scrutinizer already automate a great share of the efforts and provide the development team and you with valuable metrics and advice. With code reviews, you can complement these valuable results even more. Usually a task for developers, you as a QA can bring a different, more destructive, perspective to code discussions. This way, you may discover additional potential issues or convention breaks, that the developers may have not seen. Most Git-Providers come with easy diff tools to reduce your effort making all these nice things very cheap.

2. The Glue between Dev, QA & Product

As a technical QA you have many different insights at your disposal. When it comes to  typical discussions in every day’s sprint life, you are in close contact to the manual testers  leveraging yourself to absorb everything from the product design side. Then you can transfer the knowledge back to Dev and, equally interesting, vice versa. You can estimate & answer relevant questions from multiple perspectives.

3. Openings for more

Tech enables you to deliver even more forms of quality. For one project, I coded Performance- and Stress Tests directly in the Rails app using ruby-jmeter and made them part of the Rake-Scripts. Rake is the conventional build system similar to Make for Ruby on Rails – Projects. This enabled me to execute them on demand and could even serve next to your automated functional texts in  scheduled test execution environments. All that while they are managed directly in the project’s Git-Repo as the source code files they are. Another big opportunity is release management, where you can play a big role. As scheduled or on-commit test executions play a major role in answering the question “Can we deploy?”, you are able to actively shape the CI/CD flow directly with the CI/CD-Tool of your choice bringing you directly to the heart of the release process for a lot of companies.

Technical QA – hot or not?

If you feel a knack for the technology around you, and if you want to explore that nefarious destructive glare in your eyes, I absolutely recommend you to go for it. You have nothing to lose and lots of interesting opportunities to gain. I made the experience that the entry is pretty smooth, and the basic technologies around Selenium or Jenkins for example are well known, documented and supported. Therefore, dive into it and prepare for adventure. 🙂

You want to have a smooth quickstart? Check out my minimal Zalenium Setup!