Cucumber in Rust – Beginner’s Tutorial

Introduction

When I started my first QA role back in 2014, my first tasks included the maintenance and extension of a large test base, that was supposed to work for 4 different projects in parallel. It was based on Cucumber and the Ruby programming language, a stack I fell more and more in love with. This love still lasts to this day.

Therefore, it is time to relive the feeling, that is working with Cucumber from a fresh perspective, once more. To achieve this feeling we are going to apply an interesting little twist: We will code and test in the Rust programming language.

Rust, Ruby. 4 Letters and a capital R. Perfect!

What is Cucumber?

Cucumber is a framework, that implements Behavior Driven Development. The rules of BDD can be summarized as formulizing the requirements step by step in a more and more technical way. We start with the written requirements by your fellow business department and reformulate the requirements into a machine-readable format. Next, we use this text version to write an automated test case, that fails, and implement the feature up until the test passes. This flow gives it the popular resemblance to Test Driven Development. Cucumber leverages BDD by providing the machine- and human-readable layer based on so-called feature files. These use the Gherkin syntax, a simple syntax based on the keywords Given, When, Then, And and But.

Cucumber is still widely used as a test runner, although BDD is rarely actually implemented due to the all-time-popular time limitation in nowaday’s software projects. Another similarity to TDD, that is rather unfortunate.

What is Rust?

Rust is a fairly new and rising system level programming language, that operates in the same markets as C++ and friends. Besides system-level performance, its main focus lies in security and safety being builtin. Furthermore, due to its security and safety-heavy design architecture, it is able to completely omit automated memory management. It just doesn’t need it, while still guaranteeing memory safety.

All of these points are topped off by an exceptional developer experience: The Rust toolchain brings its full-fledged API documentation and its popular text book right to your command line-operating finger tips, and even compiler errors are designed as tiny educational lessons.

Our Test Object: A Simple AES Encryption Tool

I coded my first working Rust app, when I was learning its renowned ownership and borrow model. Usually, when it comes to the First App ™, I tend to write Fibonacci calculators in all kinds of setups: Fibonacci REST APIs, Fibonacci CLI calculators, Fibonacci FFI libs inside a Flutter app… But this time, i wanted something different. Something, that actually does stuff on a level worthy to let it be called a „system application“. So I decided to write a simple AES string encryption tool, that I gave the unspeakably cute name „Encrsypter“. It is based on aes-gcm, an AES encryption library (or „crate“ in Rust terms), that got audited successfully by the nccgroup a few months ago.

The full source code is available here. [Update: I’m currently working on an updated post with the new Cucumber-rs version. For this tutorial, please checkout the branch cukes_0.6.0 and, of course, stay tuned for the update. 😉]

For training purposes, I recommend removing the tests/ directory, because we will successively build it up, as we go through the tutorial.

Writing Cucumber-based Tests

Before we add the sources for our test cases, let’s be aware of the test object’s project layout. We will start with the following directories and files:

Cargo.lock, Cargo.toml, src/, constants.rs, decryptor.rs, encryptor.rs and main.rs. No Cucumber tests yet.
encrsypter’s project directory without Cucumber tests

Before we can code the test, we must add a cargo-compatible subproject structure. On your favorite command line, please create the following directories with these terminal commands (all directories relative to the project root):

mkdir tests
mkdir tests/features
mkdir tests/steps

We will create and store our feature file, that specifies the test steps of our Cucumber test, in the features/ subdirectory, whereas the steps‘ implementations will go to steps/. But first of all, we will prepare the central configuration. As described in the official documentation, we create a file called cucumber.rs in tests/ with the following content:

#[path = "../src/encryptor.rs"] mod encryptor;
#[path = "../src/decryptor.rs"] mod decryptor;
#[path = "./steps/encrypt_decrypt_steps.rs"] mod encrypt_decrypt_steps;
use cucumber::cucumber;
use std::borrow::Cow;

pub struct World<'a> {
    encryptor: encryptor::Encryptor<'a>,
    decryptor: decryptor::Decryptor<'a>,
    encrypted_base64: String,
    decrypt_result: String
}

impl cucumber::World for World<'_> {}
impl std::default::Default for World<'_> {
    fn default() -> World<'static> {
        let key = &[1; 32];
        let nonce = &[3; 12];

        World { encryptor: encryptor::Encryptor{ input: Cow::Borrowed(""), key, nonce },
                decryptor: decryptor::Decryptor{ file_path: "./testfile.txt", key, nonce },
                encrypted_base64: "".to_string(),
                decrypt_result: "".to_string()
        }
    }
}

cucumber! {
    features: "./tests/features/", // Path to our feature files
    world: crate::World, // The world needs to be the same for steps and the main cucumber call
    steps: &[
        encrypt_decrypt_steps::steps // the `steps!` macro creates a `steps` function in a module
    ]
}

The World struct contains the mutable instances of our test objects: The encryptor and decryptor, that serve to encrypt and decrypt messages using AES. Further, we will maintain special fields to keep track of their respective outputs. The cucumber! block serves as our entry point, where we perform the basic configuration, that gets our Cucumber test up and running: We…

  • … tell Cucumber where to find feature files.
  • … specify the test’s World struct, that contains our test objects.
  • … declare the module, that contains our step implementations.

The next part of configuration is done in the project’s Cargo.toml. Again according to the official documentation, we should specify dependencies and a test directive like this:

[[test]]
name = "cucumber"
harness = false # Allows Cucumber to print output instead of libtest

[dev-dependencies]
cucumber = { package = "cucumber_rust", version = "^0.6.0" } 
base64 = "0.12.3"

In terms of dependencies, we need the cucumber_rust package to run our tests, then we need the base64 package, because we will work with and do assertions on raw bytes. Although not entirely necessary, it comes in handy for visualisation purposes.

Under [[test]], we give our Cucumber test a name, and we route execution outputs to stdout. We will see its use later, when we finally come to the executing part.

Alright, the config is done. Now we are ready to specify our first test. We will encrypt a small „Hello World!“ message, give it a rough sanity check, and then we decrypt it back and hope, that the decrypted output matches our input. Under ./tests/features, please create the file encryptor.feature. The containing test specification should roughly look like this:

Feature: Encrypt messages and write them to a file.

  Scenario: Encrypt a simple Hello World - message.
    Given I have an encryptor initialized with input "Hello World!"
     Then I should see "Hello World!" in the test encryptors input field
     When I encrypt the Encryptor's input
     Then testfile.txt exists
      And testfile.txt is not empty
     When I decrypt testfile.txt
     Then the decrypted result should be "Hello World!"

This describes, what we want to accomplish; we want to encrypt the string „Hello World!“, check, whether the output is there and whether it is not completely broken. Then we want to decrypt that output back and check, whether the output is the same as our input message. Next, we have to actually automate this test by implementing the Givens, Whens, Thens and Ands.

Step Implementation Files

So far we have told Cucumber, where to find its stuff, and we created a written test specification. Great, we are almost there. The last step is to weave the magic into the Gherkin steps, that do the heavy lifting, when Cucumber reads a step in the current feature file. Lets check out the following example step and see, what that means:


when "I encrypt the Encryptor's input" |world, _step| {
    world.encryptor.write_encrypted();
};

This means whenever the Cucumber engine finds a step, that matches „When I encrypt the Encryptor’s input“ inside the feature file, the code within the closure is executed. Here, we encrypt some random text.

Regular expressions are usable, too:

given regex r#"^I have an encryptor initialized with input "([\w,\s,!]+)"$"# (String) |world, text_to_encrypt, _step| {
        // the # are necessary to prevent the inner quotations marks as part of the String
        world.encryptor.input = Cow::Owned(text_to_encrypt);
    };

This step defines the text, that we want to encrypt using the When step from above. Here, the text is derived from the feature file by matching the regular expression in r# and the enclosing capture group ([\w,\s,!]+). The value, that was read by the capture group, goes to the closure parameter after world, in this case text_to_encrypt. Note that the „r“ in r# stands for „raw string“ instead of „regular expression“. Raw strings are a means to spare us from copious amounts of escape slashes within the regular expression string; otherwise, they are regular strings. I won’t go into too much detail here. If you want to learn more about them, check out this post.

By using the regular expression above, we could have written the steps in our feature file like the following:

Given I have an encryptor initialized with input "Hi I am Floh"
=> encryptor input is "Hi I am Floh"

Given I have an encryptor initialized with input "99 bottles of beer on the wall…"
=> encryptor input is "99 bottles of beer on the wall…"

Given I have an encryptor initialized with input "Your ad here"
=> encryptor input is "Your ad here"

Putting all the knowledge together, here is the sample implementation for our test steps. Please put it into ./tests/steps/encrypt_decrypt_steps.rs (related to the project root).

use cucumber::steps;
use std::fs;
use std::path::Path;
use std::borrow::Cow;

steps!(crate::World<'static> => {
    given regex r#"^I have an encryptor initialized with input "([\w,\s,!]+)"$"# (String) |world, text_to_encrypt, _step| {
        // the # are necessary to prevent the inner quotations marks as part of the String
        world.encryptor.input = Cow::Owned(text_to_encrypt);
    };

    then regex r#"^I should see "([\w,\s,!]+)" in the test encryptors input field"# (String) |world, expected_text, _step| {
        assert_eq!(expected_text, world.encryptor.input);
    };

    when "I encrypt the Encryptor's input" |world, _step| {
        world.encryptor.write_encrypted();
    };

    then "testfile.txt exists" |_world, _step| {
       let testfile_path = Path::new("./testfile.txt");
       assert_eq!(testfile_path.exists(), true);
    };

    then "testfile.txt is not empty" |world, _step| {
        let enc_message = fs::read("./testfile.txt").expect("Could not read test file.");
        world.encrypted_base64 = base64::encode(&enc_message);

        assert_eq!(world.encrypted_base64.len() > (0 as usize), true);
    };

    when "I decrypt testfile.txt" |world, _step| {
        world.decrypt_result = world.decryptor.read_decrypted();
    };

    then regex r#"^the decrypted result should be "([\w,\s,!]+)"$"# (String) |world, expected_text, _step| {
        assert_eq!(expected_text, world.decrypt_result);
    };
});

Now we are ready for the first test run. Please execute the following in your favorite terminal:

cargo test --test cucumber

If all goes well, it shows us a positive test result:

Positive result of our Cucumber test. 1 fearture with 1 scenario containing 7 steps, all green and checkmarked in the terminal.
The test passed. Yay!

Conclusion: Cucumber in Different Languages

This is by no means the end of Cucumber’s options and possibilities. There are many many many more well maintained ports for many different platforms out there. Not all of them may be offcial, e.g. the Rust port we used today, but they are nonetheless maintained and fully functional. And they contributes to its well deserved popularity as well as the official ports. This is what counts in the end.

For Cucumber-rs it’s not the end of possibilities, too, as version 0.7 has been released recently. It brings asynchronous test support and a new builder-based approach to the table. I’m hyped to try it out, especially because I love asynchronous coding (Please don’t judge me..)

But for now, this is a good starting point to read more about other facettes of test automation. For example, you can learn how to set up a Zalenium cluster for distributed browser UI testing. It is well-suited to be combined with Cucumber. If you’d rather learn more about unconventional and unstructured automation testing, you might like my article about fuzzing in Java. Have a great day!

Fuzzing in Java – How and Why

Back in Summer 2019 we had a workweek full of tech talks and presentations, where we explored various topics from advanced DevOps practises to biometric engines. We had eeeverything. Of course testing-me had to live up to his urge and enrolled to all listed talks regarding his favorite IT-discipline: System Design Processes, Enterprise-Scale QA… and then there was that particular presentation about a simple yet effective test automation technique called fuzzing.

That one got me. I listened with an evil grin and decided to give it a shot. And that’s what we are going to do today!

Fuzzing – as explained in the talk – is a testing technique, that feeds the application a huge amount of random input data with different types and checks, which of them crashes the application. Simple enough. This can happen in a Black Box fashion by bombarding the public API or in a more White Box fashion by instrumenting the application code in order to get even more coverage and insides-related details.

In today’s tutorial, we will go through a Black Box Fuzzing setup written in Java. That’s because I’m more of a Black Box Testing person, and my main field of action is Java-based enterprise applications. More exactly, we will prepare a happy little Play Framework-based web application, that somehow got a commercial 3rd party conversion library called „Legacy“ imposed upon. Next we QAs want to have a first glance at Legacy’s state of quality to see, whether the purchase was at least somewhat worth it.

Prerequisites

This tutorial assumes that you have sbt and Maven installed. Since I wanted to try the Play Framework as a nice little side learning, we have to get along with sbt, but don’t worry: We need it only to compile the app. If you are curious, you can use it to run the app, too, but that’s 100% optional.

Maven on the other hand is used to operate the fuzz tests and thus will be our bread and butter tool.

Our Setup

Here’s the link to Happy Little Webapp’s source code repository. In ./app you can find the code of our Legacy-Module next to the web app’s controllers and (unused) views. Technically it’s not a blackbox, since I had to write the example code by myself, but let’s assume, we as the testers don’t know anything about it’s details, except for the public methods‘ signatures.

First, open the sbt shell: In your terminal of choice, enter the command sbt. Next, in the sbt-shell we just opened, we enter compile to compile the app’s code. Afterwards, if you are curious about what the app actually does, you can type run to start it. Now you can perform a request in your browser like:

http://localhost:9000/dollar2euro/58

It should display 53.36. Not as correct as we would expect it to be, because the used factor for the calculation is static and likely outdated. But for testing purposes, let’s assume, it is sufficient.

Next, we take care of our fuzz tests located in ./fuzztests. The pom.xml already knows about their location, so by using it, we can execute the tests right away. The fuzzing will be executed with a maven plugin called jqf-fuzz. Please see its github repository for the code and its well-elaborated documentation. With all that coming together, we are ready to fuzz.

Get the fuzzing started

First, we have to install the jqf-fuzz Maven plugin by doing a simple:

mvn clean test-compile

This downloads the jqf-fuzz plugin to our local maven repository and compiles the test sources. Now we have access to 2 new maven goals: jqf:fuzz executes the fuzz tests, and jqf:repro replays failed test cases to hunt down associated defects. Both goals expect several input parameter defined by JVM parameters (-D on the CLI) and/or by definition within the POM. This allows for a rich set of customization, that is both user- and CI-friendly. For demonstration purposes, I already configured the parameter time in the POM so that the test runs for 10 seconds, that still provides us with lots of input. Further, I predefined the fuzz test class to be executed. Therefore, the only parameter we must provide from the terminal is our test method -Dmethod=dollar2euro. We will do that in a minute, but first let’s have a look at the fuzz test class.

Let’s run the test

This is what we gonna unleash upon our web app:

@RunWith(JQF.class)
public class LegacyConverterFuzzer {

private static LegacyConverter legacyConverter;

@BeforeClass
public static void beforeClass(){
    legacyConverter = new LegacyConverter();
}

@Fuzz
public void dollar2euro(Object input){ // this is where the fun things happen
   try {
       System.out.println("Input: " + input.toString());
       System.out.println("Output: " + legacyConverter.dollar2euro(input));
   } catch (Throwable e) {
       System.out.println(e.getClass().getName() + ":" + 
                          e.getMessage());
   }
}

[... some more Fuzz-Tests, please see the repository linked above...]
}

Legacy’s executives promised, that any input works fine. Okay! Then we perform the test dynamic-typed by using an Object-typed input parameter.

Alright, that’s the code. Let’s fire it up. On your terminal, please do:

mvn jqf:fuzz -Dmethod=dollar2euro

Here’s an excerpt from the results as seen in my terminal. The output will vary for each new test run, because, as we said earlier, the input values in fuzz tests are random.

java.lang.NumberFormatException: Character 텈 is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: edu.berkeley.cs.jqf.fuzz.junit.quickcheck.InputStreamGenerator$1@4fc3c165
java.lang.NumberFormatException: Too many nonzero exponent digits.
Input: 뤇皽
java.lang.NumberFormatException: Character 뤇 is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: ky
java.lang.NumberFormatException: Character k is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: FixedClock[+898773291-08-05T17:23:55.165612278Z,UTC]
java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
Input: -8475850143961316955
Output: -7797782132444411598.60
Input: bn
java.lang.NumberFormatException: Character b is neither a decimal digit number, decimal point, nor "e" notation exponential mark.
Input: 16:19:25.242056065Z
java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
Input: -895394919-05-23T23:50:04.780324820
java.lang.NumberFormatException: Character array is missing "e" notation exponential mark.
Input: 11:14:21.890848137Z

Phew! We got a lot of NumberFormatExceptions. So much about „any input works“. Our PO should know about that.

6 months full of arguments later, the supplier delivered API version v1.0.1 of his LegacyConverter ensuring a static-typed API. He changed dollar2euro to the following:

    public String dollar2euro(BigDecimal input){
        BigDecimal dollars = input.setScale(2, BigDecimal.ROUND_HALF_EVEN);

Of course, we have to adapt our controller, too. For playground reasons, we keep that change as simple as possible.

    public Result dollar2euro(String dollars) { 
        return ok(importantConverter.dollar2euro(
                  BigDecimal.valueOf(Double.valueOf(dollars))));
    }

When we enter non-numeric inputs, the app will still fail, but at least it’s on us now.

Alright, the fixes are applied. Now in our test class, we see a sweet little type check error: We have to change the test method’s input parameter’s type accordingly to BigDecimal, too. This makes our fuzz test static-typed.

Afterwards, we recompile the tests and repeat the fuzz:

mvn clean test-compile
mvn jqf:fuzz -Dmethod=dollar2euro

giving us (excerpt):

Input: 152
Output: 139.84
Input: -1000
Output: -920.00
Input: -771298122
Output: -709594272.24
Input: 80372941329620235
Output: 73943106023250616.20
Input: 272536
Output: 250733.12
Input: -1000
Output: -920.00
Input: -2625164447481769740006272317
Output: -2415151291683228160805770531.64
Input: 9340202544
Output: 8592986340.48
Input: -34567
Output: -31801.64
Input: 17223398969630190416957297
Output: 15845527052059775183600713.24

Much better!

Conclusion – What did we achieve by fuzzing?

We have seen, how we can use fuzzing to create a vast storm of static or dynamic-typed test inputs and thus create hundreds of different test cases. From the output logs we can learn, what inputs can be handled by our application and – more interesting – what not. This provides us with an insightful first glance at the quality, a great starting point for further functional test cases, and, of course, with even more application bombing by using our favorite CI system.

From here, we can follow the functional testing track with even more elaborated automation or dive deeper into Java Fuzzing with JQF-Fuzz by checking out its paper. And if you still need motivation to automate your tests, check this one out.

Last but not least a huge shoutout to the great people at X41 D-SEC, who held the exciting talk, that inspired me and made me put fuzzing into my tool box.