Blog

  • Solving Newlines in MySQL TO_BASE64()

    How to remove unwanted line breaks in Base64 output

    When working with binary data in MySQL- such as certificates, keys, or other BLOB content-the TO_BASE64() function is the easiest way to export binary values in a portable text format.
    But many developers run into an annoying detail:

    MySQL automatically inserts newlines into the Base64 output.

    This can break APIs, JSON output, or scripts that expect Base64 to be a single continuous line. In this article, we’ll look at why this happens and how to reliably remove these unwanted newlines.

    Why MySQL inserts newlines in TO_BASE64()

    MySQL follows the MIME Base-64 specification RFC-2045. This standard requires that each encoded line must be no longer than 76 characters. Because of this, MySQL inserts a newline every 76 characters in the Base64-encoded string.

    This made sense historically, e.g. for email compatibility, but in modern applications it often creates more problems than it solves.

    The problem in practice

    Example:

    SELECT TO_BASE64(certificate)
    FROM distributor
    WHERE name = 'ACME';

    Instead of one long Base64 line, the output may look like:

    MIID5TCCA8+gAwIBAgIQB3h1g/...
    ui83iFkV103ef28hiUIgUGuiA/...
    g93kP6a1Dq7rzsQ==

    Those newlines are part of the actual string—not just formatting in the client.

    The solution: Remove newlines from Base64

    MySQL does not offer an option like “NO LINE BREAKS”.
    However, there’s a simple and reliable alternative solution:

    SELECT REPLACE(REPLACE(TO_BASE64(certificate), '\n', ''), '\r', '')
    FROM distributor
    WHERE name = 'ACME';

    This reliably strips out all MySQL-inserted line breaks and returns a clean, single-line Base64 value.

    What makes the solution great?

    It

    • works in all MySQL versions
    • is fast and lightweight (REPLACE() is very efficient)
    • ensures that all MIME-style breaks are removed
    • preserves the original BLOB unchanged
    • works equally with Unix and windows newlines

    These points make it a nice workaround that can be used in scripts, backends, and export pipelines.

    Conclusion

    If you export binary data like certificates from MySQL, you will eventually run into that nasty issue, but don’t be afraid,
    the fix is simple:

    Strip newlines using REPLACE() and you’re good to go.

    You get a clean, single-line Base64 string that plays nicely with JSON, APIs, and configuration files.

    So long, thanks for reading and have a great day.

    I got your mouth wet for more certificate fun and SSL? This post is about setting up HTTPS for Maven. And adventure by itself. Or if you’d rather like to see something different? Here’s some testautomation fun with Rust.

    Share it with:
  • Python Child Processes and subprocess run

    If Python likes to do except for KI and prey on mice it is automation. Test automation, process automation, tiny little helper supporting finance people all over the world… automation is on everyone’s lips – and on everyone’s project schedules. And due to the nature of automation in fields other than regular software development we might want to use what the operating system is already providing us. That’s where subprocess run comes in handy to create child processes in no time. We will see how to start an external child process from within python and we will use it to implement a simple automation use case. Let’s-a-go.

    Quick child process sample with subprocess run

    Okay let’s not go that fast. Before we head into the real thing, let’s train with a simple example. We will let the OS print the today’s date through a Python child pocess. Please see the following listing:

    Execute that code in your favorite console or IDE. You should get an output line similar to this:

    Sun Mar 9 19:01:22 CET 2025

    The first parameter is pretty straight forward. We provide the run method with the program command we would like to start using the list notation – date in this case. Another popular example could be ["ls", "-al"]. Try it out by replacing the date command with that one. In the meantime I tell you what the other method parameters do.

    capture_output=True

    This makes sure that we save the output in process.stdout for later use instead of just firing the command and forgetting the output. That would be kinda not-handy when we automate more complex things.

    encoding=“UTF-8″

    Without setting a nice string encoding we would get a byte-representation of process.stdout when printing that to the console. That’s not nice. Therefore we set the encoding to UTF-8 and get a nice and clean output line.

    Advanced automation example: Create an X509 certificate using child processes

    Alright, now that we a warmed up we will tackle a more realworld automation example. Let’s imagine you are tasked to secure a Maven repository with TLS and you need to generate an X509 certificate. If you are an avid reader of my blog this might sound familiar to you and you are right! Just recently I posted about SSL/TLS for Maven repos where we generate a Java Keystore and use that to connect to a repo via HTTPS. For this post here we take the certificate generation shell script and execute that from python. Let’s see how we achieve that:

    Execute that in a separate .py file in your favorite IDE and you should receive a neat set of X509 files including our target: endUser.crt.

    My favorite method options for subprocess run

    Now we got to know capture_output and encoding already. What else does subprocess run support? Here are 3 of my favorite method parameters:

    check

    If this is set to True and the child process terminates with a non-zero exit code – i.e. terminates with an error – a CalledProcessError is thrown. If False the process just terminates and your program will go on as if nothing happened. I am a big fan of failing fast so this is big sugar to me.

    shell

    If set to True your child process can use advanced shell features like pipes and filename wildcards. This is especially useful when it comes to various IT automation tasks.

    env

    This takes a str-to-str-dictionary containing your personal custom set of environment variables tailored to your child process by you. This set overrides the default set that is derived from its parent. This comes in handy if you want to start your process in a more isolated way.

    You are curious and want more now? Feel free to check the Python subprocess docs. There you’ll be served.

    Final thoughts

    Phew! Now we definitely earned our after-work soda. We have seen what the subprocess run – method is capable of and we solved real-world problems using child processes provided by that. We got ready to automate aaall the things that’s for sure. Wait, you still want more automation with Python? If you really do and you are willing to dip into software testing I have one to offer that covers test automation in Python. Alas it’s automation, too. Right? Or did you think that Python is not your thing now but you got inspired and want more shell? That’s perfectly fine, too. Here we create Linux users directly in your favorite shell.

    Share it with:
  • Docker Compose: Custom Commands to Keep your Container up

    Sometimes you just want to keep up your container without running the intended process, for example to diagnose an unexpected shutdown during starting up the container’s process. To achieve that, we make use of the option in docker and docker-compose to overwrite the container’s innate command that is usually executed during your container’s runtime.

    Preparing our experiment

    First of all we gotta pull the docker image for our experiments. For illustration purposes we use an image that is completely unrelated to any command we’re going to use later:

    docker pull maven:3.9.9-eclipse-temurin-8-alpine

    That’s all. Now let’s head into our experiments.

    docker container run COMMAND – parameter

    When executing docker run we can not only set an image name but also add a custom command that overrides the image’s default command. Let me illustrate that by showing you the date of today as of writing this post – using maven’s docker image.

    >docker run maven:3.9.9-eclipse-temurin-8-alpine date
    Sat Mar 8 14:21:43 UTC 2025

    where maven:3.9.9-eclipse-temurin-8-alpine is the docker image and date the custom command that replaces the default mvn execution.

    docker compose directive – command:

    Being a decorator on top of docker’s base CLI docker compose can do anything that plain docker run can do. Thus we can do custom command executions here as well. In fact, it’s much more fun than using the default CLI. But thats just me. Now without further ado:

    And there is our today’s date.

    What commands can we use in docker?

    In this section I present my favorite commands to keep my docker containers open. The most intuitive is probably:

    sleep infinity

    That one is pretty self-explanatory. We just sleep for an infinite amount of time or until we manually cancel the process. The constant infinity is defined in the GNU world and serves us as a representation of positive infinity.

    The next one is:

    top

    We just let your container dispay it’s realtime ressource usage thus keeping your container up and running. Time to docker exec into it and do our thing. Little caveat: Some images may strike top out to reduce the container’s size and to focus on it’s core business. If that’s the case, we should try my top-most favorite. This one should be available quite often and looks so cool:

    tail -f /dev/null

    That one waits on a neverending file and tries to spit out whatever it finds at the end of it – in this case dark and pure nothingness.

    Final thoughts

    This post was a personal matter of heart to me having had the use case several times. But since im getting old *sigh* I tended to forget what commands I can use. Now that’s finally a thing of the past. Give it a bookmark and it will keep reminding you as well. Do you have a favorite command to keep things up? Feel free to drop me a line in the comments down below. And if you still can’t get enough of Docker or Maven here are some recommendations just for you: My most recent star is this post about configuring TLS for HTTPS-secured Maven repos. And finally for the more docker-inclined people out there this post about docker ps – formatting might make your daily life a little simpler.

    Have a sunny day!

    Share it with:
  • Problem reading geckodriver versions: error sending request for url

    Today, while hunting down a nasty issue in a longer Selenium/Cukes scenario outline, I was flooded with a particular annoying warning:

    WARNING:
    org.openqa.selenium.manager.SeleniumManagerlambda$runCommand$1
    Problem reading geckodriver versions: error sending request for
    url (https://raw.githubusercontent.com/SeleniumHQ/selenium/trunk/common/geckodriver/geckodriversupport.json).
    Using latest geckodriver version

    org.openga.selenium.manager.SeleniumManagerlambda$runCommand$1
    WARNING: Exception managing firefox: error sending request for
    url (https://github.com/mozilla/geckodriver/releases/latest)

    You too? Then you are at the right place.

    The reason

    Selenium is trying to check for an update for your geckodriver executable. What is planned as a nice convenience feature can be quite an annoyance in a restricted environment. When Selenium is executed within such an environment, it cannot contact the remote geckodriver version dictionary and nags us with „Problem reading geckodriver versions: error sending request for url …“.

    Thankfully the Selenium team seems to be aware of that and provided us with a simple fix. Sadly that fix is quite non-explicit but that’s not going to stop us anytime soon.

    Fixing the „Problem reading geckodriver versions: error sending request for url“ warning

    The update check happens if – and only if – Selenium derives the geckodriver path from Seleniums automatic lookup feature (i.e. when it checks the system PATH variable). Therefore all we have to do is setting the geckodriver path explicitly. In Java we do that like this:

    By setting that Java system property we tell Selenium explicitly what geckodriver to use and we prevent it from doing any additional shenanigans on the driver binary thus silencing the warning. Using an appropriate if-clause you can tailor it to any kind of environment. You have full internet access? Cool! Then your system geckodriver might be perfectly sufficient. It’s all in your hands.

    Final thoughts

    Alright! This should relieve you from that „Problem reading geckodriver versions: error sending request for url“ warning. When working with complex Selenium Webdriver tests, it is overall nice that the framework logs so verbose and provides us with all these features. But sometimes it can be a little full of itself. The good thing is though that there usually is a way to customise Selenium’s behavior to our personal needs, whether it is working in a restricted network area or with full access to the internet – Selenium knows it all.

    Though sometimes it has its difficulties to articulate itself – just as we humans do. Anyways, thank you for reading. If you are still curious about what’s possible with Java in QA, here’s an interesting post about fuzzing in Java. If you don’t particularly care about Java but you would like to see more test automation action in general, I’d recommend trying Cucumber in Rust.

    Have a great day!

    Share it with:
  • Gitlab CI Error – Cannot create local repository at: /.m2/repository

    When you are using downstream pipelines in Gitlab CI for complex Maven Projects as I do, you may have stumbled accross this error at least once: Cannot create local repository at: /.m2/repository. The Gitlab runner tries to create a local Maven repository at /.m2/repository – so in the topmost directory – and fails horribly due to an obvious lack of permissions. Let‘ see what happens here.

    The Gitlab CI Setup

    In my case I had a shell-based Gitlab runner that executes a test suite based on a few artifacts published by a Docker runner. Due to versioning and system test level realism reasons I was not able to use the artifacts directly. Unfortunate, but it shouldn’t pose that kind of a problem, does it?

    Now when it comes to cloning the test suite on my shell runner, this cryptic error has been dropped. And naturally I was like „WTF is this runner trying to do!?“

    The root cause in Gitlab CI downstream jobs

    What I was not aware of is that per default the „child job“ inherits all the custom and Gitlab-provided runner variables from the triggering parent job that is still executed on a Docker runner. Now since the trigger happens from a Docker runner job, the variables my child job receives are poison for a baremetal environment that is my shell runner. Not least because things actually do happen at / in a Docker-based Gitlab execution environment, which is perfectly fine, but not on a shell runner.

    The solution: What I had to do to make the CI jobs work

    To fix the problem I had to set this on my trigger job definition:

    This did not just fix the error for me, but it also made perfect sense. Due to the different execution environments – shell vs. docker and project A vs. project B – I have a different set of requirements for my Gitlab CI test job. Therefore we have another case of an error leading to better software design. In addition I learned another piece of Gitlab’s sometimes quite obscure default settings.

    Conclusion

    I hope this helps you during your day to day journey through the jungle that are Gitlab CI downstream jobs. As I’m an avid QA engineer, so if you want to read more about writing actual automated tests. Also I take care of deeper coding basics like working with threads in java. If you’d rather want to read up about Gitlab CI’s inherit keyword, here’s the link to the relevant section of the official Gitlab CI documentation. Feel free to have a look!

    Have a great weekend everybody!

    Share it with:
  • Java Threads – an Introduction for all Skill Levels

    Today we are talking about threads in Java. Threads are an essential means to implement concurrent algorithms and are therefore a key part of efficient everyday data processing. Imagine having a webserver without threads serving only one customer at a time: Every customer would need to pull a number for every single web request they make and our online shop would have a bad time.

    So let’s head into it!

    In modern days – Java threads with Lambda expressions

    Since Java 8 we have been blessed with Lambda expression. Basically Lambda expressions serve the purpose of unnamed functions and are written as shown here:

    (param1, param2, [...], paramN) -> {
        statement1; 
        statement2;
        [...]
        statementN;
    }

    If you just need one statement, it looks even better:

    (param1, [...], paramN) -> statement

    By harnessing their power we are able to write beautiful first level function style concurrent algorithms.

    Let’s assume we want to sort a list with an even number of integers. We know that all ints in the second half of the list are greater than the greates number in the first half. Then we can do this:

    Try it out!

    Notice that we feed the threads in our Java code with Lambdas to tell them what to do. Neat, isn’t it?

    Legacy Java Threads with Runnable instances

    And how has it been done in the past? When I started programming with Java around 5 and 6 we had to declare a full-blown Runnable-instance that we subsequently fed to the thread. Runnable is an interface type which means that during declaration we also had to implement it’s signature method run(). This method finally contains the functionality that the thread is supposed to run. Let me show what I mean:

    Try it out!

    It was a bloated mess, but since it’s likely in use in modern day code, we got to list it here.

    Released in Java 21: Virtual Threads

    Virtual threads are part of Project Loom that aims at bringing lightweight and high-throughput concurrency models to the JVM. These virtual threads don’t run on kernel but on JVM level and thus are fully managed by the JVM. That means you as the coder do not need to keep track of them at all. The JVM will do that for you while you can rely on the coding APIs you already know and love. That is because virtual threads are created almost the same way as the native threads we have seen earlier. Let me show what I mean:

    As shown above you the virtual threads API hands you a Thread instance that you can use in ways you are already familiar with. Here we wait for our virtual threads to terminate.

    Another important advantage over conventional threads is their ability to scale. Since the JVM manages both ressources and lifetime of virtual threads it is done in a thoroughly optimized way. In addition, because these threads run directly in the JVM instead of the OS kernel, we can omit a lot of OS ressource management overhead. Both factors result in a significant increase of throughput.

    Conclusion

    So long. I hope I could help you effectively processing data by leveraging the power of Java threads. Please note that the examples shown above are optimized for the topic’s illustration only. Your way to sort a list of integers is certainly much better. If you’d like to read more about me doing Java, I have a post for you, where I talk about web test automation with Java. If you are already familiar with that and want to try something new, I recommend trying fuzz testing in Java. You will gain a fresh view on test automation and honestly it’s a blast to bomb an application with just random stuff.

    Last but not least, if you want to dive in really deep into the world of Java threads, check out this article about Thread Pooling. It is quite advanced, but intuitive and powerful once you get the hang of it.

    You have more questions regarding Java, me or anything else? As usual, feel free to drop me a line down below. I read every single one of your comments. Have a great day!

    Home
    Share it with:
  • Character array is missing „e“ notation exponential mark

    In my previous article about Fuzzing in Java, I encountered an interesting NumberFormatException: „Character array is missing „e“ notation exponential mark“. That one sparked a lot of curiosity. Today, let’s take a closer look at what exactly this error message means.

    What Does This NumberFormatException Mean?

    Let’s start with a simplified version of a method, dollar2euro, which converts a USD input to EUR:

    Now, suppose we call this method using the string "뤇皽" as input.
    This would produce the following error:

    Input: 뤇皽
    java.lang.NumberFormatException: Character 뤇 is neither a decimal digit number, decimal point, nor "e" notation exponential mark.

    The message tells us that the character 뤇 is neither a decimal digit, a decimal point, nor an „e“ used in exponential notation. In mathematics and programming, exponential notation expresses numbers like 1e3 to represent 1000 compactly. Java expects input strings for BigDecimal to contain only valid number representations — digits, decimal points, or the character e for exponents.

    Since „뤇“ is none of these, Java immediately throws a NumberFormatException.

    How Can You Avoid This Error?

    To prevent this error, we should ensure that the value passed to the BigDecimal constructor is a valid number.
    Ideally, you would work directly with numeric types like double or int whenever possible.

    In our method, this would only require a small change to the parameter type and the parsing logic.

    If you still prefer to work with String inputs, make sure the string represents a valid number format.
    An example of a valid input would be:

    "1337.012342

    By sticking to properly formatted input, you can avoid NumberFormatException issues like the one described here.

    One side note here: Proper validation of user input is a critical security measure recommended by OWASP to prevent injection vulnerabilities and ensure application robustness. Using and handling this exception may help you here depending on your use case.

    Conclusion

    I hope this article helped clarify what the error „Character array is missing ‚e‘ notation exponential mark“ means in Java. While today’s example was constructed for clarity, such issues often arise unexpectedly when dealing with user input or external data sources.

    If you have encountered a similar error in a real-world scenario, feel free to share it in the comments. I would be happy to take a look!

    If you’re interested in diving deeper into testing and automation, you might enjoy my tutorials on Selenium with Java and Selenium with Python.
    Or, if you want something a little more adventurous, check out my Cucumber in Rust tutorial.

    Happy testing and have a great day!

    Home
    Share it with:
  • Stable Diffusion: „LayerNormKernelImpl“ not implemented for ‚Half‘

    Today I installed the Stable Diffusion Web UI by AUTOMATIC1111 on my ol‘ reliable Macbook Pro 2015 following this stable diffusion installation guide. And not only is it fun to do AI stuffs on an Intel Mac with no GPU whatsoever, of course I also had to run into a „first try error“ ™. This time it was that one:

    RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

    Okay, cool.

    So how do we fix that?

    Well it turns out that this „half“ thingy, that relates to floating point sizes, can be turned off. To do that, let’s revisit your stable-diffusion-webui installation directory and open the shell script webui-user.sh using your most beloved code editor. Here you will find the following 2 lines:

    # Commandline arguments for webui.py, for example: export COMMANDLINE_ARGS="--medvram --opt-split-attention"
    #export COMMANDLINE_ARGS=""

    What we got to do now is: First we remove the leading ‚#‚ character of the second line to uncomment it. Next, we fill the variable with "--skip-torch-cuda-test --no-half" to make it look like this:

    export COMMANDLINE_ARGS="--skip-torch-cuda-test --no-half"

    Restart the stable diffusion web ui using your webui.sh and it should behave as expected. All of that has been tested with stable-diffusion-webui commit cf2772fab0af5573da775e7437e6acdca424f26e, which was the most recent stable version at the time of writing.

    If you are on a Windows machine, the fix should be basically the same: Open the webui-user.bat and change COMMANDLINE_ARGS to match the following:

    set COMMANDLINE_ARGS="--skip-torch-cuda-test --no-half"

    Restart your server and everything should be alright. (Disclaimer: Untested due to the lack of Windows machines.)

    Conclusion

    So that’s all, hope this helps. If not, feel free to let me know in the comments below. If you find a better fix, or my post is outdated, or you found any other issues, please let me know as well. Let’s keep it as complete as possible for future users. And if you are still curious about working with Python apps or Python in general, here is another quick and handy post about handling environment variables. And here we talk about my second most favorite topic: Test Automation.

    Best regards, and have a nice week!

    Home

    Share it with:
  • Introduction to Web Test Automation – Java Edition

    The internet is an incredible market place hosting millions and millions of software products to make our lives indefinitely easier. To keep up with each other, software grew more and more elaborate to provide us customers with lots of different features and services to finally make us purchase a thing or two. Now due to the resulting increase of complexity we must take special care not to break existing features while developing the next big version brimming full of features. Testing that the next code increment does not break existing stuff is what we testers call „regression testing“, and there is one especially economical way of doing that: test automation.

    That’s what we’re going to do today. We gonna test and we gonna automate and we do all that in Java this time. Because everyone knows about Java, right? So let’s write once and run everywhere. Given we have a browser there.

    Shopping list

    To start our test automation journey – again -, we will work with the following tools. The versions are the ones I used at the time of writing. I might update them in the future.

    After downloading and installing these tools from their respective web sites, we’re ready to start our system under test.

    Deploying the system under test

    To achieve this, we simply run the following command:

    docker run --name spree --rm -d -p 3000:3000 spreecommerce/spree:3.6.4

    If it went well, a quick glance at http://localhost:3000/ should show a catalog page full of cute Rails merch. Now we will install and prepare Selenium.

    Geckodriver installation and management

    Contrary to the Python version, we use WebDriverManager to free us from the installation hassle of Geckodriver, hence we are done with this step. The tip was sent to me by my good friend Sho. Thank you, mate. I owe you one! 🙂

    Initializing our test automation project

    Given a successful Gradle installation as linked in the shopping list we can now create our test project. Please open a terminal in a root directory of your choice and do:

    mkdir jata_tutorial && cd jata_tutorial
    gradle init

    When prompted answer with 1 for a basic project and then 2 for Kotlin as our Gradle DSL.

    Of course we will use the new APIs as well, since we love to live on the edge here. Please choose „yes“ here.

    Finally, when asked, give your project the default name that is the project root directory’s name. Now your boilerplate project is set up and ready to go.

    Now it’s time to define our dependencies. In build.gradle.kts please introduce the following lines:

    plugins {
        java
    }
    
    group = "your.groupid"
    version = "1.0-SNAPSHOT"
    
    repositories {
        mavenCentral()
    }
    
    dependencies {
        implementation("org.slf4j:slf4j-api")
        implementation("ch.qos.logback:logback-classic:1.4.5")
        implementation("org.apache.commons:commons-lang3:3.12.0")
    
        testImplementation("org.seleniumhq.selenium:selenium-java:4.7.2")
        testImplementation("io.github.bonigarcia:webdrivermanager:5.3.1")
        testImplementation(platform("org.junit:junit-bom:5.9.1"))
        testImplementation("org.junit.jupiter:junit-jupiter:5.9.1")
    }
    
    tasks.getByName<Test>("test") {
        useJUnitPlatform()
    }

    This sets up our project with some basic metadata like our groupId and a (somewhat) meaningful version. We defined to use Java as our JVM language and JUnit as our test runner. Additionally we will have access to Maven’s central repository, where Gradle can download our project dependencies that we defined in the section of that exact same name.

    With these dependencies, we are able to:

    • automagically download and install webdriver executables for Selenium (in our case: Geckodriver)
    • perform Selenium actions including opening and closing browser windows, enter URLs and clicking stuffs
    • generate random test input strings
    • log things to the console in a nice timestamped format (without being scared of Log4Shell)
    • and do all that in a modern JUnit5 – runner

    Everything that we need to do very soon.

    The boilerplate

    Next up, we create the skeleton of our first test class. In src/test/java, please create a file named CrossTests.java containing the following code. Please note that .java source files should be named after the contained class. Here: CrossTests.

    import org.junit.jupiter.api.AfterEach;
    import org.junit.jupiter.api.BeforeAll;
    import org.junit.jupiter.api.BeforeEach;
    import org.junit.jupiter.api.Test;
    import org.openqa.selenium.WebDriver;
    
    class CrossTests {
    
        WebDriver driver;
    
        @BeforeAll
        static void setupClass() {
            
        }
    
        @BeforeEach
        void setup() {
    
        }
    
        @AfterEach
        void teardown() {
    
        }
    
        @Test
        void isOpen() {
    
        }
    }
    

    Great. That looks almost like a real test already. Finally we are going to open a browser window.

    Starting and closing the browser

    In setupClass we call the WebDriverManager and let it work its Geckodriver-initializing magic.

    @BeforeAll
    static void setupClass() {
        WebDriverManager.firefoxdriver().setup();
    }
    

    Then we ensure that a browser is opened before each of the individual tests marked with @Test:

    @BeforeEach
    void setup() {
        this.driver = new FirefoxDriver();
        this.driver.manage().timeouts().implicitlyWait(Duration.of(5, ChronoUnit.SECONDS));
    }

    This opens a Firefox window whenever we execute one of the class‘ test methods. This is a very pure and uncustomised instance; we could configure it further if we wanted. But for our purpose at the moment, it is perfectly fine.

    Now before we perform any actions on the page, we must make sure that the browser window is closed appropriately after each test run. In teardown, we do:

        @AfterEach
        void teardown() {
            this.driver.quit();
        }

    Caveat: confusion potential

    Selenium’s WebDriver class has a close()-method, but please always use quit() instead. It performs a clean browser termination with regards to Selenium’s execution flow, whereas close() will just terminate the browser causing you lots of warnings in the log.

    Alright! We have taken care of the browser handling. Time to do stuffs with it.

    Opening the landing page

    The most basic action ist to open a web page. Thankfully this task is handled by Selenium with a simple one-liner. Given that our sample web shop is up and running, we do:

    this.driver.get("http://localhost:3000/");

    We introduce the line to the test case method isOpen, which we introduced earlier in our boiler plate. But before we head into executing the test case, we have to be aware of the fact: We as the user can see the page being open, but the computer cannot. We have to programmatically verify that our expectation „landing page is open“ is met.

    Verifying the outcome

    To do that we verify these 3 conditions:

    • the window title contains the landing page’s HTML title
    • the most prominent logo exists on the web page and
    • it’s actually visible (i.e. it does not have its display property set to none).

    We add this code directly below the line that opens the website:

    assertTrue(this.driver.getTitle().contains("Spree Demo"),
               "'Spree Demo' is not in the browser's title.");
    
    WebElement logoElement = this.driver.findElement(By.id("logo"));
    assertTrue(logoElement.isDisplayed());

    The first line peeks into the page title, which is another staple of Selenium, and then checks if it contains „Spree Demo“. Using JUnit’s assertTrue, we verify that this is the case, and if not, we print a custom error message. Condition 1 done.

    The second line depicts the method you will probably use the most whilst moving forward in your test automation career: Webdriver#findElement goes through the page’s DOM and grabs the element that meets the given criteria. Here we search for an element with the HTML id „logo“. If it fails to find the element, it drops a NoSuchElementError. Therefore, we just implicitly verified condition 2.

    The return value is a WebElement object that we can do clicks, inputs and various other actions on. We will use the object’s method isDisplayed() to check if the logo is visible therefore verifying our 3rd condition.

    Straight-forward so far. At this point your code should look like that:

    import io.github.bonigarcia.wdm.WebDriverManager;
    import org.junit.jupiter.api.*;
    import org.openqa.selenium.By;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.WebElement;
    import org.openqa.selenium.firefox.FirefoxDriver;
    
    import java.time.Duration;
    import java.time.temporal.ChronoUnit;
    
    import static org.junit.jupiter.api.Assertions.*;
    
    class CrossTests {
    
        WebDriver driver;
    
        @BeforeAll
        static void setupClass() {
            WebDriverManager.firefoxdriver().setup();
        }
    
        @BeforeEach
        void setup() {
            this.driver = new FirefoxDriver();
            this.driver.manage().timeouts().implicitlyWait(Duration.of(5, ChronoUnit.SECONDS));
        }
    
        @AfterEach
        void teardown() {
            this.driver.quit();
        }
    
        @Test
        void isOpen() {
            this.driver.get("http://localhost:3000/");
            assertTrue(this.driver.getTitle().contains("Spree Demo"),
                    "'Spree Demo' is not in the browser's title.");
    
            WebElement logoElement = this.driver.findElement(By.id("logo"));
            assertTrue(logoElement.isDisplayed());
        }
    
    }
    

    Let’s execute it. In a terminal window, please switch to the project root and do:

    ./gradlew test

    If everything went well, your console should display a big OK:

    BUILD SUCCESSFUL in 7s
    4 actionable tasks: 2 executed, 2 up-to-date

    Congratulations, you just executed your first Java-Selenium-test:
    Our shop’s landing page can be opened and it is displayed correctly.

    What is this driver thingy by the way?

    Simply put, the driver – in our case Geckodriver – is the glue between our code and Firefox. The same would be valid for Chrome with Chromedriver respectively. This means it is the interface to our browser handling all the requests that we perform :

    • findElement
    • click
    • „are you displayed?“
    • perform key strokes
    • „grab that text of an element“
    • etc. pp.

    Under the hood, drivers are http server applications that receive requests sent by Selenium everytime we perform an action similar to these mentioned above. The driver then interacts with the browser in the way we tasked it to and sends a JSON-based response that we can base our future work on. Everything is encapsulated by Selenium so that we can work with simple objects during automating our tests without having to deal with HTTP or JSON.

    Performing and validating browser actions

    Now that we know the basics of how to write automated tests with Java and Selenium, let’s develop a more complex test.

    The best candidates for automation are cross tests. These are tests that cover representative cross sections of the product. One example could be: Put items into the cart, view the cart page and perform the checkout. It is a great practise to start test automation for new projects by coding these cross tests first.

    For the sake of brevity here, we will write up the test until reaching the checkout page. The rest is homework. Don’t worry, I got you covered with the full source code in my bitbucket repo.

    Ok, without further ado, here’s the code. Please add the following method to the test class.

    @Test
    void findSpreeBagAndCheckout()throws InterruptedException{
        this.driver.get("http://localhost:3000/");
    
        // Landing and catalog page
        this.driver.findElement(By.cssSelector("[href='/t/spree']")).click();
        this.driver.findElement(By.cssSelector("a[href*='spree-bag']")).click();
    
        // Product page
    
        this.driver.findElement(By.cssSelector("input#quantity")).clear();
        this.driver.findElement(By.cssSelector("input#quantity")).sendKeys("3");
        this.driver.findElement(By.id("add-to-cart-button")).click();
    
        // Cart page
        assertTrue(this.driver.getTitle().contains("Shopping Cart"),"'Shopping Cart' is not in the browser's title.");
        this.driver.findElement(By.cssSelector("img[src$='/spree_bag.jpeg']")).isDisplayed();
        String qtyValue=this.driver.findElement(By.cssSelector(".line_item_quantity")).getAttribute("value");
        assertEquals(qtyValue,"3","The item quantity is not the same we typed in!");
    
        String cartTotal=this.driver.findElement(By.cssSelector(".cart-total > .lead")).getText();
        assertEquals(cartTotal,"$68.97","The total cart value does not match our expectations. That's a blocker!");
    }

    What’s new here?

    If we look back at our first landing page test, you will notice that we use a lot of new things. We will cover them one by one in the next section.

    CSS selectors

    One thing that catches the eye is that we looked for elements by using CSS expressions. You may have heard about them from your frontend peers, or maybe you have even worked with them yourself already. If not, don’t worry. We will cover them in a followup post. For now, I will give you a few translations:

    [href='/t/spree']

    „Give me a page element, whose attribute href is equal to ‚/t/spree’“. Quick reminder: href is used in link elements for defining the target URL that is opened, when the user clicks the link.

    a[href*='spree-bag']

    „Give me any link („anchor“) element, whose attribute href contains ’spree-bag‘.“

    img[src$='/spree_bag.jpeg']

    „Give me an image element, whose attribute src ends with ‚/spree_bag.jpg‘.“

    input#quantity

    „Give me an input element, whose HTML id is equal to ‚quantity‘.“ Yes, we could have used By.ID with „quantity“ here, but in this case I wanted to be explicit about the element type input.

    .line_item_quantity

    „Give me an element that has line_item_quantity in its class attribute.“ The leading dot indicates that we are looking for elements that has the specified class in its class list. It is roughly equivalent to [class*='line_item_quantity'].

    .cart-total > .lead

    „Give me any element that has the lead class and that’s preceded by any element that has the cart-total class.“

    By using CSS selectors you can be very specific about what element you want to use in your test case. This makes CSS selectors incredibly powerful and versatile while maintaining a solid degree of simplicity, especially compared to XPath (ugh).

    New properties and methods used in our test case

    Aside from CSS selectors, we used several new methods and properties in our automated test.

    click() issues a click on the target element. Straight-forward.

    .getText() returns the element’s written text. In <p>Some text</p> for example, we would get access to „Some text“.

    getAttribute("someHtmlAttribute") gives us the content of an element’s HTML attribute. In our test we fetch an element’s value attribute that is often used to set the content of an input element. Another example might be the link element we talked about earlier. Given we want to have its href attribute for some further verifications; on our page:

    <a id="mylink" href="https://www.example.org">an example link</a>

    Then in our test we could do:

    targetHref = this.driver.findElement(By.id("mylink"))
                  .getAttribute("href");
    this.doSomeStuffsOn(targetHref);

    .sendKeys("My desired input") imitates key strokes on an element. We use it to type any string we want into our target text input element. In our test for example that would be the „quantity“ input box.

    And finally, with clear() we have a little helper in our toolbox that removes any content from an input element. In our test we use that to remove the preset value. Otherwise we would accidentally order 31 bags: „3“ from us and „1“ from the value preset by the page.

    Continuing your test automation journey from here

    Alright, let’s sum up what we accomplished:

    We have seen how to open and close a browser window, do various actions on the page and verify their outcomes. Finally, we used that to write a fairly large cross test, all of that on a full fledged ecommerce web app. Great job! Now where should we go next? For the next post, as promised, I’d like to go back one step and give you a deeper introduction to CSS expressions, because they will accompany you for a long time during your test automation journey. Afterwards we have a big problem to solve that you probably already noticed:

    Copious amounts of repetition.

    To tackle that we will leverage cucumber-jvm, a Behavior Driven Development framework that makes it possible to write test cases in human-readable text form on top of a DRY Java code base. Afterwards we will apply the Page Object Pattern, a test automation – specific design pattern that reduces repetition even more.

    Okay! To keep you busy while I am busy, how about finishing the checkout? As a reminder, you can find the full source code in my bitbucket repo at any time. Alternatively you can spoil yourself a little about Cucumber in Rust or try Python instead of Java.

    Stay curious and see you in the next post!

    Home
    Share it with:
  • Introduction to Web Test Automation in Python

    The internet is an incredible market place hosting millions and millions of software products to make our lives indefinitely easier. To keep up with each other, software grew more and more elaborate in order to provide us customers with lots of different features and services to make us purchase a thing or two. Now due to the resulting increase of complexity we must take special care not to break existing features while developing the next big version. Testing that the next code increment does not break existing stuff is what we testers call „regression testing“, and there is one especially economical way of doing that: test automation.

    That’s what we’re going to do today. We gonna test and we gonna automate and we do all that in Python, because with its well-designed concepts Python is suitable for beginners and experts alike. So let’s start!

    Shopping list

    To start our test automation journey, we will work with the following tools. The versions are the ones I used at the time of writing. I might update them in the future.

    • Docker, preferably in its latest version
    • Python 3 version 3.10.8
    • Firefox Browser 106.0.1
    • geckodriver for Firefox 0.32.0
    • a Spree Commerce docker image version 3.6.4 as our system under test
      (will be pulled automagically in the next step)
    • Selenium for Python 4.5.0
    • unittest as our first test runner (comes with Python 3)

    After downloading and installing Docker and Python from their respective web sites, we’re ready to start our system under test.

    Deploying the system under test

    To achieve this, we simply run the following command:

    docker run --name spree --rm -d -p 3000:3000 spreecommerce/spree:3.6.4

    If it went well, a quick glance at http://localhost:3000/ should show a shopping catalog page full of cute Rails merch. Now we will prepare and install Selenium.

    Installing Geckodriver and Selenium

    First we need to download geckodriver that will be responsible for sending our actions to the browser. Please see our shopping list for a download page. Download and extract the appropriate archive and put the extracted executable on your PATH. Please do that in a way that fits your OS and your preferences. I’m a mac user, therefore I move it to /usr/local/bin:

    $ mv ~/Downloads/geckodriver /usr/local/bin/ 

    If everything went well,

    $ geckodriver -h 

    should print the version and an instruction page. If not, feel free to drop me a Q in the comments below. Otherwise we move on to installing Selenium.

    We will leverage pip3 to quickly go through that step:

    pip3 install selenium==4.5.0

    Now we’re all set. Let’s go headlong into the real trouble.

    Test runner boilerplate

    First of all, we create the skeleton of our test class. We write a simple Python unittest class that we will fill in with Selenium web test automation goodness as we go further. Our goal is to see the basic structure of a unittest-based automated test case that we will leverage to execute our browser magic. In your favorite project directory, please create a file named cross_tests.py and fill it with the following code. We will talk about the file name a bit later. Don’t worry, it was not a technical decision.

    import unittest
    
    # Start with a base class and derive it from 
    # unittest.TestCase
    class TestSpreeShop(unittest.TestCase):
    
        # setUp is executed at the start...
        def setUp(self):
            pass
    
        # ... and tearDown at the end of each test case.
        def tearDown(self):
            pass
    
        # This will be our first test:
        # We will open the shop's homepage
        # and verify that it worked.
        # For now, we will just let it pass.
        def test_isOpen(self):
            pass
    
    
    # The main entry point of our unittest-based execution 
    # script. Think of it as an actual main method similar 
    # to Java's or C's.
    if __name__ == '__main__':
        unittest.main()
    

    Great, that almost looks like a real test already. Now finally we are going to open a browser window.

    Starting and closing the browser

    We write a small support function:

    def open_browser(self):
        return webdriver.Firefox()
    

    And we apply it in setUp:

    def setUp(self):
        self.driver = self.open_browser()

    This opens a Firefox window it its most pure form whenever we execute one of the class‘ test methods. We could configure it further in open_browser, but for our purpose at the moment, this is perfectly fine.

    Now before we perform any actions on the page, we must make sure that the browser window is closed appropriately after each test run. In tearDown, we do:

    def tearDown(self):
        self.driver.quit()

    Caveat: confusion potential

    Selenium’s webdriver has a close()-method, but please always use quit() instead. It takes care of clean browser termination with regards to Selenium’s execution, whereas close() will just terminate the browser causing warnings.

    Alright! We have taken care of the browser handling. Time to do stuffs with it.

    Opening the landing page

    The most basic action ist to open a web page. Thankfully this task is handled by Selenium with a simple one-liner. Given our sample web shop is up and running, we do:

    self.driver.get("http://localhost:3000/")

    We introduce the line to the test case method test_isOpen, which we introduced in our boiler plate. But before we head into executing the test case, we have to be aware of the fact that we as the user can see the page being open, but the computer can’t. We have to programmatically verify that our expectation „landing page is open“ is met.

    Verifying the outcome

    To do that we verify these 3 conditions:

    • the window title contains the landing page’s HTML title
    • the most prominent logo exists on the web page and
    • it’s actually visible (i.e. it does not have its display property set to none).

    We add this code directly below the get – line from before:

    self.assertTrue("Spree Demo" in self.driver.title, 
                    "'Spree Demo' is not in the browser's title.")
    
    logo_element = self.driver.find_element(By.ID, "logo")
    self.assertTrue(logo_element.is_displayed())

    The first line peeks into the page title, which is another staple of Selenium, and then checks, if it contains „Spree Demo“ with Python’s in-operator. Using unittest's assertTrue, we verify that this is the case, and if not, we print a custom error message. Condition 1 done.

    The second line depicts the method you will probably use the most whilst moving forward in your test automation career: find_element goes through the page’s DOM and grabs the element that meets the given criteria. Here we search for an element with the HTML id „logo“. If it fails to find the element, it drops a NoSuchElementError. Therefore, we just implicitly verified condition 2.

    The return value is a WebElement object that we can do clicks, inputs and various other actions on. We will use the object’s method is_displayed() to check, if the logo is visible therefore verifying our 3rd condition.

    Straight-forward so far. At this point your code should look like that:

    import unittest
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    
    # Start with a base class and derive it from unittest.TestCase
    class TestSpreeShop(unittest.TestCase):
    
        # setUp is executed at the start...
        def setUp(self):
            self.driver = self.open_browser()
    
        # ... and tearDown at the end of each test case.
        def tearDown(self):
            self.driver.quit()
    
        def test_isOpen(self):
            self.driver.get("http://localhost:3000/")
            self.assertTrue("Spree Demo" in self.driver.title, 
                            "'Spree Demo' is not in the browser's title.")
    
            logo_element = self.driver.find_element(By.ID, "logo")
            self.assertTrue(logo_element.is_displayed())
    
    
        # Support method. We will use it to open a browser instance in setUp.
        def open_browser(self):
            return webdriver.Firefox()
    
    # The main entry point of our unittest-based test execution script.
    # Think of it as an actual main method similar to Java's or C's.
    if __name__ == '__main__':
        unittest.main()
    

    Let’s execute it by doing:

    python3 cross_tests.py # Linux or MacOS
    python cross_tests.py # Windows

    If all went well, your console should display a big OK:

    ➜  pyta_tutorial git:(main) python3 cross_tests.py
    .
    ----------------------------------------------------------------------
    Ran 1 test in 4.719s
    
    OK

    Congratulations, you just executed your first Python-Selenium-test:
    Our shop’s landing page can be opened and it is displayed correctly.

    What is this driver thingy by the way?

    Simply put, the driver is the glue between our code and Gecko (or Chromedriver respectively), thus it’s the interface to our browser window. It handles all the requests that we perform like find_element, click, „are you displayed?“, perform key strokes, „grab that text of an element“ etc. pp. Under the hood, Gecko is an http server, which receives requests sent by the driver everytime we perform one of the above actions. The driver then interacts with the browser in the way we tasked it to and sends a JSON-based response object that we can base our future work on. Everything is encapsulated by Selenium so that we can work with simple objects during test automation development without having to deal with HTTP or JSON.

    Performing and validating browser actions

    Now that we know the basics of how to write automated tests with Python and Selenium, let’s develop a more complex example.

    The best candidates for automated tests are cross tests. These are tests that cover representative cross sections of the product. One example could be: Put items into the cart, view the cart page and perform the checkout. This is the reason we named our test file cross_tests.py. It is good practise to start test automation for a new project by coding these cross tests first.

    For the sake of brevity here, we will write up the test until the checkout page. The rest is homework. Don’t worry, I got you covered with the full source code in my bitbucket.

    Ok, without further ado, here’s the code. Please add the following method to the test class:

    def test_findSpreeBagAndCheckout(self):
        self.driver.get("http://localhost:3000/")
    
        # Landing and catalog page
        self.driver.find_element(By.CSS_SELECTOR, "[href='/t/spree']").click()
        self.driver.find_element(By.CSS_SELECTOR, "a[href*='spree-bag']").click()
    
        # Product page
        self.driver.find_element(By.CSS_SELECTOR, "input#quantity").clear()
        self.driver.find_element(By.CSS_SELECTOR, "input#quantity").send_keys("3")
        self.driver.find_element(By.ID, "add-to-cart-button").click()
            
        # Cart page
        self.assertTrue("Shopping Cart" in self.driver.title, 
                      "'Shopping Cart' is not in the browser's title.")
        self.driver.find_element(By.CSS_SELECTOR, "img[src$='/spree_bag.jpeg']").is_displayed()
            
        qty_value = self.driver.find_element(By.CSS_SELECTOR, ".line_item_quantity").get_attribute("value")
        self.assertEqual(qty_value, "3", "The item quantity is not the same we typed in!")
            
        cart_total = self.driver.find_element(By.CSS_SELECTOR, ".cart-total > .lead").text
        self.assertEqual(cart_total, "$68.97", "The total cart value does not match our expectations. That's a blocker!")

    What’s new here?

    If we look back at our first landing page test, you will notice that we use a lot of new things. We will cover them one by one now.

    CSS selectors

    One thing that catches the eye is that we looked for elements by using CSS expressions. You may have heard about them from your frontend peers, or maybe you have even worked with them yourself already. If not, don’t worry. We will cover them in a followup post. For now, I will give you a few translations:

    [href='/t/spree']

    „Give me a page element, whose attribute href is equal to ‚/t/spree’“. Quick reminder: href is used in link elements for defining the target URL that is opened, when the user clicks the link.

    a[href*='spree-bag']

    „Give me any link element, whose attribute href contains ’spree-bag‘.“

    img[src$='/spree_bag.jpeg']

    „Give me an image element, whose attribute src ends with ‚/spree_bag.jpg‘.“

    input#quantity

    „Give me an input element, whose HTML id is equal to ‚quantity‘.“ Yes, we could have used By.ID with „quantity“ here, but I wanted to be explicit about the element type input.

    .line_item_quantity

    „Give me an element that has line_item_quantity in its class attribute.“ The leading dot indicates that we are looking for elements that has the specified class. It is roughly equivalent to [class*='line_item_quantity'].

    .cart-total > .lead

    „Give me any element that has the lead class and that’s preceded by any element that has the cart-total class.“

    By using CSS selectors you can be very specific about what element you want to use in your test case. This makes CSS selectors incredibly powerful and versatile while maintaining a solid degree of simplicity, especially compared to XPath (Ugh).

    New properties and methods used in our test case

    Aside from CSS selectors, we used several new methods and properties in our new automated test.

    click() issues a click on the target element. Straight-forward.

    .text contains the element’s written text. In <p>Some text</p> for example, we would get access to „Some text“.

    get_attribute("someHtmlAttribute") gives us the content of an element’s HTML attribute. In our test we fetch an element’s value attribute that is often used to set the content of an input element. Another example might be the link element we talked about earlier. Given we want to have its href attribute for some further verifications. On our page:

    <a id="mylink" href="https://www.example.org">an example link</a>

    Then in our test we could do:

    target_href = self.driver.find_element(By.ID,"mylink")
                  .get_attribute("href")
    self.do_some_shenanigans_on(target_href)

    send_keys("My desired input") imitates key strokes on an element. We use it to type any string we want into our target text input element. In our test, that would be the „quantity“ input box.

    And finally, with clear() we have a little helper that clears an input element from any content. In our test we use that to remove the preset value. Otherwise we would accidentally order 31 bags: „3“ from us and „1“ from the value preset by the page.

    Continuing from here

    Alright, let’s sum up what we accomplished:

    We have seen how to open and close a browser window, do various actions on the page, verify their outcomes and finally we used that to write a fairly large cross test. And all of that on a full fledged ecommerce web app. Great job! Now where should we go next? For the next post, as promised, I’d like to go back one step and give you a deeper introduction to CSS expressions, because they will accompany you for a long time during your test automation journey. Afterwards we have to tackle a big problem you probably already noticed:

    Copious amounts of repetition.

    To tackle that we will leverage Behave, a Behavior Driven Development framework that makes it possible to write test cases in human-readable text form on top of a DRY Python code base. Afterwards we will apply the Page Object Pattern, a test automation – specific design pattern to reduce repetition even more.

    Okay. To keep you busy while I am busy, how about finishing the checkout? As a reminder, you can find the full source code in my bitbucket repo. Or you can spoil yourself a little about Cucumber in Rust.

    See you in the next post!

    Home
    Share it with: