Gitlab CI Error – Cannot create local repository at: /.m2/repository

When you are using downstream pipelines in Gitlab CI for complex Maven Projects as I do, you may have stumbled accross this error at least once: Cannot create local repository at: /.m2/repository. The Gitlab runner tries to create a local Maven repository at /.m2/repository – so in the topmost directory – and fails horribly due to an obvious lack of permissions. Let‘ see what happens here.

The Gitlab CI Setup

In my case I had a shell-based Gitlab runner that executes a test suite based on a few artifacts published by a Docker runner. Due to versioning and system test level realism reasons I was not able to use the artifacts directly. Unfortunate, but it shouldn’t pose that kind of a problem, does it?

Now when it comes to cloning the test suite on my shell runner, this cryptic error has been dropped. And naturally I was like „WTF is this runner trying to do!?“

The root cause in Gitlab CI downstream jobs

What I was not aware of is that per default the „child job“ inherits all the custom and Gitlab-provided runner variables from the triggering parent job that is still executed on a Docker runner. Now since the trigger happens from a Docker runner job, the variables my child job receives are poison for a baremetal environment that is my shell runner. Not least because things actually do happen at / in a Docker-based Gitlab execution environment, which is perfectly fine, but not on a shell runner.

The solution: What I had to do to make the CI jobs work

To fix the problem I had to set this on my trigger job definition:

This did not just fix the error for me, but it also made perfect sense. Due to the different execution environments – shell vs. docker and project A vs. project B – I have a different set of requirements for my Gitlab CI test job. Therefore we have another case of an error leading to better software design. In addition I learned another piece of Gitlab’s sometimes quite obscure default settings.

Conclusion

I hope this helps you during your day to day journey through the jungle that are Gitlab CI downstream jobs. As I’m an avid QA engineer, so if you want to read more about writing actual automated tests. Also I take care of deeper coding basics like working with threads in java. If you’d rather want to read up about Gitlab CI’s inherit keyword, here’s the link to the relevant section of the official Gitlab CI documentation. Feel free to have a look!

Have a great weekend everybody!

Share it with:

Java Threads – an Introduction for all Skill Levels

Today we are talking about threads in Java. Threads are an essential means to implement concurrent algorithms and are therefore a key part of efficient everyday data processing. Imagine having a webserver without threads serving only one customer at a time: Every customer would need to pull a number for every single web request they make and our online shop would have a bad time.

So let’s head into it!

In modern days – Java threads with Lambda expressions

Since Java 8 we have been blessed with Lambda expression. Basically Lambda expressions serve the purpose of unnamed functions and are written as shown here:

(param1, param2, [...], paramN) -> {
    statement1; 
    statement2;
    [...]
    statementN;
}

If you just need one statement, it looks even better:

(param1, [...], paramN) -> statement

By harnessing their power we are able to write beautiful first level function style concurrent algorithms.

Let’s assume we want to sort a list with an even number of integers. We know that all ints in the second half of the list are greater than the greates number in the first half. Then we can do this:

Try it out!

Notice that we feed the threads in our Java code with Lambdas to tell them what to do. Neat, isn’t it?

Legacy Java Threads with Runnable instances

And how has it been done in the past? When I started programming with Java around 5 and 6 we had to declare a full-blown Runnable-instance that we subsequently fed to the thread. Runnable is an interface type which means that during declaration we also had to implement it’s signature method run(). This method finally contains the functionality that the thread is supposed to run. Let me show what I mean:

Try it out!

It was a bloated mess, but since it’s likely in use in modern day code, we got to list it here.

Released in Java 21: Virtual Threads

Virtual threads are part of Project Loom that aims at bringing lightweight and high-throughput concurrency models to the JVM. These virtual threads don’t run on kernel but on JVM level and thus are fully managed by the JVM. That means you as the coder do not need to keep track of them at all. The JVM will do that for you while you can rely on the coding APIs you already know and love. That is because virtual threads are created almost the same way as the native threads we have seen earlier. Let me show what I mean:

As shown above you the virtual threads API hands you a Thread instance that you can use in ways you are already familiar with. Here we wait for our virtual threads to terminate.

Another important advantage over conventional threads is their ability to scale. Since the JVM manages both ressources and lifetime of virtual threads it is done in a thoroughly optimized way. In addition, because these threads run directly in the JVM instead of the OS kernel, we can omit a lot of OS ressource management overhead. Both factors result in a significant increase of throughput.

Conclusion

So long. I hope I could help you effectively processing data by leveraging the power of Java threads. Please note that the examples shown above are optimized for the topic’s illustration only. Your way to sort a list of integers is certainly much better. If you’d like to read more about me doing Java, I have a post for you, where I talk about web test automation with Java. If you are already familiar with that and want to try something new, I recommend trying fuzz testing in Java. You will gain a fresh view on test automation and honestly it’s a blast to bomb an application with just random stuff.

Last but not least, if you want to dive in really deep into the world of Java threads, check out this article about Thread Pooling. It is quite advanced, but intuitive and powerful once you get the hang of it.

You have more questions regarding Java, me or anything else? As usual, feel free to drop me a line down below. I read every single one of your comments. Have a great day!

Home
Share it with:

Character array is missing „e“ notation exponential mark

I observed that many people out there visiting my blog have great interest in a particular NumberFormatException. It goes by the message Character array is missing „e“ notation exponential mark and was mentioned in my post about fuzzing in Java.

Today we will take a more in depth look at it.

How does that NumberFormatException look like?

To visualize the error effectively, let’s look at the following extreme example drawn from my fuzzing post mentioned above.

Consider the following (sub-par implemented) function dollar2euro that takes any input (hopefully it’s a number though!) and tries to convert it from USD to EUR:

public String dollar2euro(Object input){
    BigDecimal inputParsed = new BigDecimal(input.toString());
    BigDecimal dollars = inputParsed.setScale(2, BigDecimal.ROUND_HALF_EVEN);

    BigDecimal multiply = dollars.multiply(BigDecimal.valueOf(0.92));
    BigDecimal euros = multiply.setScale(2, BigDecimal.ROUND_HALF_EVEN);
    return String.valueOf(euros);
}

Now what happens, if we play along the method signature and put in the following characters: „뤇皽“? If you know what they mean, feel free to drop me a comment down below. Please note that this is exactly what my fuzztest tried to do. And of course it drops a heavy NumberFormatException upon us:

Input: 뤇皽
java.lang.NumberFormatException: Character 뤇 is neither a decimal digit number, decimal point, nor "e" notation exponential mark.

And what does it mean?

The problem here is that „뤇皽“ is not a number. The first and second part Character 뤇 is neither a decimal digit number, decimal point are straight forward. But what does the third part nor „e“ notation exponential mark mean?

In maths, we have the option of expressing numbers in power form with base 10. This comes in handy when we want to express Googol (the number that inspired Google’s name) – a 1 with 100 zeroes. Accordingly we would write 10100 instead. In Java you can do something very similar, but there are better sources to check how to properly use the e-format. For our context let’s acknowledge that it is a different way of displaying numbers. Which likewise could not be found in our input.

But how do we fix that?

We have to make sure that we provide a String to the BigDecimal Constructor that looks like a valid number – if we want to provide a String at all. An actual number like a double or int would be even better. That would cost us no more than a change in the method’s signature and a small adaption to line 2. If we really want to provide a String, we still can do that, but then we have to make sure that it is properly formatted. A valid input example would be: „1337.012342„.

If you apply this simple rule, you should be spared from ‚Character array is missing „e“ notation exponential mark‘ errors in the future.

Conclusion

So long! I hope this little post gave you an idea about what the error message ‚Character array is missing „e“ notation exponential mark‘ means. Of course this case was kinda constructed and you probably have a less explicit case. If so, feel free to post it in the comments down below and we see what we can do. But for demonstration purposes it should have provided a clear image of what is going on in your program.

As my linked post about fuzzing in Java implies, I’m an avid test automation person. If you want more about automating tests with Java, check out my introduction tutorial about Selenium in Java. If you are more of the Python person, no problem. Here’s the same Selenium tutorial in Python. And if Python and Java are both too mainstream for you, I recommend my tutorial about Cucumber in Rust .

Happy test automating & have a nice day!

Home
Share it with:

Stable Diffusion: „LayerNormKernelImpl“ not implemented for ‚Half‘

Today I installed the Stable Diffusion Web UI by AUTOMATIC1111 on my ol‘ reliable Macbook Pro 2015 following this stable diffusion installation guide. And not only is it fun to do AI stuffs on an Intel Mac with no GPU whatsoever, of course I also had to run into a „first try error“ ™. This time it was that one:

RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

Okay, cool.

So how do we fix that?

Well it turns out that this „half“ thingy, that relates to floating point sizes, can be turned off. To do that, let’s revisit your stable-diffusion-webui installation directory and open the shell script webui-user.sh using your most beloved code editor. Here you will find the following 2 lines:

# Commandline arguments for webui.py, for example: export COMMANDLINE_ARGS="--medvram --opt-split-attention"
#export COMMANDLINE_ARGS=""

What we got to do now is: First we remove the leading ‚#‚ character of the second line to uncomment it. Next, we fill the variable with "--skip-torch-cuda-test --no-half" to make it look like this:

export COMMANDLINE_ARGS="--skip-torch-cuda-test --no-half"

Restart the stable diffusion web ui using your webui.sh and it should behave as expected. All of that has been tested with stable-diffusion-webui commit cf2772fab0af5573da775e7437e6acdca424f26e, which was the most recent stable version at the time of writing.

If you are on a Windows machine, the fix should be basically the same: Open the webui-user.bat and change COMMANDLINE_ARGS to match the following:

set COMMANDLINE_ARGS="--skip-torch-cuda-test --no-half"

Restart your server and everything should be alright. (Disclaimer: Untested due to the lack of Windows machines.)

Conclusion

So that’s all, hope this helps. If not, feel free to let me know in the comments below. If you find a better fix, or my post is outdated, or you found any other issues, please let me know as well. Let’s keep it as complete as possible for future users. And if you are still curious about working with Python apps or Python in general, here is another quick and handy post about handling environment variables. And here we talk about my second most favorite topic: Test Automation.

Best regards, and have a nice week!

Home

Share it with:

Introduction to Web Test Automation – Java Edition

The internet is an incredible market place hosting millions and millions of software products to make our lives indefinitely easier. To keep up with each other, software grew more and more elaborate to provide us customers with lots of different features and services to finally make us purchase a thing or two. Now due to the resulting increase of complexity we must take special care not to break existing features while developing the next big version brimming full of features. Testing that the next code increment does not break existing stuff is what we testers call „regression testing“, and there is one especially economical way of doing that: test automation.

That’s what we’re going to do today. We gonna test and we gonna automate and we do all that in Java this time. Because everyone knows about Java, right? So let’s write once and run everywhere. Given we have a browser there.

Shopping list

To start our test automation journey – again -, we will work with the following tools. The versions are the ones I used at the time of writing. I might update them in the future.

After downloading and installing these tools from their respective web sites, we’re ready to start our system under test.

Deploying the system under test

To achieve this, we simply run the following command:

docker run --name spree --rm -d -p 3000:3000 spreecommerce/spree:3.6.4

If it went well, a quick glance at http://localhost:3000/ should show a catalog page full of cute Rails merch. Now we will install and prepare Selenium.

Geckodriver installation and management

Contrary to the Python version, we use WebDriverManager to free us from the installation hassle of Geckodriver, hence we are done with this step. The tip was sent to me by my good friend Sho. Thank you, mate. I owe you one! 🙂

Initializing our test automation project

Given a successful Gradle installation as linked in the shopping list we can now create our test project. Please open a terminal in a root directory of your choice and do:

mkdir jata_tutorial && cd jata_tutorial
gradle init

When prompted answer with 1 for a basic project and then 2 for Kotlin as our Gradle DSL.

Of course we will use the new APIs as well, since we love to live on the edge here. Please choose „yes“ here.

Finally, when asked, give your project the default name that is the project root directory’s name. Now your boilerplate project is set up and ready to go.

Now it’s time to define our dependencies. In build.gradle.kts please introduce the following lines:

plugins {
    java
}

group = "your.groupid"
version = "1.0-SNAPSHOT"

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.slf4j:slf4j-api")
    implementation("ch.qos.logback:logback-classic:1.4.5")
    implementation("org.apache.commons:commons-lang3:3.12.0")

    testImplementation("org.seleniumhq.selenium:selenium-java:4.7.2")
    testImplementation("io.github.bonigarcia:webdrivermanager:5.3.1")
    testImplementation(platform("org.junit:junit-bom:5.9.1"))
    testImplementation("org.junit.jupiter:junit-jupiter:5.9.1")
}

tasks.getByName<Test>("test") {
    useJUnitPlatform()
}

This sets up our project with some basic metadata like our groupId and a (somewhat) meaningful version. We defined to use Java as our JVM language and JUnit as our test runner. Additionally we will have access to Maven’s central repository, where Gradle can download our project dependencies that we defined in the section of that exact same name.

With these dependencies, we are able to:

  • automagically download and install webdriver executables for Selenium (in our case: Geckodriver)
  • perform Selenium actions including opening and closing browser windows, enter URLs and clicking stuffs
  • generate random test input strings
  • log things to the console in a nice timestamped format (without being scared of Log4Shell)
  • and do all that in a modern JUnit5 – runner

Everything that we need to do very soon.

The boilerplate

Next up, we create the skeleton of our first test class. In src/test/java, please create a file named CrossTests.java containing the following code. Please note that .java source files should be named after the contained class. Here: CrossTests.

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openqa.selenium.WebDriver;

class CrossTests {

    WebDriver driver;

    @BeforeAll
    static void setupClass() {
        
    }

    @BeforeEach
    void setup() {

    }

    @AfterEach
    void teardown() {

    }

    @Test
    void isOpen() {

    }
}

Great. That looks almost like a real test already. Finally we are going to open a browser window.

Starting and closing the browser

In setupClass we call the WebDriverManager and let it work its Geckodriver-initializing magic.

@BeforeAll
static void setupClass() {
    WebDriverManager.firefoxdriver().setup();
}

Then we ensure that a browser is opened before each of the individual tests marked with @Test:

@BeforeEach
void setup() {
    this.driver = new FirefoxDriver();
    this.driver.manage().timeouts().implicitlyWait(Duration.of(5, ChronoUnit.SECONDS));
}

This opens a Firefox window whenever we execute one of the class‘ test methods. This is a very pure and uncustomised instance; we could configure it further if we wanted. But for our purpose at the moment, it is perfectly fine.

Now before we perform any actions on the page, we must make sure that the browser window is closed appropriately after each test run. In teardown, we do:

    @AfterEach
    void teardown() {
        this.driver.quit();
    }

Caveat: confusion potential

Selenium’s WebDriver class has a close()-method, but please always use quit() instead. It performs a clean browser termination with regards to Selenium’s execution flow, whereas close() will just terminate the browser causing you lots of warnings in the log.

Alright! We have taken care of the browser handling. Time to do stuffs with it.

Opening the landing page

The most basic action ist to open a web page. Thankfully this task is handled by Selenium with a simple one-liner. Given that our sample web shop is up and running, we do:

this.driver.get("http://localhost:3000/");

We introduce the line to the test case method isOpen, which we introduced earlier in our boiler plate. But before we head into executing the test case, we have to be aware of the fact: We as the user can see the page being open, but the computer cannot. We have to programmatically verify that our expectation „landing page is open“ is met.

Verifying the outcome

To do that we verify these 3 conditions:

  • the window title contains the landing page’s HTML title
  • the most prominent logo exists on the web page and
  • it’s actually visible (i.e. it does not have its display property set to none).

We add this code directly below the line that opens the website:

assertTrue(this.driver.getTitle().contains("Spree Demo"),
           "'Spree Demo' is not in the browser's title.");

WebElement logoElement = this.driver.findElement(By.id("logo"));
assertTrue(logoElement.isDisplayed());

The first line peeks into the page title, which is another staple of Selenium, and then checks if it contains „Spree Demo“. Using JUnit’s assertTrue, we verify that this is the case, and if not, we print a custom error message. Condition 1 done.

The second line depicts the method you will probably use the most whilst moving forward in your test automation career: Webdriver#findElement goes through the page’s DOM and grabs the element that meets the given criteria. Here we search for an element with the HTML id „logo“. If it fails to find the element, it drops a NoSuchElementError. Therefore, we just implicitly verified condition 2.

The return value is a WebElement object that we can do clicks, inputs and various other actions on. We will use the object’s method isDisplayed() to check if the logo is visible therefore verifying our 3rd condition.

Straight-forward so far. At this point your code should look like that:

import io.github.bonigarcia.wdm.WebDriverManager;
import org.junit.jupiter.api.*;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;

import java.time.Duration;
import java.time.temporal.ChronoUnit;

import static org.junit.jupiter.api.Assertions.*;

class CrossTests {

    WebDriver driver;

    @BeforeAll
    static void setupClass() {
        WebDriverManager.firefoxdriver().setup();
    }

    @BeforeEach
    void setup() {
        this.driver = new FirefoxDriver();
        this.driver.manage().timeouts().implicitlyWait(Duration.of(5, ChronoUnit.SECONDS));
    }

    @AfterEach
    void teardown() {
        this.driver.quit();
    }

    @Test
    void isOpen() {
        this.driver.get("http://localhost:3000/");
        assertTrue(this.driver.getTitle().contains("Spree Demo"),
                "'Spree Demo' is not in the browser's title.");

        WebElement logoElement = this.driver.findElement(By.id("logo"));
        assertTrue(logoElement.isDisplayed());
    }

}

Let’s execute it. In a terminal window, please switch to the project root and do:

./gradlew test

If everything went well, your console should display a big OK:

BUILD SUCCESSFUL in 7s
4 actionable tasks: 2 executed, 2 up-to-date

Congratulations, you just executed your first Java-Selenium-test:
Our shop’s landing page can be opened and it is displayed correctly.

What is this driver thingy by the way?

Simply put, the driver – in our case Geckodriver – is the glue between our code and Firefox. The same would be valid for Chrome with Chromedriver respectively. This means it is the interface to our browser handling all the requests that we perform :

  • findElement
  • click
  • „are you displayed?“
  • perform key strokes
  • „grab that text of an element“
  • etc. pp.

Under the hood, drivers are http server applications that receive requests sent by Selenium everytime we perform an action similar to these mentioned above. The driver then interacts with the browser in the way we tasked it to and sends a JSON-based response that we can base our future work on. Everything is encapsulated by Selenium so that we can work with simple objects during automating our tests without having to deal with HTTP or JSON.

Performing and validating browser actions

Now that we know the basics of how to write automated tests with Java and Selenium, let’s develop a more complex test.

The best candidates for automation are cross tests. These are tests that cover representative cross sections of the product. One example could be: Put items into the cart, view the cart page and perform the checkout. It is a great practise to start test automation for new projects by coding these cross tests first.

For the sake of brevity here, we will write up the test until reaching the checkout page. The rest is homework. Don’t worry, I got you covered with the full source code in my bitbucket repo.

Ok, without further ado, here’s the code. Please add the following method to the test class.

@Test
void findSpreeBagAndCheckout()throws InterruptedException{
    this.driver.get("http://localhost:3000/");

    // Landing and catalog page
    this.driver.findElement(By.cssSelector("[href='/t/spree']")).click();
    this.driver.findElement(By.cssSelector("a[href*='spree-bag']")).click();

    // Product page

    this.driver.findElement(By.cssSelector("input#quantity")).clear();
    this.driver.findElement(By.cssSelector("input#quantity")).sendKeys("3");
    this.driver.findElement(By.id("add-to-cart-button")).click();

    // Cart page
    assertTrue(this.driver.getTitle().contains("Shopping Cart"),"'Shopping Cart' is not in the browser's title.");
    this.driver.findElement(By.cssSelector("img[src$='/spree_bag.jpeg']")).isDisplayed();
    String qtyValue=this.driver.findElement(By.cssSelector(".line_item_quantity")).getAttribute("value");
    assertEquals(qtyValue,"3","The item quantity is not the same we typed in!");

    String cartTotal=this.driver.findElement(By.cssSelector(".cart-total > .lead")).getText();
    assertEquals(cartTotal,"$68.97","The total cart value does not match our expectations. That's a blocker!");
}

What’s new here?

If we look back at our first landing page test, you will notice that we use a lot of new things. We will cover them one by one in the next section.

CSS selectors

One thing that catches the eye is that we looked for elements by using CSS expressions. You may have heard about them from your frontend peers, or maybe you have even worked with them yourself already. If not, don’t worry. We will cover them in a followup post. For now, I will give you a few translations:

[href='/t/spree']

„Give me a page element, whose attribute href is equal to ‚/t/spree'“. Quick reminder: href is used in link elements for defining the target URL that is opened, when the user clicks the link.

a[href*='spree-bag']

„Give me any link („anchor“) element, whose attribute href contains ’spree-bag‘.“

img[src$='/spree_bag.jpeg']

„Give me an image element, whose attribute src ends with ‚/spree_bag.jpg‘.“

input#quantity

„Give me an input element, whose HTML id is equal to ‚quantity‘.“ Yes, we could have used By.ID with „quantity“ here, but in this case I wanted to be explicit about the element type input.

.line_item_quantity

„Give me an element that has line_item_quantity in its class attribute.“ The leading dot indicates that we are looking for elements that has the specified class in its class list. It is roughly equivalent to [class*='line_item_quantity'].

.cart-total > .lead

„Give me any element that has the lead class and that’s preceded by any element that has the cart-total class.“

By using CSS selectors you can be very specific about what element you want to use in your test case. This makes CSS selectors incredibly powerful and versatile while maintaining a solid degree of simplicity, especially compared to XPath (ugh).

New properties and methods used in our test case

Aside from CSS selectors, we used several new methods and properties in our automated test.

click() issues a click on the target element. Straight-forward.

.getText() returns the element’s written text. In <p>Some text</p> for example, we would get access to „Some text“.

getAttribute("someHtmlAttribute") gives us the content of an element’s HTML attribute. In our test we fetch an element’s value attribute that is often used to set the content of an input element. Another example might be the link element we talked about earlier. Given we want to have its href attribute for some further verifications; on our page:

<a id="mylink" href="https://www.example.org">an example link</a>

Then in our test we could do:

targetHref = this.driver.findElement(By.id("mylink"))
              .getAttribute("href");
this.doSomeStuffsOn(targetHref);

.sendKeys("My desired input") imitates key strokes on an element. We use it to type any string we want into our target text input element. In our test for example that would be the „quantity“ input box.

And finally, with clear() we have a little helper in our toolbox that removes any content from an input element. In our test we use that to remove the preset value. Otherwise we would accidentally order 31 bags: „3“ from us and „1“ from the value preset by the page.

Continuing your test automation journey from here

Alright, let’s sum up what we accomplished:

We have seen how to open and close a browser window, do various actions on the page and verify their outcomes. Finally, we used that to write a fairly large cross test, all of that on a full fledged ecommerce web app. Great job! Now where should we go next? For the next post, as promised, I’d like to go back one step and give you a deeper introduction to CSS expressions, because they will accompany you for a long time during your test automation journey. Afterwards we have a big problem to solve that you probably already noticed:

Copious amounts of repetition.

To tackle that we will leverage cucumber-jvm, a Behavior Driven Development framework that makes it possible to write test cases in human-readable text form on top of a DRY Java code base. Afterwards we will apply the Page Object Pattern, a test automation – specific design pattern that reduces repetition even more.

Okay! To keep you busy while I am busy, how about finishing the checkout? As a reminder, you can find the full source code in my bitbucket repo at any time. Alternatively you can spoil yourself a little about Cucumber in Rust or try Python instead of Java.

Stay curious and see you in the next post!

Home
Share it with:

Introduction to Web Test Automation in Python

The internet is an incredible market place hosting millions and millions of software products to make our lives indefinitely easier. To keep up with each other, software grew more and more elaborate in order to provide us customers with lots of different features and services to make us purchase a thing or two. Now due to the resulting increase of complexity we must take special care not to break existing features while developing the next big version. Testing that the next code increment does not break existing stuff is what we testers call „regression testing“, and there is one especially economical way of doing that: test automation.

That’s what we’re going to do today. We gonna test and we gonna automate and we do all that in Python, because with its well-designed concepts Python is suitable for beginners and experts alike. So let’s start!

Shopping list

To start our test automation journey, we will work with the following tools. The versions are the ones I used at the time of writing. I might update them in the future.

  • Docker, preferably in its latest version
  • Python 3 version 3.10.8
  • Firefox Browser 106.0.1
  • geckodriver for Firefox 0.32.0
  • a Spree Commerce docker image version 3.6.4 as our system under test
    (will be pulled automagically in the next step)
  • Selenium for Python 4.5.0
  • unittest as our first test runner (comes with Python 3)

After downloading and installing Docker and Python from their respective web sites, we’re ready to start our system under test.

Deploying the system under test

To achieve this, we simply run the following command:

docker run --name spree --rm -d -p 3000:3000 spreecommerce/spree:3.6.4

If it went well, a quick glance at http://localhost:3000/ should show a shopping catalog page full of cute Rails merch. Now we will prepare and install Selenium.

Installing Geckodriver and Selenium

First we need to download geckodriver that will be responsible for sending our actions to the browser. Please see our shopping list for a download page. Download and extract the appropriate archive and put the extracted executable on your PATH. Please do that in a way that fits your OS and your preferences. I’m a mac user, therefore I move it to /usr/local/bin:

$ mv ~/Downloads/geckodriver /usr/local/bin/ 

If everything went well,

$ geckodriver -h 

should print the version and an instruction page. If not, feel free to drop me a Q in the comments below. Otherwise we move on to installing Selenium.

We will leverage pip3 to quickly go through that step:

pip3 install selenium==4.5.0

Now we’re all set. Let’s go headlong into the real trouble.

Test runner boilerplate

First of all, we create the skeleton of our test class. We write a simple Python unittest class that we will fill in with Selenium web test automation goodness as we go further. Our goal is to see the basic structure of a unittest-based automated test case that we will leverage to execute our browser magic. In your favorite project directory, please create a file named cross_tests.py and fill it with the following code. We will talk about the file name a bit later. Don’t worry, it was not a technical decision.

import unittest

# Start with a base class and derive it from 
# unittest.TestCase
class TestSpreeShop(unittest.TestCase):

    # setUp is executed at the start...
    def setUp(self):
        pass

    # ... and tearDown at the end of each test case.
    def tearDown(self):
        pass

    # This will be our first test:
    # We will open the shop's homepage
    # and verify that it worked.
    # For now, we will just let it pass.
    def test_isOpen(self):
        pass


# The main entry point of our unittest-based execution 
# script. Think of it as an actual main method similar 
# to Java's or C's.
if __name__ == '__main__':
    unittest.main()

Great, that almost looks like a real test already. Now finally we are going to open a browser window.

Starting and closing the browser

We write a small support function:

def open_browser(self):
    return webdriver.Firefox()

And we apply it in setUp:

def setUp(self):
    self.driver = self.open_browser()

This opens a Firefox window it its most pure form whenever we execute one of the class‘ test methods. We could configure it further in open_browser, but for our purpose at the moment, this is perfectly fine.

Now before we perform any actions on the page, we must make sure that the browser window is closed appropriately after each test run. In tearDown, we do:

def tearDown(self):
    self.driver.quit()

Caveat: confusion potential

Selenium’s webdriver has a close()-method, but please always use quit() instead. It takes care of clean browser termination with regards to Selenium’s execution, whereas close() will just terminate the browser causing warnings.

Alright! We have taken care of the browser handling. Time to do stuffs with it.

Opening the landing page

The most basic action ist to open a web page. Thankfully this task is handled by Selenium with a simple one-liner. Given our sample web shop is up and running, we do:

self.driver.get("http://localhost:3000/")

We introduce the line to the test case method test_isOpen, which we introduced in our boiler plate. But before we head into executing the test case, we have to be aware of the fact that we as the user can see the page being open, but the computer can’t. We have to programmatically verify that our expectation „landing page is open“ is met.

Verifying the outcome

To do that we verify these 3 conditions:

  • the window title contains the landing page’s HTML title
  • the most prominent logo exists on the web page and
  • it’s actually visible (i.e. it does not have its display property set to none).

We add this code directly below the get – line from before:

self.assertTrue("Spree Demo" in self.driver.title, 
                "'Spree Demo' is not in the browser's title.")

logo_element = self.driver.find_element(By.ID, "logo")
self.assertTrue(logo_element.is_displayed())

The first line peeks into the page title, which is another staple of Selenium, and then checks, if it contains „Spree Demo“ with Python’s in-operator. Using unittest's assertTrue, we verify that this is the case, and if not, we print a custom error message. Condition 1 done.

The second line depicts the method you will probably use the most whilst moving forward in your test automation career: find_element goes through the page’s DOM and grabs the element that meets the given criteria. Here we search for an element with the HTML id „logo“. If it fails to find the element, it drops a NoSuchElementError. Therefore, we just implicitly verified condition 2.

The return value is a WebElement object that we can do clicks, inputs and various other actions on. We will use the object’s method is_displayed() to check, if the logo is visible therefore verifying our 3rd condition.

Straight-forward so far. At this point your code should look like that:

import unittest
from selenium import webdriver
from selenium.webdriver.common.by import By

# Start with a base class and derive it from unittest.TestCase
class TestSpreeShop(unittest.TestCase):

    # setUp is executed at the start...
    def setUp(self):
        self.driver = self.open_browser()

    # ... and tearDown at the end of each test case.
    def tearDown(self):
        self.driver.quit()

    def test_isOpen(self):
        self.driver.get("http://localhost:3000/")
        self.assertTrue("Spree Demo" in self.driver.title, 
                        "'Spree Demo' is not in the browser's title.")

        logo_element = self.driver.find_element(By.ID, "logo")
        self.assertTrue(logo_element.is_displayed())


    # Support method. We will use it to open a browser instance in setUp.
    def open_browser(self):
        return webdriver.Firefox()

# The main entry point of our unittest-based test execution script.
# Think of it as an actual main method similar to Java's or C's.
if __name__ == '__main__':
    unittest.main()

Let’s execute it by doing:

python3 cross_tests.py # Linux or MacOS
python cross_tests.py # Windows

If all went well, your console should display a big OK:

➜  pyta_tutorial git:(main) python3 cross_tests.py
.
----------------------------------------------------------------------
Ran 1 test in 4.719s

OK

Congratulations, you just executed your first Python-Selenium-test:
Our shop’s landing page can be opened and it is displayed correctly.

What is this driver thingy by the way?

Simply put, the driver is the glue between our code and Gecko (or Chromedriver respectively), thus it’s the interface to our browser window. It handles all the requests that we perform like find_element, click, „are you displayed?“, perform key strokes, „grab that text of an element“ etc. pp. Under the hood, Gecko is an http server, which receives requests sent by the driver everytime we perform one of the above actions. The driver then interacts with the browser in the way we tasked it to and sends a JSON-based response object that we can base our future work on. Everything is encapsulated by Selenium so that we can work with simple objects during test automation development without having to deal with HTTP or JSON.

Performing and validating browser actions

Now that we know the basics of how to write automated tests with Python and Selenium, let’s develop a more complex example.

The best candidates for automated tests are cross tests. These are tests that cover representative cross sections of the product. One example could be: Put items into the cart, view the cart page and perform the checkout. This is the reason we named our test file cross_tests.py. It is good practise to start test automation for a new project by coding these cross tests first.

For the sake of brevity here, we will write up the test until the checkout page. The rest is homework. Don’t worry, I got you covered with the full source code in my bitbucket.

Ok, without further ado, here’s the code. Please add the following method to the test class:

def test_findSpreeBagAndCheckout(self):
    self.driver.get("http://localhost:3000/")

    # Landing and catalog page
    self.driver.find_element(By.CSS_SELECTOR, "[href='/t/spree']").click()
    self.driver.find_element(By.CSS_SELECTOR, "a[href*='spree-bag']").click()

    # Product page
    self.driver.find_element(By.CSS_SELECTOR, "input#quantity").clear()
    self.driver.find_element(By.CSS_SELECTOR, "input#quantity").send_keys("3")
    self.driver.find_element(By.ID, "add-to-cart-button").click()
        
    # Cart page
    self.assertTrue("Shopping Cart" in self.driver.title, 
                  "'Shopping Cart' is not in the browser's title.")
    self.driver.find_element(By.CSS_SELECTOR, "img[src$='/spree_bag.jpeg']").is_displayed()
        
    qty_value = self.driver.find_element(By.CSS_SELECTOR, ".line_item_quantity").get_attribute("value")
    self.assertEqual(qty_value, "3", "The item quantity is not the same we typed in!")
        
    cart_total = self.driver.find_element(By.CSS_SELECTOR, ".cart-total > .lead").text
    self.assertEqual(cart_total, "$68.97", "The total cart value does not match our expectations. That's a blocker!")

What’s new here?

If we look back at our first landing page test, you will notice that we use a lot of new things. We will cover them one by one now.

CSS selectors

One thing that catches the eye is that we looked for elements by using CSS expressions. You may have heard about them from your frontend peers, or maybe you have even worked with them yourself already. If not, don’t worry. We will cover them in a followup post. For now, I will give you a few translations:

[href='/t/spree']

„Give me a page element, whose attribute href is equal to ‚/t/spree'“. Quick reminder: href is used in link elements for defining the target URL that is opened, when the user clicks the link.

a[href*='spree-bag']

„Give me any link element, whose attribute href contains ’spree-bag‘.“

img[src$='/spree_bag.jpeg']

„Give me an image element, whose attribute src ends with ‚/spree_bag.jpg‘.“

input#quantity

„Give me an input element, whose HTML id is equal to ‚quantity‘.“ Yes, we could have used By.ID with „quantity“ here, but I wanted to be explicit about the element type input.

.line_item_quantity

„Give me an element that has line_item_quantity in its class attribute.“ The leading dot indicates that we are looking for elements that has the specified class. It is roughly equivalent to [class*='line_item_quantity'].

.cart-total > .lead

„Give me any element that has the lead class and that’s preceded by any element that has the cart-total class.“

By using CSS selectors you can be very specific about what element you want to use in your test case. This makes CSS selectors incredibly powerful and versatile while maintaining a solid degree of simplicity, especially compared to XPath (Ugh).

New properties and methods used in our test case

Aside from CSS selectors, we used several new methods and properties in our new automated test.

click() issues a click on the target element. Straight-forward.

.text contains the element’s written text. In <p>Some text</p> for example, we would get access to „Some text“.

get_attribute("someHtmlAttribute") gives us the content of an element’s HTML attribute. In our test we fetch an element’s value attribute that is often used to set the content of an input element. Another example might be the link element we talked about earlier. Given we want to have its href attribute for some further verifications. On our page:

<a id="mylink" href="https://www.example.org">an example link</a>

Then in our test we could do:

target_href = self.driver.find_element(By.ID,"mylink")
              .get_attribute("href")
self.do_some_shenanigans_on(target_href)

send_keys("My desired input") imitates key strokes on an element. We use it to type any string we want into our target text input element. In our test, that would be the „quantity“ input box.

And finally, with clear() we have a little helper that clears an input element from any content. In our test we use that to remove the preset value. Otherwise we would accidentally order 31 bags: „3“ from us and „1“ from the value preset by the page.

Continuing from here

Alright, let’s sum up what we accomplished:

We have seen how to open and close a browser window, do various actions on the page, verify their outcomes and finally we used that to write a fairly large cross test. And all of that on a full fledged ecommerce web app. Great job! Now where should we go next? For the next post, as promised, I’d like to go back one step and give you a deeper introduction to CSS expressions, because they will accompany you for a long time during your test automation journey. Afterwards we have to tackle a big problem you probably already noticed:

Copious amounts of repetition.

To tackle that we will leverage Behave, a Behavior Driven Development framework that makes it possible to write test cases in human-readable text form on top of a DRY Python code base. Afterwards we will apply the Page Object Pattern, a test automation – specific design pattern to reduce repetition even more.

Okay. To keep you busy while I am busy, how about finishing the checkout? As a reminder, you can find the full source code in my bitbucket repo. Or you can spoil yourself a little about Cucumber in Rust.

See you in the next post!

Home
Share it with:

Google Playstore Releases: Things I Wish I Knew Earlier

So recently I released my first app in the Google Playstore: a simple real estate calculator for the German market that goes by the name Hauschecker. I happily and avidly coded, did some changes here, made some learnings there, and eventually an MVP was finished that looked awful, but did its job. In accordance to Eric Ries‘ awesome book „The Lean Startup“, I want to release fast and start gathering feedback as quickly as possible. So I went to the Google Play Console ready to upload my created-with-love app bundle…

…and… yeah.

Google Playstore poses quite a challenge

But the good thing with challenges is: You learn and you get better. And Playstore-releases are not that much different from running a marathon, getting a productive appointment in a German Bürgeramt or, well, developing an app. Therefore let’s head into the things I learned the hard way so you don’t have to.

A trivial one for starters: An app icon is required.

And with „required“, I mean required-HTML-multipart-form-field-required. A form field with that nasty little asterisk character ‚*‘ that tells you there will be no going-forward without filling it in. And as if that’s not enough, Google has a very specific opinion about how it’s supposed to look like: 32bit PNG -format with 512×512 px, and God forbid it’s larger than 1meg. And in addition it hast to follow Google’s icon design specs, which we happened to fulfill apparently. In addition, we cannot use the Flutter default app icon, as Google has the copyright. (Oh btw, I use Flutter.)

Cost: about 20 bucks. Maybe creating a simple dummy here is possible. Memo to myself for the next app.

This one’s a bit nastier: A feature graphic is required.

The next thing I was confronted with is the concept of feature graphics, that is a required form field, too. And again: Google is very specific about that picture: a 1024×500 px JPEG or 24bit PNG without alpha. Okay..

I get that it makes sense: It explains what the app is supposed to do, conveys feelings and shows use cases in an intuitive way, and in addition it plays a major role, if you have a marketing video for your app. But it was certainly a thing to make – or rather have it made.

Cost: about 40 bucks.

25$ for a Google Playstore dev account

This one hit me cold: Google Paystore (pun semi-intended) charges you a one time fee of 25$. At first it was a surprise to me, but on the other hand it’s still cheap compared to what Apple is doing with their 99 bucks a year. I clenched my teeth, paid and moved on.

Cost: well, 25 bucks.

A little bit of text to the mix

Google also wants us to provide a short and a long description. Okay. I would be a lousy internet entrepreneur-in-training, if I couldn’t explain my idea with a few words, so that’s fine by me.

Be prepared to bring screenshots

Another important (and required) Playstore asset is screenshots. Screenshots for phone screens as well as 7“ and 10.5“ tablet screens. To create these screenshots I ran the app in a Chrome instance using the CLI command flutter run first. Then I copied the localhost link to the resulting webapp and pasted it into a Firefox, because I prefer its dev and screenshot tools. As an alternative you could use appropiate phone and tablet emulators with a desktop snipping tool, or you could use Chrome directly, but all these were too much of an unnecessary hassle for me. So let’s continue with Firefox. We open the dev tools, click on the little devices icon to the right and choose an appropiate phone or „ipad“ for the respective device sizes. The webapp is shown in the specific dimensions and now we use its screenshot tool by right-clicking into the app and use the „Take screenshot“ function. We change some inputs and take another one. Rinse and repeat until we have the required 2 screenshots for each device type.

Ironically, as diligent as Google usually is regarding their specifications and form field validations, they let me do this:

Misplaced phone screenshot within the device outline provided by and shown in the Google Playstore.

Thanks, Google…

Playstore Queries & Questionaires

Be prepared for a huge interview about various different topics of major and minor significance; Google wants to know eeeverything. I mean, child protection!? I just want to do some basic percetage calculation. But it can’t be helped… we have to push through. Thankfully, most of the questions are basic multiple choices and checkboxes. They won’t impose too much of a hassle. But it was a surprise for me as a firsttimer.

Sometimes you win, sometimes you learn – and then you win

Phew, that was a journey. My first release for Hauschecker 1.0.0 took me a whooping two and a half weeks, and I had to delay a lot of things in life (and spend a significant amount of money) to make it happen. But I pushed through, and now having a full-fledged working Android app in the Google Playstore is such a great feeling I can’t describe. In fact, it was so great I recently pushed version 1.0.1. So for today I hope I could show you my learnings and hopefully you will make it in half the time it took me. If not, feel free to drop me a question in the comments below. If you are interested: Here I talk about another big accomplishment I achieved in my career. This one tells about the ISTQB Test Management Certificate and the perks it brings to the table.

Have a great day & happy coding!

Home
Share it with:

AccessDenied in psutil

psutil is an interesting Python package that provides us with valuable insights about running processes, their memory and CPU usage and many more key aspects for monitoring and profiling processes across all major platforms. Thus, it is an incredibly useful tool for system admins, developers and testers alike. Now one of my favourite methods is cmdline(). Defined in the package’s Process class, it yields the whole CLI command of a running process in array form. There is one important catch on Windows though that got me kinda by surprise: While iterating casually through all my processes I suddenly got an AccessDenied error.

What did I try?

For an advanced verification in a process management – related test case I wanted to print the executed CLI command in a log file, so basically this:

import psutil

for pc in psutil.process_iter():
    print(pc.cmdline())

But that will just throw a painful AccessDenied error at us. Uff.

Why did that happen?

The reason is that psutil is quite consequent: It really lists every running process. That means even SYSTEM or root processes . That’s okay, it might even be interesting in one case or another, and you can still access selected attributes like the name() of a root process, if you want. But for me, it doesn’t suffice. I want to see the full-fledged cmdline(), but I understand that SYSTEM processes are none of my business. Once I came to term with that fact, the solution was easy: We just skip them.

The solution

What I did was applying a try-except around the loop’s inner statement:

import psutil

for pc in psutil.process_iter():
    try:
        print(pc.cmdline())
    except psutil.AccessDenied:
        continue

The continue statement will make sure that the processes that I’m not supposed to see are happily skipped without hurting the rest of the program flow.

But what if I need to monitor foreign processes?

In that case, we would need to execute the script within the process owner’s user context. That might be a bit fiddly in Windows depending on the use case, but of course that’s still possible. Just remember to keep the try-except block, because there still will be processes you wouldn’t be allowed to see.

Mine, for example.

Conclusion

So far for today. I hope this little Q3A (quick question quick answer) could shed some light upon that surprising AccessDenied error. If you want to learn more about psutil, I strongly recommend the readthedocs page. Otherwise, if you want to see more quick tips, I have one more for Python about environment variables. That one covers Python’s environment variable handling. As an alternative, if you are – like me – into containers, here is a handy docker ps trick useful for monitoring tasks as well.

Happy coding!

Home
Share it with:

Python Environment Variables: getenv() vs. environ[ ]

Last week, I was about to execute a test run using one of our functional test suites that requires a certain environment variable – and I forgot to set it, whoops.. The answer was harsh and generic: A good ol‘ Python KeyError. That put me up with two questions:

  1. What options do I have to get environment variables with Python and
  2. which one is the best?

Let’s start checking them out.

Introducing os.environ[key]

The first option is a simple dictionary that is prefilled with all environment variables when starting the Python process. This provides us with a simple-to-use interface that we already know and love.

Let’s try it out:

For python environment variables with os.environ[key], do the following: export MY_VAR="test", python3 -i, import os and os.environ["MY_VAR"]. This should yield 'test'.
Python environment variables with os.environ[key]

But what happens, when we query a variable that is not set? Then we hopefully prepared our try-except Block, because that call is going down the river faster than sound. This happens because dictionaries throw generic KeyErrors, when the queried key could not be found in the dict. That’s what happened in my test run.

os.getenv(key, default=None)

The second option follows a more higher-level approach by providing a function that takes our key in question and an optional default value. The usage is straight forward and it won’t crash as harshly as the first option does.

Let’s try it out:

For python environment variables with os.getenv(key, default), do the following: export MY_VAR="test", python3 -i, import os and os.getenv("MY_VAR"). This should yield 'test'. If you query MY_VAR2 instead, it should yield None. If you do os.getenv("MY_VAR2", "default"), it should yield default.
Python environment variables with os.getenv and default value handling

As we can see, we can now use any key we want; worst case is a result of None. But the increased flexibility comes with a price: We have to take care of whatever the function returns. Even harder, due to the fact that the default defaults to None (pun semi-intended), this solution is prone to hidden bugs. Therefore use it with care and set an appropriate default value if possible.

Conclusion: Which one works best for you?

We saw both options now, but which one works best (for you)? At the end of the day, it comes down to preference regarding two factors:

  1. Do you like your code to be more low-level or high-level?
  2. Do you want your fails fast or more controlled?

This is a decision you have to make, but once you have it, you find everything you need to get your environment variables with Python right here.

So long for the quick journey into my Python life. Usually I talk more about Rust as in my Cucumber Rust – tutorial and in this cheat sheet about Rust modules. But since Python is my main language on the job, there will definitely be more coming up soon. So if you are a friend of british comedy, feel free to stay tuned. Additionally, since container technologies are getting more and more important in my day job as well, I will write more about these techs, too, starting with this little docker ps – trick. Or, if you rather keep coding, check out my new Java threads tutorial.

Happy holidays and a merry Christmas Eeve!

Home
Share it with:

docker ps formatting – Cheat Sheet

Checking the status of Docker containers is an important starting point, when it comes to debugging container clusters. docker ps is an ideal tool to get you started. Unfortunately, the output can sometimes be quite verbose and overwhelming.

The --format parameter is supposed to ease the pain here, but Go templates can be quite tedious to write every time, too. To counter that a little, I reveal my favorite ready-to-alias docker ps calls in today’s post. I hope they make your everyday Docker life a little easier.

# Are my expected containers up?
docker ps --format "table {{.Names}}\t{{.Status}}"

# What ports do my containers expose?
docker ps --format "table {{.Names}}\t{{.Ports}}"

# What networks are my containers assigned to?
docker ps --format "table {{.Names}}\t{{.Networks}}"

All of them can be extended with more docker ps command line parameters, even when aliased later on.

As you may have noticed, my first column is always the container name followed by the property I’d like to analyse. This is my personal way to keep the output compact and focused on the problem at hand, but of course you can query any property you like. Check out the official documentation for a comprehensive list of available properties and pick what you need for the task.

Do you have a favourite formatting? Or do you want to talk about any other question? Feel free to let me know in the comments below. I plan to publish more cheat sheet-style posts in the future. This helps me out to share recent learnings in a focused way and provides you with quick help and valuable insights. You can find another example here, where I demonstrate how to set up Linux users with low maintenance work afterwards. Or if you are into Rust like I am, the Rust modules cheatsheet might be a valuable companion, when it comes to structuring your code with Rust Modules.

Until next time!

Home
Share it with: