Moving from TestNG to Cucumber

This is a guest post by Monika Khandelwal, a Member of the Technical Staff at Qubole.

Qubole offers an enterprise-grade, cloud-native big data analytics platform and we’re always making improvements to ensure we provide the highest quality of service to our customers.

In this blog post, we’ve outlined our motivation to switch from TestNG to Cucumber. This move turned out to be great even though Cucumber is generally considered to be a specification tool and not a test framework. In addition, because the majority of the analyses & workflows processed by our customers are complex and domain-driven in nature, the need to use a collaborative test tool was of utmost importance.

Challenges with TestNG

For UI testing, initially, we were using the Selenium TestNG framework which was running almost 250-300 test cases as part of our Functional Test Suite. We were using all of the available benefits of TestNG like grouping, ordering, parallel execution, etc.

At a minor scale, when the product had limited features with fewer test cases and a small team, TestNG was easy to maintain. But as the product and the team continued to grow, we began to experience communication gaps. Some of the frequent questions being asked were :

1. What tests are we running?
2. How will we find missing scenarios?
3. How much is the test coverage?
4. Can anyone add/fix tests?
5. How can we debug/analyze the tests?
6. How do users read the reports?

Reports generally contained stack traces that were hard to decipher by all the team members, especially those from a non-technical background. Because of this, we wanted to add a degree of usability across the board.

Migrating to Cucumber

In his blog, Aslak Hellesøy writes that Cucumber was born out of the idea of automating acceptance tests, functional requirements, and software documentation into one format.

Cucumber uses a business vocabulary, which is well understood by anyone on the team including end-users, stakeholders, and the engineering team. It increases the readability of reports by giving a clear picture of what is being executed and what is missing.

Technical details

We could reuse our Selenium WebDriver framework following a Page Object Model design pattern. Additional work was to eliminate all TestNG-related files and introduce the Cucumber-specific Feature and Step Definition files. 

Here is an example of the transformation from TestNG to Cucumber

Below is a TestNG test case to run a hive query and verify the result:

@Test(groups = { "analyze.hive" }, priority = 1) public void test_RunHiveQuery() throws Exception {
    homePage.goToAnalyze().compose().selectHiveCommand().hiveStatementOrPath("Query Path").enterHivePath(AnalyzeV2DataProvider.generateCommandDetails(CommandTypes.HIVE, QueryType.valueOf(dataType), "").get("query")).runQuery().waitForQueryToComplete(CommandStatus.SUCCESS).assertForEmptyResultQuery();
}

Using Cucumber, we transformed the same test case into a business use case, which everybody could understand. Our test cases became use cases written in plain English as shown below:

#1 Run a Hive Query and verify various Parameters Scenario: Given User is on Analyze Page When User enters path "HIVE_QUERY_PATH_PRODUCING_EMPTY_RESULTS" and run Then Verify results are empty

Cucumber scenarios are not an executable form of tests. To make it work, Step Definition contains the supporting code. It acts as the translator between the feature file and the base code which can be written as follows:

public class HiveSteps
{
    CommonResourcesSetup commonDeclaration = CommonResourcesSetup.getInstance();

    @Given("^User is on Analyze Page$")
    public void navigateToAnalyzePage() throws Throwable
    {
        commonDeclaration.homePage.goToAnalyze();
    }

    @When("^User enters path \"([^\"]*)\" and run$")
    public void selectHiveQueryPath(String dataType) throws Exception
    {
        commonDeclaration.analyzeCommon.compose().selectHiveCommand().hiveStatementOrPath("Query Path").enterHivePath(AnalyzeV2DataProvider.generateCommandDetails(CommandTypes.HIVE, QueryType.valueOf(dataType), "").get("query")).runQuery().waitForQueryToComplete(CommandStatus.SUCCESS);
    }

    @Then("^Verify results are empty$")
    public void verifyResultsAreEmpty() throws Exception
    {
        commonDeclaration.analyzeCommon.assertForEmptyResultQuery();
    }
}

The JUnit runner class is used to trigger the tests. It points to the feature file to be picked and executed.

@RunWith(Cucumber.class)
@CucumberOptions(format = { "pretty", "html:target/cucumber", "json:target/cucumber-analyze.json" }, glue = { "commonsteps", "analyzev2" }, features = "src/test/resources/Features/AnalyzeV2", tags = { "~@toBeAutomated", "~@newaccount", "~@smoke" })
public class AnalyzeTest
{
}

By default, Cucumber provides a lot of built-in functionality like background and hooks to better handle and manage your test suite. Similar to groups in TestNG, we can group a set of tests under a tag in Cucumber and choose to either include or exclude them during execution.

End to End Test

We created a feature file called end to end.feature and documented around 20 scenarios, which incorporate frequently used workflows and how a new user will use the product. For any new hire, executing those Scenarios manually acts as a product walk-through. We also use it as the basis for our release sign-off.

Contribution

Cucumber has the ability to support more than 60 spoken languages. As a user-friendly tool, it has encouraged developers to contribute to the feature file, review the scenarios and identify missing tests.

Reporting

For reporting, we have installed the Cucumber Reports Plugin into our Jenkins job and added the job to a nightly run. It publishes a neat tabular as well as graphical report, which is evaluated across the team and stakeholders. Below is an example of a passed test case:

In case of failure, it attaches a screenshot link and adds the URL at failure as shown below: 

In Summary

Incorporating Cucumber in UI functional tests has increased the overall business value as we get a broader picture of how our customers use the product. Feature files are written soon after the requirements are frozen, thus business stakeholders can give feedback at an early stage.

Test code maintenance is low and distributed because it is easy for anyone to add a new feature file or update the existing one. After all, we shouldn’t forget “quality is everyone’s responsibility!”