A Conclusive Allure Report Tutorial with Sample Report

12 months ago 42

In our Allure Report Tutorial, we will be explaining the steps to generate an Allure report and show you the different features that are available as reporting is an integral part of every automation testing framework. Without proper reports,...

In our Allure Report Tutorial, we will be explaining the steps to generate an Allure report and show you the different features that are available as reporting is an integral part of every automation testing framework. Without proper reports, you will not be able to analyze the results of your execution and take further action. Though automation tests are created and executed by people with technical knowledge, the people who read the reports may not possess too much technical knowledge. So it is important for reports to be visually strong and easy to understand.

We have chosen Allure as it is an open-source test reporting tool that can be used to represent the results of your test execution concisely in the form of interactive and comprehensive web reports. We also have first-hand experience in using Allure reports to create top-notch reports while delivering high-quality automation testing services to our clients. So using that experience, we have created and used a sample report in our Allure Report tutorial to make it easier for you to understand.

Allure Report Tutorial

You’d have to follow the below-mentioned installation steps if you haven’t yet installed Allure on your computer. If you already have Allure installed and are aware of the dependencies, you can directly proceed further in our Allure Report Tutorial. We have listed the important annotations you’ll have to know, defined the project structure, and provided the code snippets you can use to create your Allure report.

Installation & Set up

1. Download the latest version as a zip archive from Github Releases.

2. Unpack the archive to the allure-commandline directory.

3. Navigate to the bin directory.

4. Use allure.bat for Windows or allure for Unix platforms.

5. Add allure to the system PATH.

Note: Allure CLI requires Java Runtime Environment to be installed.

Add the Dependencies & Run

Update the Properties section in the Maven pom.xml file Add Selenium, JUnit4 and Allure-JUnit4 dependencies in POM.xml Update Build Section of pom.xml in Allure Report Project. Create Pages and Test Code for the pages

Once you have completed all the above steps, you can run the test and generate your Allure Report by following the steps mentioned later on in our Allure report tutorial. Now let’s take a look at the code for your POM.xml file.

Pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.example</groupId> <artifactId>Selenium-Allure-Demo</artifactId> <version>1.0-SNAPSHOT</version> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <selenium.version>3.141.59</selenium.version> <testng.version>7.4.0</testng.version> <allure.testng.version>2.14.0</allure.testng.version> <maven.compiler.plugin.version>3.5.1</maven.compiler.plugin.version> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> <aspectj.version>1.9.6</aspectj.version> <maven.surefire.plugin.version>3.0.0-M5</maven.surefire.plugin.version> </properties> <build> <plugins> <!-- Compiler plug-in --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>${maven.compiler.plugin.version}</version> <configuration> <source>${maven.compiler.source}</source> <!--For JAVA 8 use 1.8--> <target>${maven.compiler.target}</target> <!--For JAVA 8 use 1.8--> </configuration> </plugin> <!-- Added Surefire Plugin configuration to execute tests --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${maven.surefire.plugin.version}</version> <configuration> <suiteXmlFiles> <suiteXmlFile>TestNG.xml</suiteXmlFile> </suiteXmlFiles> <argLine> -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar" </argLine> </configuration> <dependencies> <!-- https://mvnrepository.com/artifact/org.aspectj/aspectjweaver --> <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjweaver</artifactId> <version>${aspectj.version}</version> </dependency> </dependencies> </plugin> </plugins> </build> <dependencies> <!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java --> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <version>${selenium.version}</version> </dependency> <!-- https://mvnrepository.com/artifact/org.testng/testng --> <dependency> <groupId>org.testng</groupId> <artifactId>testng</artifactId> <version>${testng.version}</version> <scope>test</scope> </dependency> <!-- https://mvnrepository.com/artifact/io.qameta.allure/allure-testng --> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-testng</artifactId> <version>${allure.testng.version}</version> <scope>test</scope> </dependency> </dependencies> </project>

Project structure with Allure Report:

In our Allure report tutorial, we have basically kept two packages called ‘pages’ and ‘tests’. The pages package will include all the pages you want to test and the tests package will include the different tests that you want to perform.

Project Structure With Allure Report

Pages Package:

There are two pages that we have added to our package, and they are the Dashboard page and the Login page.

Dashboard page:
package AllureDemo.pages; import io.qameta.allure.Step; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.testng.Assert; public class DashboardPage { WebDriver driver; By dashboardPageTitle = By.xpath("(//span[.='Dashboard'])[1]"); By options = By.xpath( "//div[@title='Assign Leave']"); public DashboardPage(WebDriver driver) { this.driver = driver; } @Step("Verify title of Dashboard page") public void verifyDashboardPageTitle() { String DashboardPageTitle = driver.findElement(dashboardPageTitle).getText(); Assert.assertTrue(DashboardPageTitle.contains("Dashboard")); } @Step("Verify Quick Launch Options on Dashboard page") public void verifyQuickLaunchOptions() { String QuickLaunchOptions = driver.findElement(options).getAttribute("title"); Assert.assertTrue(QuickLaunchOptions.contains("Assign Leave")); } }
Login page:
package AllureDemo.pages; import io.qameta.allure.Step; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.testng.Assert; public class LoginPage { WebDriver driver; By userName = By.name("username"); By password = By.name("password"); By titleText = By.name("username"); By login = By.xpath("//button[@type='submit']"); By errorMessage = By.xpath("//p[.='Invalid credentials']"); public LoginPage(WebDriver driver) { this.driver = driver; } // Set user name in the textbox public void setUserName(String strUserName) { driver.findElement(userName).sendKeys(strUserName); } // Set password in the password textbox public void setPassword(String strPassword) { driver.findElement(password).sendKeys(strPassword); } // Click the login button public void clickLogin() { driver.findElement(login).click(); } @Step("Verify title of Login Page") public void verifyPageTitle() { String loginPageTitle = driver.findElement(titleText).getAttribute("name"); Assert.assertTrue(loginPageTitle.contains("username")); } /* Failed Test */ @Step("Verify error message when invalid credentail is provided") public void verifyErrorMessage() { String invalidCredentialErrorMessage = driver.findElement(errorMessage).getText(); Assert.assertTrue(invalidCredentialErrorMessage.contains("Invalid credentials")); } @Step("Enter username and password") public void login(String strUserName, String strPasword) { // Fill in the user name this.setUserName(strUserName); // Fill in the password this.setPassword(strPasword); // Click the Login button this.clickLogin(); } }

Tests Package:

For our Allure Report tutorial, we have added 3 tests by the names base test, dashboard test, and login test. Let’s take a look at the code snippets for each of these tests below.

Base test:
package AllureDemo.tests; import AllureDemo.pages.DashboardPage; import AllureDemo.pages.LoginPage; import io.qameta.allure.Step; import org.openqa.selenium.WebDriver; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa.selenium.firefox.FirefoxDriver; import org.testng.annotations.AfterClass; import org.testng.annotations.AfterMethod; import org.testng.annotations.BeforeClass; import org.testng.annotations.BeforeMethod; import java.util.concurrent.TimeUnit; public class BaseTest { public static WebDriver driver; LoginPage objLogin; DashboardPage objDashboardPage; @Step("Start the application") @BeforeMethod public void setup() { System.setProperty("webdriver.chrome.driver", "C:\\Users\\Codoid\\Documents\\QA-SAM-March2023-Stories\\resources\\drivers\\chromedriver_win.exe"); driver = new ChromeDriver(); driver.manage().window().maximize(); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); driver.get("https://opensource-demo.orangehrmlive.com/"); } @Step("Stop the application") @AfterMethod public void close() { driver.close(); } }
DashboardTests
package AllureDemo.tests; import AllureDemo.pages.DashboardPage; import AllureDemo.pages.LoginPage; import io.qameta.allure.*; import org.testng.ITestNGListener; import org.testng.annotations.Listeners; import org.testng.annotations.Test; @Epic("Web Application Regression Testing") @Feature("Dashboard Page Tests") @Listeners(TestExecutionListener.class) public class DashboardTests extends BaseTest { LoginPage objLogin; DashboardPage objDashboardPage; @Severity(SeverityLevel.BLOCKER) @Test(priority = 0, description = "Verify Dashboard Page") @Description("Test Description : After successful login to application opens Dashboard page") @Story("Successful login of application opens Dashboard Page") public void DashboardTest() { objLogin = new LoginPage(driver); // Login to the application objLogin.login("Admin", "admin123"); // Go to the dashboard page objDashboardPage = new DashboardPage(driver); // Verify the dashboard page objDashboardPage.verifyQuickLaunchOptions(); } private class TestExecutionListener implements ITestNGListener { } }
LoginTests
package AllureDemo.tests; import AllureDemo.pages.DashboardPage; import AllureDemo.pages.LoginPage; import io.qameta.allure.*; import org.testng.annotations.Listeners; import org.testng.annotations.Test; @Epic("Regression Tests") @Feature("Login Tests") @Epic("Web Application Regression Testing") @Feature("Login Page Tests") @Listeners(TestExecutionListener.class) public class LoginTests extends BaseTest { LoginPage objLogin; DashboardPage objDashboardPage; @Severity(SeverityLevel.NORMAL) @Test(priority = 0, description = "Verify Login Page") @Description("Test Description : Verify the title of Login Page") @Story("Title of Login Page") public void verifyLoginPage() { // Create Login Page object objLogin = new LoginPage(driver); // Verify login page text objLogin.verifyPageTitle(); } /* Failed Test */ @Severity(SeverityLevel.BLOCKER) @Test(priority = 1, description = "Login with invalid username and password") @Description("Test Description : Login Test with invalid credentials") @Story("Unsuccessful Login to Application") public void invalidCredentialTest() { // Create Login Page object objLogin = new LoginPage(driver); objLogin.login("test", "test123"); // Verify login page text objLogin.verifyErrorMessage(); } }

TestExecutionListener

package AllureDemo.tests; import io.qameta.allure.Attachment; import org.openqa.selenium.OutputType; import org.openqa.selenium.TakesScreenshot; import org.openqa.selenium.WebDriver; import org.testng.ITestListener; import org.testng.ITestResult; public class TestExecutionListener implements ITestListener { @Attachment(value = "Screenshot of {0}", type = "image/png") public byte[] saveScreenshot(String name, WebDriver driver) { return (byte[]) ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES); } @Override public void onTestFailure(ITestResult result) { saveScreenshot(result.getName(), BaseTest.driver); } }

TestNG XML

TestNG in Allure Report Tutorial

Test Execution

In order to execute your tests, you have to open command prompt, go to the project directory, and write the below command.

allure serve allure-results

Allure in Cucumber BDD

You can also add Allure to your Cucumber BDD framework by adding the below dependency in the POM.xml file.

<!-- https://mvnrepository.com/artifact/io.qameta.allure/allure-cucumber-jvm --> <dependency> <groupId>io.qameta.allure</groupId> <artifactId>allure-cucumber6-jvm</artifactId> <version>2.14.0</version> </dependency> </dependencies>

Add the following plugin to your Runner cukes file by referring to the below screenshot

"io.qameta.allure.cucumber6jvm.AllureCucumber6Jvm"

Allure in Cucumber BDD

Sample Allure Report

Before we proceed to see the sample report in our Allure report tutorial, let’s first explore the 5 most important or commonly used annotations used in Allure. You can expand the capabilities of an Allure report by using more annotations, but these 5 will help you generate your basic first report.

Allure Report Overview Categories in Allure Report Suites in Allure Report Graphs in Allure Report Behaviors in Allure Report Timeline in Allure Report Packages in Allure Report

Important Annotations used in the Allure report

@Step – This annotation is used as a step definition to define any modifier with the parameter. @Epic – This annotation is used to define the large component or a whole product under which the Allure depends on. @Feature – We can use it to represent all the possible test coverage options based on the Test scenarios. @Stories – Every Epic has many stories and each of those stories represent a sub-product or sub-component. @Attachment – At the end of our test execution, we will have to add screenshots to our reports. And this annotation can be used to define the position of the screenshot.

Allure Report Overview

Allure Report Overview

As you can see in the above screenshot, the Allure report overview has different sections such as suites, environment, categories, and so on. Let’s find out what they are in our Allure report tutorial.

Suites

A suite is a group of tests that share a common context such as a specific test environment or a particular test category. Suites are represented by colored blocks in the report’s timeline, and you can drill down into the details of each suite by clicking on them.

Environment:

The environment section contains information about the operating system, browser, framework, and version of the tools used.

Features By Stories:

In order to group test cases by features or user stories in an Allure report, you can use the @Feature or @Story annotations in your test code.

Categories:

Once this annotation is added to the test, the test cases will be grouped under the “smoke” and “regression” categories when your Allure report is generated. It is also possible to use the allure.category property to add the category information to the test case.

Executors:

Allure also supports custom executors that can be used when your current executors do not match the requirements of your project. Kindly note that they have to be implemented as a separate component.

Graphs:

Graphical representation of the status, severity, duration, and duration trend will also be available in an Allure report’s overview.

Timeline:

This section in an Allure report’s overview provides a graphical representation of the test execution. It will show the duration of each test case, and also specifies the order in which they were executed.

Categories in Allure Report

Categories In Allure Report

Next up in our Allure report tutorial, we’re going to explore the Categories tab that gives you the way to create custom defects classification for your test results.

By default, there will be two categories of defects in an Allure report:

Product defects (Tests that failed due to an issue in the product) Test defects (Broken tests)

But you also have the option to create custom defect classifications such as skipped tests, outdated tests, and so on. You can do so by adding a categories.json file to the allure-results directory before report generation.

categories.json
[ { "name": "Ignored tests", "matchedStatuses": ["skipped"] }, { "name": "Infrastructure problems", "matchedStatuses": ["broken", "failed"], "messageRegex": ".*bye-bye.*" }, { "name": "Outdated tests", "matchedStatuses": ["broken"], "traceRegex": ".*FileNotFoundException.*" }, { "name": "Product defects", "matchedStatuses": ["failed"] }, { "name": "Test defects", "matchedStatuses": ["broken"] } ]

(mandatory) category name

(optional) list of suitable test statuses. Default [“failed”, “broken”, “passed”, “skipped”, “unknown”]

(optional) regex pattern to check test error message. Default “.*”

(optional) regex pattern to check stack trace. Default “.*”

Test result falls into the category if their status is in the list and both the error message and stack trace match the pattern.

Suites in Allure Report

Suites In Allure Report

As you can see in the above screenshot, the Suites tab shows a standard structural representation of the executed tests, grouped by the suites and classes that can be found.

Graphs in Allure Report

Graphs In Allure Report

Graphs allow you to see different statistics collected from the test data such as status breakdown, severity, and duration diagrams. They will be very effective in helping us understand the status of our test execution easily.

Behaviors in Allure Report

Behaviors In Allure Report

Behaviors group the test results according to Epic, Feature, and Story tags. It also consists of the status of each and every execution and has a detailed description as well.

Timeline in Allure Report

Timeline In Allure Report

Allure adaptors will collect the accurate timings of the tests and arrange them in a sequential or parallel structure based on the timing. This visualization of the test execution will be available in the Timeline section of our Allure report.

Packages in Allure Report

Packages In Allure Report

The packages tab represents a tree-like layout of the test results, grouped by different packages as shown in the above image.

Advantages of Allure Report

In general, effective reporting can help shorten the development/QA lifecycle by providing both the developers and testers with all the information they will need. As seen above in our Allure report tutorial, it enables us to categorize the test failures into bugs, broken tests, and logs. Other vital information such as steps, fixtures, attachments, timings, history, and integrations with TMSs are also available to configure bug-tracking systems.

Managers and stakeholders will have a clear understanding of the project’s big picture, progress, timeline, defects clustering info, and so on. Now let’s take a look at the technical advantages that Allure reports have to offer.

Allure is an open-source report engine built using AngularJS. It supports various frameworks such as JUnit, TestNG, Cucumber-Java, Selenide, Pytest, behav, Jasmine, Mocha, RSpec, Spock, PHPUnit, SpecFlow, NUnit, MSTest, etc. You can create visually rich reports that can be easily understood even by non-technical team members such as Stakeholders and Business Analysts. It has a command line interface to generate reports based on maven surefire plugins. It supports popular CI/CD platforms like Jenkins, TeamCity, Banboo, Gradle, Maven, and CodeFresh. Modularity and extensibility of Allure guarantee that you will always be able to fine-tune it to make Allure suit your needs better.
Related Blogs

TestNG Tutorial : An Introduction to its Attributes and Features

How to use Selenium WebDriver Event Listener?

Conclusion

As a lightweight, flexible, multilingual test report tool, Allure Framework provides a neat web report that not only illustrates what has been tested in a concise manner. It also gives everyone involved in developing the maximum amount of useful information from testing every day. We hope that our Allure Report tutorial has given you a clear understanding of how to generate Allure reports for your project. As a leading automation testing company, we have also written a blog highlighting the key differences between Allure report and Extent report. So make sure to subscribe to our newsletter to not miss out on any of our latest blogs and updates.

The post A Conclusive Allure Report Tutorial with Sample Report appeared first on Codoid.

View Entire Post

Read Entire Article
  1. Homepage
  2. Automation Testing
  3. A Conclusive Allure Report Tutorial with Sample Report

Related

More News From Codoid Blog

Trending

1. John Fetterman
2. Karen Read
3. Florida Panthers
4. Georgia baseball
5. Malawi
6. Tony Evans
7. Rebecca Grossman
8. iOS 18 beta
9. Maren Morris
10. Nvidia stock price

Popular

Woman Injured After Attempted Robbery at M Resort Casino Parking Lot in Henderson, NV. Security Failure?
3 weeks ago

You Are Magical
3 weeks ago

Boost your co-working space with social hours
3 weeks ago

Self-Build Construction Loan Options: The Essential Guide
2 weeks ago

WELCOME
just now

2024 Mister O1 - Lake Mary, FL
2 weeks ago
© Obiaks 2024. All rights are reserved