Allure test report complete learning notes

catalogue

brief introduction
Install Allure
Structure of Allure test report
Java TestNG integration Allure Report
Python Pytest integrates Allure Report

brief introduction

If you want to make the test report more beautiful, you will find the allure test report in the search engine. Allure officially provides an online website. You can see it by visiting the following links:

https://demo.qameta.io/allure/

You can switch to Chinese report in the lower left corner:

Which boss doesn't like such a good test report?

Install Allure

1. At GitHub releases:

https://github.com/allure-framework/allure2/releases

Or Maven Central:

https://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/

Download the installation package and unzip it.

2. Find the bin directory, and Windows uses allure Bat, Unix uses allure:

3. For the convenience of subsequent use, you can add the full PATH of bin to the system environment variable PATH, and then directly enter the allocate command on the command line.

The official also gives the installation methods of ppa for Linux, brew for Mac and scoop for Windows. However, for our people, it is better to download the package manually.

After installation, you can check whether the installation is successful:

$ allure --version
2.17.2

Because Allure is written in Java, if it cannot run, check whether Java is installed.

Structure of Allure test report

Overview

Suites

Graphs

Timeline

Behaviors

Packages

Java TestNG integration Allure Report

integrate

In Maven's POM Add to XML:

<properties>
    <aspectj.version>1.8.10</aspectj.version>
</properties>
<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-testng</artifactId>
        <version>LAST_VERSION</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.20</version>
            <configuration>
                <argLine>
                    -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                </argLine>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

Then execute the command:

$ mvn clean test

An allure test report will be generated in the target / allure results directory, and then execute the command:

$ allure serve target/allure-results

Allure will create a Jetty server, pull up the default browser and open the test report.

@Description

Add Test Description:

package my.company.tests;
import org.junit.Test;
import io.qameta.allure.Description;
@Test
public class MyTests {
    @Test
    @Description("Some detailed test description")
    public void testSomething() throws Exception {
        ...
    }
}

@Step

Add the test Step description. In addition to the text, Allure2 can obtain parameters in @ Step through the reflection mechanism, such as:

public class User {
     private String name;
     private String password;
     ...
}

import io.qameta.allure.Step;
...
@Step("Type {user.name} / {user.password}.")
public void loginWith(User user) {
     ...
}

@Attachment

Add the attachment of the test report, which can be of type String or byte []. If it is not explicit, Allure will call toString() to try implicit conversion:

import io.qameta.allure.Attachment;
...
@Attachment
public String performedActions(ActionSequence actionSequence) {
    return actionSequence.toString();
}
@Attachment(value = "Page screenshot", type = "image/png")
public byte[] saveScreenshot(byte[] screenShot) {
    return screenShot;
}

Type is used to specify the MIME type. It is not required. Allure will automatically adapt according to the content.

In addition to comments, you can also add:

import io.qameta.allure.Allure;
...
Allure.addAttachment("My attachment", "My attachment content");
Path content = Paths.get("path-to-my-attachment-contnet");
try (InputStream is = Files.newInputStream(content)) {
    Allure.addAttachment("My attachment", is);
}

Links

Add hyperlinks with 3 annotations @ Link, @ Issue, @ TmsLink (test management system):

import io.qameta.allure.Link;
import io.qameta.allure.Issue;
import io.qameta.allure.TmsLink;
@Link("https://example.org")
@Link(name = "allure", type = "mylink")
public void testSomething() {
     ...
}
@Issue("123")
@Issue("432")
public void testSomething() {
     ...
}
@TmsLink("test-1")
@TmsLink("test-2")
public void testSomething() {
     ...
}

The link type can be defined in properties, and Allure will replace {} with the value in the annotation:

allure.link.mylink.pattern=https://example.org/mylink/{}
allure.link.issue.pattern=https://example.org/issue/{}
allure.link.tms.pattern=https://example.org/tms/{}

@Severity

Add severity identifier:

package org.example.tests;
import org.junit.Test;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;
public class MyTest {
    @Test
    @Severity(SeverityLevel.CRITICAL)
    public void testSomething() throws Exception {
        // ...
    }
}

Agile logo

In agile development, there are three concepts: Epic, Feature and Stories, which are used for hierarchical management of requirements. Similarly, they can be applied to hierarchical management of testing:

package org.example.tests;
import org.junit.Test;
import io.qameta.allure.Epic;
import io.qameta.allure.Feature;
import io.qameta.allure.Story;
@Epic("Allure examples")
@Feature("Junit 4 support")
public class MyTest {
    @Test
    @Story("Base support for bdd annotations")
    @Story("Advanced support for bdd annotations")
    public void testSomething() throws Exception {
        // ...
    }
}

Python Pytest integrates Allure Report

integrate

Execute the following command to install:

$ pip install allure-pytest

It will install both allure pytest and allure Python commons packages.

Then specify the Allure test report Directory:

$ pytest --alluredir=/tmp/my_allure_results

Finally, execute the command to open the report:

$ allure serve /tmp/my_allure_results

Allure will create a Jetty server, pull up the default browser and open the test report.

pytest result status

Allure will mark according to the result status of pytest:

import pytest
def test_success():
    """this test succeeds"""
    assert True
def test_failure():
    """this test fails"""
    assert False
def test_skip():
    """this test is skipped"""
    pytest.skip('for a reason!')
def test_broken():
    raise Exception('oops')

pytest property

xfail

@pytest.mark.xfail(condition=lambda: True, reason='this test is expecting failure')
def test_xfail_expected_failure():
    """this test is an xfail that will be marked as expected failure"""
    assert False
@pytest.mark.xfail(condition=lambda: True, reason='this test is expecting failure')
def test_xfail_unexpected_pass():
    """this test is an xfail that will be marked as unexpected success"""
    assert True

Will show in the Allure report:

skipif

@pytest.mark.skipif('2 + 2 != 5', reason='This test is skipped by a triggered condition in @pytest.mark.skipif')
def test_skip_by_triggered_condition():
    pass

Will be displayed in the Allure report:

fixtures

@pytest.fixture(params=[True, False], ids=['param_true', 'param_false'])
def function_scope_fixture_with_finalizer(request):
    if request.param:
        print('True')
    else:
        print('False')
    def function_scope_finalizer():
        function_scope_step()
    request.addfinalizer(function_scope_finalizer)
@pytest.fixture(scope='class')
def class_scope_fixture_with_finalizer(request):
    def class_finalizer_fixture():
        class_scope_step()
    request.addfinalizer(class_finalizer_fixture)
@pytest.fixture(scope='module')
def module_scope_fixture_with_finalizer(request):
    def module_finalizer_fixture():
        module_scope_step()
    request.addfinalizer(module_finalizer_fixture)
@pytest.fixture(scope='session')
def session_scope_fixture_with_finalizer(request):
    def session_finalizer_fixture():
        session_scope_step()
    request.addfinalizer(session_finalizer_fixture)
class TestClass(object):
    def test_with_scoped_finalizers(self,
                                    function_scope_fixture_with_finalizer,
                                    class_scope_fixture_with_finalizer,
                                    module_scope_fixture_with_finalizer,
                                    session_scope_fixture_with_finalizer):
        step_inside_test_body()

Will be displayed in the Allure report (fixtures will be displayed in Set up and Tear down):

For the result status in the fixture, Allure will also mark:

import pytest
@pytest.fixture
def skip_fixture():
    pytest.skip()
@pytest.fixture
def fail_fixture():
    assert False
@pytest.fixture
def broken_fixture():
    raise Exception("Sorry, it's broken.")
def test_with_pytest_skip_in_the_fixture(skip_fixture):
    pass
def test_with_failure_in_the_fixture(fail_fixture):
    pass
def test_with_broken_fixture(broken_fixture):
    pass

parametrize

import allure
import pytest
@allure.step
def simple_step(step_param1, step_param2 = None):
    pass
@pytest.mark.parametrize('param1', [True, False], ids=['id explaining value 1', 'id explaining value 2'])
def test_parameterize_with_id(param1):
    simple_step(param1)
@pytest.mark.parametrize('param1', [True, False])
@pytest.mark.parametrize('param2', ['value 1', 'value 2'])
def test_parametrize_with_two_parameters(param1, param2):
    simple_step(param1, param2)
@pytest.mark.parametrize('param1', [True], ids=['boolean parameter id'])
@pytest.mark.parametrize('param2', ['value 1', 'value 2'])
@pytest.mark.parametrize('param3', [1])
def test_parameterize_with_uneven_value_sets(param1, param2, param3):
    simple_step(param1, param3)
    simple_step(param2)

Allure will display each test and id:

And operation details:

Allure properties

@allure.step

import allure
import pytest
from .steps import imported_step
@allure.step
def passing_step():
    pass
@allure.step
def step_with_nested_steps():
    nested_step()
@allure.step
def nested_step():
    nested_step_with_arguments(1, 'abc')
@allure.step
def nested_step_with_arguments(arg1, arg2):
    pass
def test_with_imported_step():
    passing_step()
    imported_step()
def test_with_nested_steps():
    passing_step()
    step_with_nested_steps()

step can read the value from the input parameter:

import allure
@allure.step('Step with placeholders in the title, positional: "{0}", keyword: "{key}"')
def step_with_title_placeholders(arg1, key=None):
    pass
def test_steps_with_placeholders():
    step_with_title_placeholders(1, key='something')
    step_with_title_placeholders(2)
    step_with_title_placeholders(3, 'anything')

step can also be in conf test Py is used on fixtures:

import allure
import pytest
@allure.step('step in conftest.py')
def conftest_step():
    pass
@pytest.fixture
def fixture_with_conftest_step():
    conftest_step()
import allure
from .steps import imported_step
@allure.step
def passing_step():
    pass
def test_with_step_in_fixture_from_conftest(fixture_with_conftest_step):
    passing_step()

allure.attach

Add an attachment to the Allure test report, Allure attach(body, name, attachment_type, extension):

body file content (or source specifies the file path)
Name file name
attachment_ Type attachment type (a value in allure.attachment_type)
Extension file extension

import allure
import pytest
@pytest.fixture
def attach_file_in_module_scope_fixture_with_finalizer(request):
    allure.attach('A text attacment in module scope fixture', 'blah blah blah', allure.attachment_type.TEXT)
    def finalizer_module_scope_fixture():
        allure.attach('A text attacment in module scope finalizer', 'blah blah blah blah',
                      allure.attachment_type.TEXT)
    request.addfinalizer(finalizer_module_scope_fixture)
def test_with_attacments_in_fixture_and_finalizer(attach_file_in_module_scope_finalizer):
    pass
def test_multiple_attachments():
    allure.attach.file('./data/totally_open_source_kitten.png', attachment_type=allure.attachment_type.PNG)
    allure.attach('<head></head><body> a page </body>', 'Attach with HTML type', allure.attachment_type.HTML)

Descriptions

You can use the decorator @ allure Description or @ allure description_ HTML add description:

import allure
@allure.description_html("""
<h1>Test with some complicated html description</h1>
<table style="width:100%">
  <tr>
    <th>Firstname</th>
    <th>Lastname</th>
    <th>Age</th>
  </tr>
  <tr align="center">
    <td>William</td>
    <td>Smith</td>
    <td>50</td>
  </tr>
  <tr align="center">
    <td>Vasya</td>
    <td>Jackson</td>
    <td>94</td>
  </tr>
</table>
""")
def test_html_description():
    assert True
@allure.description("""
Multiline test description.
That comes from the allure.description decorator.
Nothing special about it.
""")
def test_description_from_decorator():
    assert 42 == int(6 * 7)
def test_unicode_in_docstring_description():
    """Unicode in description.
    Этот тест проверяет юникод.
    Hello, man.
    """
    assert 42 == int(6 * 7)

You can also use allure. XML in your code dynamic. Description dynamically add description:

import allure
@allure.description("""
This description will be replaced at the end of the test.
""")
def test_dynamic_description():
    assert 42 == int(6 * 7)
    allure.dynamic.description('A final description.')

Titles

You can use the decorator @ allure Title add Title:

import allure
import pytest
@allure.title("This test has a custom title")
def test_with_a_title():
    assert 2 + 2 == 4
@allure.title("This test has a custom title with unicode: Привет!")
def test_with_unicode_title():
    assert 3 + 3 == 6

Parameter values can be obtained:

@allure.title("Parameterized test title: adding {param1} with {param2}")
@pytest.mark.parametrize('param1,param2,expected', [
    (2, 2, 4),
    (1, 2, 5)
])
def test_with_parameterized_title(param1, param2, expected):
    assert param1 + param2 == expected

You can also use allure. XML in your code dynamic. Title Dynamic addition:

@allure.title("This title will be replaced in a test body")
def test_with_dynamic_title():
    assert 2 + 2 == 4
    allure.dynamic.title('After a successful test finish, the title was replaced with this line.')

Links

@allure.link, @allure.issue and @ allure testcase:

import allure
TEST_CASE_LINK = 'https://github.com/qameta/allure-integrations/issues/8#issuecomment-268313637'
@allure.link('https://www.youtube.com/watch?v=4YYzUTYZRMU')
def test_with_link():
    pass
@allure.link('https://www.youtube.com/watch?v=Su5p2TqZxKU', name='Click me')
def test_with_named_link():
    pass
@allure.issue('140', 'Pytest-flaky test retries shows like test steps')
def test_with_issue_link():
    pass
@allure.testcase(TEST_CASE_LINK, 'Test case title')
def test_with_testcase_link():
    pass

@allure.link provides clickable hyperlinks.

@allure.issue will have a clickable icon.

Issue link template is defined through -- allocate link pattern:

$ pytest directory_with_tests/ --alluredir=/tmp/my_allure_report \
 --allure-link-pattern=issue:http://www.mytesttracker.com/issue/{}

Retries

Allure has a Retries tab to show the retried tests:

import allure
import random
import time
@allure.step
def passing_step():
    pass
@allure.step
def flaky_broken_step():
    if random.randint(1, 5) != 1:
        raise Exception('Broken!')
def test_broken_with_randomized_time():
    passing_step()
    time.sleep(random.randint(1, 3))
    flaky_broken_step()

Agile logo

import allure
def test_without_any_annotations_that_wont_be_executed():
    pass
@allure.story('epic_1')
def test_with_epic_1():
    pass
@allure.story('story_1')
def test_with_story_1():
    pass
@allure.story('story_2')
def test_with_story_2():
    pass
@allure.feature('feature_2')
@allure.story('story_2')
def test_with_story_2_and_feature_2():
    pass

You can also specify to run tests using -- allure epics, -- allure features, -- allure Stories:

$ pytest tests.py --allure-stories story_1,story_2
collected 5 items
tests.py ...                                                                    [100%]
============================== 3 passed in 0.01 seconds ==============================
$ pytest tests.py --allure-features feature2 --allure-stories story2
collected 5 items
tests.py ...                                                                     [100%]
=============================== 2 passed in 0.01 seconds ==============================

Severity

Use @ allure Severity or allure severity_ level:

import allure
def test_with_no_severity_label():
    pass
@allure.severity(allure.severity_level.TRIVIAL)
def test_with_trivial_severity():
    pass
@allure.severity(allure.severity_level.NORMAL)
def test_with_normal_severity():
    pass
@allure.severity(allure.severity_level.NORMAL)
class TestClassWithNormalSeverity(object):
    def test_inside_the_normal_severity_test_class(self):
        pass
    @allure.severity(allure.severity_level.CRITICAL)
    def test_inside_the_normal_severity_test_class_with_overriding_critical_severity(self):
        pass

--Allow severities is used to specify which severity tests to run:

$ pytest tests.py --allure-severities normal,critical
collected 5 items
bdd_annotations_demo/test_severity_labels.py ...                                [100%]
================================ 3 passed in 0.01 seconds ============================

reference material:

https://docs.qameta.io/allure-report/

Added by tomdumont on Tue, 25 Jan 2022 03:33:42 +0200

Programming VIP

Allure test report complete learning notes

catalogue

brief introduction

Install Allure

Structure of Allure test report

Overview

Categories

Suites

Graphs

Timeline

Behaviors

Packages

Java TestNG integration Allure Report

integrate

@Description

@Step

@Attachment

Links

@Severity

Agile logo

Python Pytest integrates Allure Report

integrate

pytest result status

pytest property

xfail

skipif

fixtures

parametrize

Allure properties

@allure.step

allure.attach

Descriptions

Titles

Links

Retries

Agile logo

Severity

Popular Keywords