In the corporate world, maintaining transparency between the project team and clients is as important as meeting their requirements for the project. Many strategies are being developed to ensure that there is no communication gap between the stakeholders and the company delivering the software. Various measures are taken to maintain good communication. One such measure is the presentation of test artifacts to team members and stakeholders.

What are test artifacts / test deliverables?

Test artifacts are also known as test deliverables. These are documents and scripts created during software testing to ensure that the application under test meets the required quality standards. These documents help to ensure that stakeholders are kept informed of the progress of the project.

There are different types of test artifacts, each serving a specific purpose. Test plans, test cases and test scripts are among the most common test artifacts used in software testing.

Test artifacts list – a list of the most used test outputs

1. Test Strategy

The test strategy is usually prepared by the test or project manager. It is an outline document that describes an approach to software development lifecycle testing that enumerates how to achieve the expected result using the available resources.

It simply provides a straightforward understanding of the objectives, tools, techniques, infrastructure and timing of the testing activities to be performed. It is also used to identify any risk factors that may arise during testing and appropriate solutions to reduce or mitigate the risk.

The test strategy mainly includes components such as:

• the objectives and scope of the testing,
• documentation formats,
• testing processes and techniques,
• team reporting structure,
• client communication strategy.

2. Test Plan

A test plan is often confused with a test strategy. It is a detailed document that covers all aspects of the testing phase, while the testing strategy is only an outline of the whole project.

The test plan describes the scope of software testing, test strategy, test results, risks, objectives and activities. It is a systematic approach that is generally used when testing software applications. It is the most important and essential activity that simply ensures that the baseline plan initially lists tasks and milestones to track or identify the progress of the project.

The test plan includes:

• the scope and objectives of the projects,
• the resources needed to complete the project,
• types of testing and approaches,
• risks,
• entry and exit points/criteria,
• dates/time schedule,
• hardware/software requirements.

3. Test Scenario

A test scenario is used to describe the functionality of an application that can be tested. It is used to make sure that end-to-end testing of a feature or software is working properly. It is derived from use cases.

Test scenario contains a situation or condition in the application from which many test cases can be created. It is also called a test condition or test option. One or more test cases can fit into one test scenario. As a result, the test scenario with test cases has a one-to-many (1:N) relationship. One test scenario includes several test cases. The test cases are developed based on a high-level test scenario.

Consider testing a secure web application that consists of many web pages. “Login Page Verification” can be considered as example of one test scenario. The tester must validate all objects (URL, input, button, actions, links, etc.) and relevant functions within the page.

4. Test Case

Test cases are an extended part of the test scenario that help in test execution. Test Case presents a detailed document that describes the cases that will help with implementation during testing. It is a document that consists of a test case name, a precondition, test steps/conditions of entry and expected results. Developing test cases also helps in identifying and tracking problems or issues in the requirements or design of a software application.

Points to include in the test case:

• Write test case id, that is a unique test case identification number.
• Write the name of the test case.
• Provide test data.
• Write all the details and descriptions related to the test case.
• Write the test procedure in steps to make the test case clear and concise = simple.
• Write the expected test result and leave room for the actual test result.

For example, consider a text field in a web application that can only accept numbers from 1 to 999. Any value that is not in this range, or any alphanumeric value, should be discarded by the system. To properly verify the functionality of the input field, the tester should write two test cases – one with any input value in the range 1 – 999 (positive) and the other with input of any invalid alphanumeric data / any numeric data that is not in the range 1 – 999 (negative).

5. Traceability Matrix

A traceability matrix is a matrix that contains tables that show and explain the relationships between many requirements and test cases. In software testing, requirements traceability is often used to link detailed requirements and test design. The final product is known as a Requirement Traceability Matrix (RTM). The RTM can be used to determine if all current project requirements are verified or covered through the designed test cases.

Some of the parameters that are included in the traceability matrix are listed below:

• request ID,
• the type of requirement, together with a description,
• the status of the test design along with the status of the test execution,
• system and unit test cases.

1. Documentation

Automated documentation tools simplify the creation of comprehensive test activity reports such as requirements traceability matrices (RTMs), progress reports and defect metrics.

1. Defect/Bug report

There are test automation tools that integrate with various bug/defect tracking tools. These tools automatically generate a bug report when a bug is found through automated test cases and test scenarios. Bug reports will be generated documenting the issue including a description of the problem, severity, steps to reproduce and any supporting information. These reports are used to track and prioritize the resolution of errors. The bug report is generated after all test cases have been executed.

It contains:

• title ID,
• environment,
• steps to reproduction,
• expected result,
• the actual result,
• attachments (screenshots, videos, text) of error,
• severity/priority,
• assignee.

6. Software Test Report

The test team usually sends out various messages at the end of each test activity to inform stakeholders or customers of the update of each phase. These test reports are intended to document the results of the testing defined in the test plan.

A software test report is therefore a document that describes all testing activities. It provides detailed information about the status of test cases, test files or test scripts for a given scope. Test reports can be generated daily or can be generated after testing is complete.

7. User Guide

When whole software is developed and ready to be deployed into the production environment, a user manual is finally created. This guide is helpful to end users and provides detailed information about the software and its use.

Reasons to have or not to have test artifacts

Several factors can influence the need for test artifacts:

• Company rules: some industries, such as aerospace, nuclear power or healthcare, are strict about testing and documentation. They may require us to produce detailed reports.
• Project needs: if we’re working on something that needs to be super good and reliable, we may need to create more test materials than usual to make sure everything is covered.
• What stakeholders want: testers, developers, project managers and customers may have different ideas about what they need from test results. Developers may need detailed reports to understand and solve problems, while customers may want a summary of how good and reliable the product is.
• What we are trying to do: the goals of testing play a role in deciding how much information we need from the test results. We need more detailed and extensive testing results to find and fix as many problems as possible before releasing the software into production. However, if our goal is only to meet a basic level of quality, more detailed testing results may not be necessary.
• How risky is the project: if it is a high-risk project, it may be necessary to create multiple test artifacts to thoroughly test the system and ensure that it meets all the required specifications.

There are several reasons why an organization may only need certain test artifacts for a project. Here are a few factors that could influence that decision:

• How complicated is the project: if it’s a relatively simple project that has only a few features, you may not need to create all the test artifacts that would be needed for a large project.
• What resources are available: if an organization is low on resources, creating all the test stuff you would need for very complex testing may not make sense.

Test artifacts in test automation

Let’s take a closer look at the different types of test artifacts and explore how they can benefit from automation:

1. Testing strategy: the testing strategy outlines the overall approach and goals for testing a software product or system. This testing strategy can describe in detail a test automation process that can then be put into practice.
2. Test plan: the test plan provides a detailed plan for test execution, including scope, resources, timing, and entry/exit criteria. Automation tools can assist in creating test plans by automatically filling in relevant information based on project parameters.
3. Test scenario: a test scenario defines a specific situation or event that needs to be tested within an application or system. Under certain conditions, it is recommended that these test scenarios are created and executed using test automation tools. This is where automating these tests could help generate ROI in the long run.
4. Test cases: test cases are detailed instructions specifying inputs, expected outputs, pre- and post-conditions for executing tests of specific functions or components of an application. Similar to test scenarios, there are certain conditions where it is recommended to automate these test cases using test automation tools.
5. Test data: test data includes all the inputs needed to perform the tests effectively. This test data is also useful in automating test cases and test scenarios.
6. Documentation: automated documentation tools simplify the creation of comprehensive test activity reports such as requirements traceability matrixes (RTMs), progress reports and defect metrics.
7. Bug/defect reporting: there are automation testing tools that integrate with various bug/defect tracking tools. These tools automatically generate a bug report when a bug is found through automated test cases and test scenarios.
8. Software testing report: summary reports provide an overview of overall testing progress and key metrics such as test coverage, success/failure rate and defect trends. There are test automation tools that generate these summary reports automatically for the executed tests.

Conclusion

To ensure product quality, it is sometimes necessary to create all the test artifacts.

These test artifacts are necessary for a thorough testing process, but it is also important to prioritize and focus on the most significant and important artifacts.

If you are an IT tester or IT automation tester and you speak German, take a look at our employee benefits and respond to our job offers!