Imagine you’re driving a car, the engine is revving, the wheels are spinning and the wind running through hair. Everything may seem fine until you hit the brakes and your car stops only after 10 seconds. Conducting thorough testing can help detect such deficiencies. In this case non-functional testing will help you test the reaction time of the braking system, how well it works, what is its speed, its performance and so on.

Non-functional testing

Non-functional testing is a key aspect of software development that is often overlooked. While functional testing ensures that the software contains the required functional features, non-functional testing focuses on aspects such as performance, security, usability, compatibility and scalability. Ignoring non-functional testing can pose significant risks including reduced system performance, security vulnerabilities, poor user experience and limited scalability.

Non functional testing vs functional testing

Functional testing checks the processes of an application against a set of requirements or specifications. Non-functional testing evaluates application features that are not critical to functionality but contribute to the end-user experience, such as system performance and reliability under load.

Simply put, non-functional testing checks the quality of a product rather than its features. In the world of software development, non-functional testing is as necessary as functional testing. Without it, the system may work perfectly fine in a controlled environment, but may fail in the real world.

When do you need non functional testing?

Non-functional testing is performed after unit testing, integration testing and functional testing have been performed. When the testers find that the software works as expected, they perform non-functional testing to assess whether it also meets the non-functional parameters. This is one of the last phases of software testing before user acceptance testing and the final release of the product into production.

Objectives of Non functional testing

Non-functional software testing is aimed at achieving these goals:

• Improve installation, configuration, performance and product management
• Collect data and metrics on performance, usability, security and other non-functional aspects of the system to identify areas for improvement and optimization
• Improving product usability, efficiency, maintainability and portability
• Reduction of manufacturing risk and costs associated with non-functional product features
• Improved knowledge of current technology and product behaviour

Benefits of Non functional testing

• Provides a higher degree of security. Security is a key element that protects systems from online threats
• Guarantees system load capacity so users can use it simultaneously
• Increases system efficiency
• No need to write test cases multiple times because they never change
• Compared to other testing procedures, the overall time requirement is less

Characteristics of non functional testing

1. Non-functional testing should be measurable, so there is no place for subjective characterizations like good, better, best, etc.
2. Exact figures are unlikely to be known at the start of the requirements process
3. It is important to prioritise requirements
4. Ensure correct identification of quality attributes within software engineering

Non functional testing – parameters

1) Security: this parameter defines how the system is secured against intentional and accidental attacks from internal and external sources. This is tested through safety testing.

2) Reliability: the degree to which any software system continuously performs specified functions without failure. It is tested through reliability testing.

3) Recoverability: this parameter checks whether the software system continues to function and whether it will recover in the event of a system failure. This is checked by recovery testing.

4) Availability: this parameter determines the degree to which the user can rely on the system during its operation. This parameter is checked by stability testing.

5) Usability: user can easily learn, work, prepare inputs and outputs through interaction with the system. This is checked by usability testing.

6) Scalability: this term refers to the degree to which any software application can expand its processing capacity to meet an increase in demand. This is verified by scalability testing.

7) Interoperability: this non-functional parameter controls the interfaces of the software system with other software systems. This is checked by interoperability testing.

8) Efficiency: the extent to which any software system can handle the capacity, quantity and response time.

9) Flexibility: this term refers to the ease with which an application can work in different hardware and software configurations. Such as minimum requirements for RAM, processor.

10) Portability: the flexibility of the software when ported from its current hardware or software environment.

11) Reusability: in this case, it is a part of the software system that can be converted for use in another application.

Non-functional testing types

1. Performance tests

Performance tests ensure several elements of software quality. They verify that it is fast, scalable, stable and reliable. These tests can find problems in software design and architecture performance.

Typically performed: measuring response time, identifying sensitive points, locating points of failure

2. Load tests

Load tests verify how the software behaves under both normal and boundary conditions, how much workload the software can handle before its performance is adversely affected. You can perform load tests by running multiple applications at the same time, subjecting the server to high traffic or downloading a large number of files.

Load tests are used to ensure fast and scalable software.

3. Stress tests

Stress tests verify how the software behaves under unusual conditions. This sets the threshold at which the software breaks. It is important to find out what happens when the system is under load. Is the correct error message displayed? Will the system fail? How will it recover?

Stress tests help testers analyse what happens when a system fails. This will ensure that the software is recoverable, stable and reliable.

4. Volume tests

Volume tests are used to verify what happens to system performance when a huge volume of data is added to the database. This is done in order to see what problems may occur as data volumes grow. It is also known as flood testing.

Volume tests verify that systems respond to certain volumes of data as expected. This is important to ensure performance and stability.

5. Security tests

Security tests check the software for errors or vulnerabilities that may compromise data. The goal of security testing is to identify any potential security risks or threats and ensure that the product is not vulnerable to hacking attacks, data leaks or other types of security issues.

Common security tests include: vulnerability scans, security scans, penetration testing, risk assessments, security audits, ethical hacking

6. Upgrade and Installation tests

Update and installation testing verifies that the software will work correctly on all computers. Thus, testing of the update is done for existing users. And installation testing is done for new users.

7. Recovery tests

Recovery tests determine how quickly the software can recover from a crash or failure. It is done by making the system fail on purpose.

This type of testing is done to see what happens to the software:

• If you disconnect the hardware.
• If you disconnect from the network during data transfer.
• When you reboot unexpectedly.

Non functional testing process

The process of non-functional testing usually consists of several key phases. Although the specific terminology may vary, the main steps involved in the non-functional testing process typically include:

• Requirements gathering (non functional testing requirements)

In the first phase of the testing process, the QA team works with stakeholders, architects, and developers to understand and document non-functional system requirements. It includes identification of performance expectations, security criteria, usability guidelines, scalability needs and other relevant non-functional attributes.

• Planning

Based on the gathered requirements , the test team creates a comprehensive test plan that outlines the non-functional testing objectives, scope, test scenarios, test environment setup and resource requirements. The plan also defines the testing approach, testing techniques and tools.

• Test case design

In this phase, the QA team designs non-functional test cases and test scripts based on the identified test scenarios. Test design involves determining the test data, selecting appropriate non-functional testing techniques, and outlining the steps to execute the tests.

• Execution

Non-functional tests are performed according to the test plan and test design. This phase includes setting up the test environment, configuring the necessary tools and running the tests. Testers monitor and measure performance, security, usability and other non-functional aspects of the system.

• Analysis and reporting

After the tests are executed, the QA team analyzes the test results and compares them with the defined non-functional requirements. Any deviations, problems or errors shall be identified, documented and reported. Test reports will be prepared.

• Bug report and re-testing

If non-functional faults are identified in the analysis phase, they shall be recorded, assigned and managed in the issue tracking system. The development team addresses these bugs and once they are resolved, the affected non-functional tests are re-run to verify the fixes and ensure that the system now meets the required non-functional criteria.

• Test closure

In the final phase, the test team will prepare a test completion report summarizing the overall non-functional testing activities, results and lessons learned. Documentation is updated and final outputs, including test artefacts and reports are archived for future reference.

Non functional testing tools

Load testing tools

Tools such as Apache JMeter, Gatling and LoadRunner help simulate high user loads and measure system performance under heavy traffic. They generate virtual user requests, monitor response times and analyze system behavior to identify performance bottlenecks.

Security testing tools

Security tools such as OWASP ZAP, Burp Suite and Nessus help in identifying vulnerabilities and weaknesses in system security. They perform vulnerability scanning, penetration testing and code analysis to assess the system’s resilience to potential security threats.

Performance monitoring tools

Performance monitoring tools provide a real-time view of system performance. Capabilities such as New Relic, AppDynamics, and Dynatrace monitor key performance metrics, track resource utilization and help identify performance issues in production environments.

Usability testing tools

Usability testing tools including UserTesting, Morae and Optimal Workshop help evaluate the user experience and usability of the system. These tools allow you to create usability tests, screen recordings and collect user feedback to assess the ease of use of the system and user satisfaction.

Test management tools

JIRA, TestRail, Asana and qTest tools offer options for managing and tracking non-functional testing activities. They help with test planning, test case management, defect tracking and reporting, facilitating effective test management throughout the non-functional testing process.

Compliance testing tools

Compliance testing tools such as OWASP Dependency-Check, SonarQube and Veracode help ensure compliance with industry standards, regulations and coding best practices. They analyze code, dependencies and configurations to identify potential compliance issues and provide recommendations for improvement.

Risks associated with non functional testing

Risk no. 1: Performance bottlenecks

Imagine a popular e-commerce website that failed to perform non-functional testing. As a result, during the Black Friday sale, the website crashes due to a sudden rise of users. This not only damages the company’s reputation, but also results in a loss of revenue.

Risk no. 2: Security vulnerabilities

Neglecting non-functional testing opens the door to vulnerabilities that hackers can exploit. Take your banking app, without proper encryption or authentication measures, sensitive customer data becomes an easy target for cybercriminals.

Risk no. 3: Poor user experience

User experience (UX) plays a key role in the success of any software application. Imagine a mobile app that takes forever to load, or a website with a cluttered layout. Users will quickly leave such apps and look for alternatives that offer a smoother and more intuitive experience.

Risk no. 4: Compatibility issues

In today’s multi-platform world, software needs to be compatible with different operating systems, browsers and devices. Neglecting non-functional testing in this aspect can lead to compatibility issues. For example, a video streaming app that only works on certain devices, or a web app that crashes on certain browsers.

Risk no. 5: Scalability issues

As the business grows, so does its user base. Neglecting non-functional scalability testing can result in a nightmare scenario. Consider a cloud software service that is experiencing rapid expansion without adequate scalability testing. The system suddenly becomes overloaded, leading to frequent crashes and outages.

Failure cases due to non functional testing – examples of failure

Disruptions in Amazon Prime Day:

In both 2018 and 2019, Amazon’s Prime Day event experienced technical issues. Customers have experienced website slowdowns, broken links and difficulty completing purchases. These incidents highlighted the criticality of dysfunctional testing, including performance testing and capacity planning, to handle the increase in traffic and ensure a seamless shopping experience during high-demand events.

Facebook’s scandal with Cambridge Analytica:

In 2018, Facebook faced a major controversy when it announced that the personal data of millions of users had been collected without consent by political consultancy Cambridge Analytica. This incident has shed light on the importance of non-functional testing in the context of data privacy and security.

Realisation of benefits

Netflix invests heavily in non-functional testing to ensure a seamless streaming experience. Through performance, scalability and load testing, Netflix is able to handle huge user traffic and deliver content without interruption.

As a tech giant, Google understands the importance of non-functional testing in delivering a reliable and efficient user experience. Their search engine and various applications undergo rigorous testing to ensure optimal performance, security and compatibility. This has made Google one of the most popular search engines.

Non functional testing best practices

To ensure effective non-functional testing, several best practices should be considered:

• Involve testing in the early stages of the SDLC(STLC). Work with stakeholders, architects and developers to understand non-functional requirements and design the system with those requirements in mind.
• Define clear and measurable objectives for non-functional testing. Establish specific targets for performance, safety, usability and other non-functional aspects of the system.
• Create a realistic test environment that closely simulates the production environment.
• Use test automation tools and frameworks. Automation helps simulate user workloads, generate consistent test data, and perform repetitive tasks, enabling more efficient testing process.
• During testing, implement reliable monitoring mechanisms to capture performance metrics such as response times, resource utilization, throughput, and error rates. These metrics provide insight into system behavior, help identify bottlenecks, and facilitate performance analysis.
• Focus on critical functions, high-risk areas, and scenarios that may cause performance degradation, security vulnerabilities, or usability issues.

Non functional tests example

Test Case #	Test Case	Domain
1	Application loading time should be no longer than 5 seconds for 1000 users accessing the application simultaneously	Performance Testing
2	The software should be installable on all versions of Windows and Mac	Compatibility Testing
3	All web images should have an alt tag	Accessibility testing

Conclusion

Non-functional testing is a comprehensive approach that helps ensure that an application is optimized for the user experience and meets the necessary quality standards. It helps developers identify and address potential problems before the app is released. This reduces the risk of failure and ensures that the application meets user expectations. So don’t forget about it, because you’ll be putting your product at unnecessary risk.

If you speak German and are an IT tester, take a look at our employee benefits and apply for our job offers.