Test automation is an important activity in software development, which is required to quickly execute test cases to check reproducibility, consistency and early detection of bugs.

One commonly used and practical artificial intelligence example is ChatGPT, an LLM language model developed by OpenAI. It has many features and helps to perform tasks efficiently even in test automation. ChatGPT can be integrated into a suite of test automation tools to validate data, streamline test case generation, and simplify complex test scenarios. Read also the article AI in software testing.

LLM definition

Large Language Models (LLMs) represent the most significant breakthrough in the field of artificial intelligence since deep learning. These models learn to recognize language patterns using machine learning concepts and large datasets. This allows them to predict answers in a grammatically correct and meaningful way for humans.

ChatGPT 4 – what is it?

ChatGPT is an artificial intelligence-driven language model that can help in creating test cases and scripts for automated testing. It is a tool that supports but does not replace human testers, thus increasing the speed and efficiency of the testing process.

How ChatGPT works?

ChatGPT works on the principle of machine learning. It’s a kind of artificial intelligence that can learn and improve based on the data it’s trained on. ChatGPT was trained on a set of texts and code that contains various types of information, including factual information, creative content and code.

When ChatGPT receives a question or request, it first analyzes it and tries to understand what is being reuquested. It then uses its knowledge and skills to generate an answer or perform a task.

ChatGPT is still under development, but it can already perform many kinds of tasks. Among its abilities are:

• Text generation: ChatGPT can generate text in a variety of styles and formats, including newspaper articles, books, code, scripts, music tracks, emails, letters, etc.
• Language translation: ChatGPT can translate between different languages.
• Answering questions: ChatGPT can answer questions in an informative way, even if they are open-ended, challenging or strange.
• Creative content generation: ChatGPT can generate different kinds of creative content such as poems, code, scripts, music tracks, emails, letters, etc.

ChatGPT and how to use it to create test cases

1. Define a test scenario:

First, decide what you want to test, such as a user action or a specific feature in the app. For example, if you are testing a search function on a web page, outline the key steps of the function.

2. Choose a framework and programming language:

Selenium supports various languages such as Java, Python and C#. Robot Framework supports Python and Cypress only supports JavaScript. Choose which framework and language is best suited to your experience. We have chosen Selenium as an example.

3. Prepare your development environment:

Install Selenium WebDriver and bindings for the selected programming language. This setting allows you to automate browser interactions in your tests.

4. Enter your data into ChatGPT:

Approach ChatGPT with your test scenario. For example, you can say: “I need to test the search function on a website. Can you suggest some test cases?”

5. Accepting test cases from ChatGPT:

ChatGPT will process your request and provide the appropriate test cases. A typical test case for the search function might involve going to the search page, searching for a blank query, entering a correct query, entering an incorrect query (e.g., various random characters, numbers, strings), clicking the search button, and verifying the results.

6. Transform test cases into Selenium scripts:

Use the ChatGPT suggestions to write a Selenium script in the language of your choice. For example, the Java script for the provided test case would include coding steps to open the browser, perform the search actions and check the results. Here is an example of Selenium Java test script for the second test case generated above:

 

 

 

 

 

 

 

 

 

 

7. Execute the test script:

Run your script to check if it has passed or if it has identified any problems.

Incorporating ChatGPT into your test case generation can streamline your testing workflow. Nevertheless, it is very important to check the cases generated by the AI to make sure that they meet your testing needs accurately and comprehensively.

Test automation examples using ChatGPT

ChatGPT for automated testing can be used in the following scenarios, among many others:

Writing test scripts using ChatGPT

ChatGPT can generate test scripts while independently assuming different scenarios to automate ChatGPT testing.

Its complete functioning was evident when we entered the test query “Create a test script with 2 expected methods to open the school website and download the brochure”. ChatGPT envisaged two methods, one to directly download the brochure via an available button and the other to navigate the site and reach the desired location to download the brochure.

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import java.util.concurrent.TimeUnit;

public class TestovanieStrankySkoly {

    public static void main(String[] args) {
        // Nastavenie cesty k ChromeDriver
        System.setProperty("webdriver.chrome.driver", "cesta_k_chromedriver\\chromedriver.exe");

        // Inicializácia ChromeDrivera
        WebDriver driver = new ChromeDriver();

        // Otvorenie webovej stránky školy
        otvorStrankuSkoly(driver);

        // Stiahnutie brožúry
        stiahniBrozuru(driver);

        // Zatvorenie okna prehliadača
        driver.quit();
    }

    public static void otvorStrankuSkoly(WebDriver driver) {
        // Otvorenie webovej stránky školy
        driver.get("https://webstrankaskoly.sk"); // Nahraďte URL adresu webovej stránky školy

        // Implicitné čakanie na dynamické prvky
        driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

        // Voliteľne: Zväčšiť okno prehliadača
        driver.manage().window().maximize();
    }

    public static void stiahniBrozuru(WebDriver driver) {
        // Predpokladá sa, že odkaz na stiahnutie brožúry sa nachádza pod názvom triedy 'brozura-odkaz'
        WebElement odkazNaBrozuru = driver.findElement(By.className("brozura-odkaz"));

        // Kliknutie na odkaz na stiahnutie brožúry
        odkazNaBrozuru.click();

        // Počkať na dokončenie sťahovania (voliteľné: môžeš potrebovať riešenie stiahnutia podľa nastavení tvojho prehliadača)
        // Napríklad, ak je to priamy odkaz na stiahnutie, možno nebudeš potrebovať ďalšie čakanie alebo riešenie stiahnutia

        // Môžeš tiež overiť sťahovanie kontrolovaním, či sa súbor stiahol do konkrétneho adresára
        // alebo kontrolou prítomnosti stiahnutého súboru v systéme po kliknutí na odkaz
    }
}

Test data generation

ChatGPT test automation includes test data generation. Here we generated test data with the prompt “generate test data of 10 patients in tabular form to study the gut microbiome of patients with comorbid diseases such as diabetes, CVD. Ensure that all relevant parameters are taken into account’.

ChatGPT generated test data with respect to several directly related parameters. It also informed us about random and unrealistic data generation and the importance of taking real data.

Using ChatGPT to change and fix code

For example, suppose you have a function that is supposed to calculate the factorial of a number, but it doesn’t give the correct output. For example, your code might look like this:

However, when you call this function with factorial(5), you get a different output instead of 120 (which is a factorial of 5). Now suppose you’re not sure why this is happening and would like help fixing it.

You can describe the problem and provide a code snippet:

“I have a Python function called factorial that is supposed to calculate the factorial of a number, but it doesn’t give me the correct output for factorial(5). I expect the result to be 120, but it shows something else. Here is the code I use:

def factorial(n):

    if n == 0 or n == 1:

        return 1

    else:

        return n * factorial(n-1)

result = factorial(5)

print(result)  # This should give 120, but it's giving a different output.

Môžeš mi pomôcť zistiť, prečo to nefunguje tak, ako sa očakávam?“

By providing a code snippet and describing the problem, ChatGPT can help identify potential problems, bugs, or improvements in your code. It can give you insights, suggest fixes or ask clarifying questions to help you solve a problem you’ve encountered.

Design of sample test cases

People’s blood groups show multiple alleles, which means that a blood group can have different genotypes. We consider this case as predicting the blood group of the child. The command (prompt) for ChatGPT to automate the tests is “Test cases to test the likelihood of a particular blood type being present in a child. Imagine a realistic scenario in which a blood type can have two genotypes”.

Data formatting

DNA sequences are stored in different formats. Our search was based on the FASTA and GenBank format. The former provides only the DNA sequence, while the latter is associated with more information.

In the ChatGPT test automation for formatting data, the command was: “Write a Python and R program to simultaneously convert DNA and protein sequences from FASTA format to GenBank format.” ChatGPT provides a separate program for each programming language to manipulate data/sequences using specific libraries.

Analysis of test results

Mutations are changes in genetic sequences. Given their huge importance, we are asking ChatGPT for automated testing to compare the frequency of mutations. Here is a query for the same “Analyze the results of two random DNA sequences with different mutation frequencies. Give the frequency of the different types of mutations in it”.

Writing documentation

Although ChatGPT can write code, its greatest strength is generating text and explanations. This can be used successfully when writing test documentation. For example, if you need to write team instructions on unit testing, you can ask ChatGPT to do it for you. Ideally, you should provide the tool with a list of topics that should be included in the documentation. For example, I’ve submitted a query to create documentation for a unit test on the topic of “Testing Login”.

Code review assistance (code reviews)

ChatGPT can help review test scripts and suggest improvements, potential problems or optimization opportunities. Picture 3 illustrates some aspects of how ChatGPT can help during the process of code reviews per question: “How can you help me with the code review process?” Read more about code reviews in an article from our Java developer: Effective code reviews: tips and tricks.

Benefits of Chat GPT 4 for test automation

• Quickly generates test cases,
• generates ideas and premises for queries,
• improves test coverage,
• effective for generalized testing,
• provides standardised test case formats,
• saves time and costs,
• increases productivity,
• integration with automation frameworks,
• ensuring security,
• speeding up code debugging.

Challenges and limitations of using ChatGPT for test automation

Although ChatGPT is seamless for test automation, there are some limitations associated with it. Here are the most important ones:

• ChatGPT encounters a lot of data during training. However, it lacks expertise in any specific area. This leads to a lack of accuracy.
• It can misinterpret queries, which has happened to us a couple of times.
• Assumptions and the ability to simulate scenarios or conditions are limited.
• Inability to infer from ambiguous queries.
• The answers are more general.
• Providing confidential information on experiments and innovations is risky.
• Not being able to actually interact with real experiments or software limits its ability to adapt.
• It is not suitable for complex test scenarios.

Best practices for using ChatGPT in test automation

Here are a few tips to overcome several problems a user may encounter when performing test automation:

• Be specific when entering keywords.
• Use ChatGPT itself to generate ideas.
• Provide manual validation of its query, focusing on each row and step.
• Avoid sharing sensitive information or ideas.
• Be sure to provide context in your questions.
• Provide background information specific to the area in question.
• Integrate ChatGPT with the traditional automation toolkit.
• Carefully consider possible biases from ChatGPT and check the results accordingly.

Disadvantages of ChatGPT for test automation

Although ChatGPT, built on top of ChatGPT 4, has many advantages and uses, there are also a few disadvantages to be aware of.

• It is based on statistical patterns and does not understand the basic meaning of

ChatGPT is built on top of GPT-3.5 (at the time of writing, the current version in the paid version is Chatgpt 4), which is an autoregressive language model. One of the biggest problems with this approach, however, is that it is very heavily dependent on statistical patterns.

This learning model predicts, using statistical models, what the next words should be based on what words have been used before. However, it does not have a basic understanding of the meaning of these words.

This means that it cannot be used as effectively in situations where the user’s questions or statements require an understanding of context that has not been previously explained.

• Gaps in learning

The underlying technology in ChatGPT, the GPT-3.5 language model, is a deep learning language model that has been trained on large datasets of human-generated content. We assume that it also learned code as text, which is why it was able to produce such precise codes.

This means that it can’t respond accurately to things it hasn’t learned before, or it may give incorrect information if its learning hasn’t been updated.

For example, if its last learning phase was on a framework that has updated half of its methods since last learning, then the code it creates will use those deprecated methods. So the user would have to make sure that the final code they are using is up to date.

• Incomplete code

Another problem with creating code with ChatGPT is that you have to deal with partially written code.

So, if you are dependent on ChatGPT based code, you would have to first understand the incomplete code, complete it, or modify it to suit your needs. And as you can imagine, it’s often challenging because there are so many things that could go wrong. Even if you manage to get what you want, the final product probably won’t be as good as if you had written the code yourself from scratch. But on the other hand, sometimes extending code or debugging code can be easier than creating iterative code from scratch.

• Prerequisites

ChatGPT is dependent on assumptions. Software testers are trained to identify these hidden factors that could potentially cause an application to fail, and when they do, they can build ways to control these problems into their test cases. But what happens when testers don’t have enough time to test all their assumptions? What if the information needed to verify the assumption is not available? When other teams, such as QA or development, are involved in the creation of the product, it can be difficult to control.

Conclusion

ChatGPT is still in its early stages and updates are constantly being made to add features or fix bugs. In addition, it is a model that is constantly learning, so the more people use it and find problems in the platform, the better and higher quality it will be in the future. Its accuracy will continue to increase and the learning gaps will continue to be filled. As we have shown in the article, it is already possible to use Chatgpt in test automation and in the future it will be even more effective in software testing.

Another AI-based tool used in software testing is GitHub’s Copilot. Read the article Github Copilot and software testing.

If you speak German and are looking for a job as an IT tester, take a look at our employee benefits and respond to the latest job offers.