Table of Contents

Exploring the power of GenAI and Schema-Driven API test generation: The results.

In a previous article (Efficient API Test Automation: Exploring the power of GenAI and Schema-Driven Test Generation), we explored the potential of employing Generative AI tools to generate API automated tests based on a provided API Schema. In this piece, we are going to dive into the practical approach, hands-on methods, prompt techniques, and how they impact the quality of the results and, ultimately, we’ll determine the real-world usefulness of the outcomes generated by the tool.

The tool:

We decided to do a first proof of concept using one of the most popular and powerful Generative AI tools, ChatGPT. 

The schema:

We are using a popular testing API from Swagger https://petstore.swagger.io/v2/swagger.json

This is a JSON file that contains the API schema of a virtual pet shop. The API contains some of the most typical endpoint's operations. To maintain brevity in this article, we will show the results of using just one endpoint:

Screenshot 2023-11-17 at 12.46.36

 

The prompts:

A prompt guides the direction of an AI's response, serving as the catalyst for relevant and contextually appropriate output. In essence, a prompt acts as both a directive and a filter. It directs the AI on what to focus on and filters out unwanted or irrelevant information, ensuring that the response is in line with the user's intention. 

We conducted experiments using various prompts to assess how ChatGPT responds to each, ranging from basic prompts with minimal information to more detailed prompts with additional context and instructions. For the purpose of this article, we will only present the results from the more complex prompt.

Act like an SDET. Analyze https://petstore.swagger.io/v2/swagger.json. You want to focus specifically on the endpoint “/pet”. Create tests using Jest and Supertest. Additionally, provide valid and invalid request payloads, add assertions based on the expected response of the endpoint, and consider any negative and positive scenarios.

The results:

Screenshot 2023-11-17 at 12.48.51

First, it provides a step-by-step guide on how to set up the project.

Screenshot 2023-11-17 at 12.50.24

Then, it generates a set of tests to cover different scenarios for that endpoint

Findings: 

  • It provides a step-by-step guide to set up the project.
  • It generated a few tests for the requested endpoint and included test payloads for valid and invalid scenarios.
  • Overall, the tests generated are of good quality, but there is room for improvement. This can be achieved either by instructing the AI to make enhancements or by allowing an automation engineer to modify the code as required.
  • It also specifies that the code generated it’s “a simplified example, and in a real-world scenario, you would want to cover more edge cases”

Link to the chat: https://chat.openai.com/share/41930e16-a9d8-49c0-8492-f598e782a112 

After observing these results, let’s explore whether ChatGPT can generate nonfunctional tests, such as Security tests.

Screenshot 2023-11-17 at 12.52.40

It starts by identifying the potential security concerns for the given endpoint.

Screenshot 2023-11-17 at 12.53.43

Then, it generates a set of security tests based on the risk analysis it made.

 

Screenshot 2023-11-17 at 12.54.27Finally, it explains the tests created and makes a recommendation about security assessments.

Findings: 

  • ChatGPT was able to identify the security concerns of the given endpoint.
  • It gives a detailed explanation of the risk assessment, which is helpful for the user to understand the potential vulnerabilities.
  • ChatGPT generated a set of security tests based only on the analysis it made of the given schema. 


Link to the chat: https://chat.openai.com/share/548e1318-5dc0-4fc5-9b50-860beb756253

Key takeaways:

  • When a user provides a JSON schema or the specific data for the endpoints they wish to test within a single prompt, ChatGPT has the capability to generate an API test automation project from the ground up. This project includes clear instructions and step-by-step guidance for the user to follow. Additionally, it incorporates illustrative test cases, complete with their respective payloads and assertions, along with valuable recommendations for project improvement.
  • Upon attempting to execute the provided code, it is fully functional. However, there may be some updates required, which are thoughtfully detailed by ChatGPT within comments embedded in the code itself, as well as in the final comments of the output.
  • It's important to note that there will always be some level of work for the automation engineer to undertake in order to review and tailor the code to meet specific requirements.
  • Naturally, in addition to the mentioned tests, there exists a plethora of other test types, such as performance, edge cases, and negative tests, that can be implemented as needed. As you can see in the Security example, ChatGPT is capable of generating nonfunctional test scenarios as well.
  • The quality of the prompt has a direct impact on the outcomes produced by the tool, and there is room for enhancement by making it more specific. In this proof of concept, our intention was to assess result quality through the utilization of a straightforward set of instructions. 

About Encora

Fast-growing tech companies partner with Encora to outsource product development and drive growth. Contact us to learn more about our software engineering capabilities.

Learn More about Encora

We are the software development company fiercely committed and uniquely equipped to enable companies to do what they can’t do now.

Learn More

Global Delivery

READ MORE

Careers

READ MORE

Industries

READ MORE

Related Insights

Future-proofing Skills: Thriving in the Transforming Energy & Utilities industry

Encora has developed video-based and game-based microlearning series for topics such as contingency ...

Read More

Real-Time Communication Simplified: A Deep Dive into Server-Sent Events (SSE)

In this blog, we dive into fundamental concepts behind client-server interactions, mainly focusing ...

Read More

Bank Modernization: Leveraging Technology for Competitive Advantage

Banks leverage innovative tech like Generative AI, data analytics, and core modernization to ...

Read More
Previous Previous
Next

Accelerate Your Path
to Market Leadership 

Encora logo

+1 (480) 991 3635

letstalk@encora.com

Innovation Acceleration

Encora logo

+1 (480) 991 3635

letstalk@encora.com

Innovation Acceleration