Spike: AI in GoTestPro
- AJ Jobanputra
2023 AI Prototype
Requires user to describe what scenario it needs to create
It works on understanding the flow, extracting context from the html and generating test steps that are compatible with gotestpro test steps.
We spent more than 1 mo on this step generation
Success Outcomes:
generated test steps
worked in gotestpro
Failure Outcomes:
inconsistent results, steps don’t always work and scenarios don’t work
Next steps:
chatgpt3.5 has changed, prompt engineering prompts have changed quite a bit
narrow scope to Salesforce Commerce as per conversation w Asif Lala
combine w Zerostep prototype
ZeroStep Prototype
We have prototype of taking a text request where user says “click on product info” and the library interacts with the browser and clicks on the product and proceeds to the product detail steps.
Success Outcomes:
Generated test steps, it executes the steps in the browser
Can import steps from a CSV file, so if user has manual test cases, then they can input test cases into the sequence
tested with sample shoe store ecommerce (air birds current customer)
Failure Outcomes:
Inconsistent results, especially with menu items or items where labels aren’t clear.
Same result that works one time, may not work successfully the next time.
Prompt engineering the text command makes a huge difference.
Code:
Details are provided below.
https://github.com/Srinivas4nowGitHub/PlayWrightGPTis the GIT repository.
Please use the following command to launch the Playwright test script:
npx playwright test
For the following requirement, kindly refer to the attached document containing AI-generated test case names:
Requirements: I need a rest endpoint so that I may add, update, and remove products. create test case names to adequately test this endpoint.
AI Test Generation from API test Generation
Using prompting we can generate aI Test Cases based on an API specification
Results
Test scenarios are good and reasonable
Next Steps:
work with the scenario
Here is a comprehensive list of potential test case names to thoroughly test a REST endpoint for adding, editing, and deleting products:
1. General Endpoint Behavior
1. Verify the endpoint responds with 200 OK for valid requests.
2. Verify the endpoint returns 404 Not Found for invalid routes.
3. Verify the endpoint returns 405 Method Not Allowed for unsupported HTTP methods (e.g., PUT on delete).
4. Verify the endpoint handles empty request bodies gracefully.
2. Add Product (POST)
1. Verify a product is successfully added with valid data.
2. Verify the response contains the correct product ID after addition.
3. Verify the endpoint rejects requests with missing required fields (e.g., name, price).
4. Verify the endpoint rejects requests with invalid field values (e.g., negative price, invalid SKU).
5. Verify duplicate products are not added if uniqueness constraints exist (e.g., SKU).
6. Verify adding a product with special characters in fields (e.g., name).
7. Verify adding a product with large payload sizes.
8. Verify adding a product with optional fields omitted.
9. Verify the endpoint handles invalid JSON format in the request body.
10. Verify the response time for adding a product is within acceptable limits.
3. Edit Product (PUT or PATCH)
1. Verify a product is successfully updated with valid data.
2. Verify updating a product returns the updated data in the response.
3. Verify attempting to update a non-existent product returns 404 Not Found.
4. Verify the endpoint rejects updates with missing required fields.
5. Verify the endpoint rejects updates with invalid data types (e.g., string for price).
6. Verify partial updates work correctly if supported (e.g., PATCH).
7. Verify optional fields remain unchanged if not included in the update request.
8. Verify the endpoint prevents updates that violate business rules (e.g., setting stock below 0).
9. Verify updating a product with special characters in fields.
10. Verify the endpoint rejects invalid JSON in update requests.
11. Verify the response time for updating a product is within acceptable limits.
4. Delete Product (DELETE)
1. Verify a product is successfully deleted with a valid product ID.
2. Verify attempting to delete a non-existent product returns 404 Not Found.
3. Verify the endpoint prevents deletion of a product linked to active orders (if applicable).
4. Verify deleting a product does not affect unrelated products.
5. Verify the response body (if any) and status code after a successful deletion.
6. Verify the endpoint rejects invalid product IDs (e.g., negative or non-numeric values).
7. Verify the endpoint handles concurrent delete requests gracefully.
8. Verify the endpoint prevents deletion with invalid authentication or permissions (if required).
9. Verify the response time for deleting a product is within acceptable limits.
5. Security and Authentication
1. Verify the endpoint requires authentication for all operations (add/edit/delete).
2. Verify the endpoint rejects unauthorized requests with 401 Unauthorized.
3. Verify the endpoint restricts access based on roles or permissions (e.g., admin-only access).
4. Verify the endpoint handles expired authentication tokens correctly.
5. Verify the endpoint prevents SQL injection attacks through input fields.
6. Verify the endpoint handles Cross-Site Scripting (XSS) attacks in string fields (e.g., product name).
7. Verify the endpoint prevents access to sensitive operations without HTTPS.
6. Error Handling
1. Verify the endpoint returns appropriate error messages for missing parameters.
2. Verify the endpoint returns 400 Bad Request for invalid input data.
3. Verify the endpoint returns 500 Internal Server Error for server-side issues.
4. Verify error messages are clear, descriptive, and actionable.
5. Verify the endpoint handles database connection issues gracefully.
7. Performance and Load Testing
1. Verify the endpoint handles a high volume of concurrent requests for adding products.
2. Verify the endpoint maintains performance under a large number of products in the database.
3. Verify the endpoint handles rate-limiting or throttling if implemented.
4. Verify the response time under normal and peak load conditions.
8. Data Integrity
1. Verify the product data remains consistent in the database after adding/editing/deleting.
2. Verify related entities (e.g., categories or inventory) are updated correctly after operations.
3. Verify rollback functionality if a transaction fails during add/edit operations.
This structured set of test cases will help ensure the endpoint is robust, secure, and performs well under various scenarios. For more advanced scenarios or test case generation, try Hix AI.
LLM Cost Performance Benchmarking
Idea is to see if the current AI setup if price effective in a chatbot/llm application. We want a way to benchark to see if an llm app is correct AND also see at what price.
tool that will run simulation of tests against a chatbot
it simulates multiple types of tests:
happy path
confusing questions
inappropriate questions
abort scenarios
it measures chatbot accuracy: did it give correct responses or not?
it also measures number of words and tokens
between test runs you get a price/performance report