Webinar 5: GenAI to Build Test Cases & Playwright Scripts (Autify)

About Company/Product

Company: Autify

Product:

Autify No-Code (original AI-powered no-code automation tool)

Autify Genesis (new AI-based test case and Playwright code generation solution)

Objective of the Webinar

The webinar aimed to showcase

• How AI (particularly LLM-based approaches) can automate the creation of test cases from diverse requirements (PDF, text, images, Figma, etc.).

• Generating fully functional Playwright scripts using an AI agent that interacts with the application in real-time.

• Demonstrating how testers can reduce effort in test design, coding, and maintenance by partnering with AI.

Presenting the Webinar

Presenter: Ryo (Co-founder and CEO of Autify)

• 10+ years of software engineering experience.

• Focus on AI-driven test automation solutions.

Host: Joe (Test Guild), moderating and facilitating Q&A.

Brief Summary of the Webinar

Autify’s Journey & Vision

• Autify started in 2019 with an AI-powered no-code solution for web/mobile test automation.

• Over time, recognized a big challenge in earlier stages of testing: designing test cases and writing scripts.

• Launched Autify Genesis to address test case creation and script generation using GenAI.

End-to-End AI-Driven Workflow

• Input: Requirements (PDF, Word, images, JIRA tickets, etc.).

• AI transforms them into Gherkin-based test scenarios.

• An AI agent then interacts with the actual web app in real-time to create Playwright test scripts with accurate locators.

Key Technical Challenges

• Purely using ChatGPT for code generation fails because it doesn’t see real DOM structures.

• Autify Genesis uses an LLM plus an AI agent to explore the site, handle authentication, dynamic elements, and gather real locators.

Impact & Early Results

• Autify’s internal usage saved 55% in test case creation time.

• POC with a client showed up to 6.6x faster test case creation and 9.9x faster script generation.

Features and Technical Aspects

AI Agent for DOM Interaction

• While Gherkin provides an abstract test scenario (e.g., “user logs in”), the AI agent inspects the web application to find correct element locators and flow steps.

Multi-Source Requirement Parsing

• Documents, images, Figma files, or tickets can be fed in.

• AI merges these contexts to generate comprehensive test scenarios.

Playwright Code Generation

• The solution auto-generates fully functional Playwright scripts (with accurate selectors) instead of “hallucinated” code.

Collaborative Chat Interface

• A chat UI lets testers correct AI mid-flow (e.g., clarifying credentials or selecting which product to click).

• Minimizes script errors and ensures coverage for dynamic or ambiguous steps.

No-Code + AI

• Autify’s original no-code platform addresses test maintenance and self-healing.

• Genesis complements it by automating the early test design and code creation phase.

Future Job Specifications

Tester Roles

• Testers become “prompt engineers”: refining AI’s output, verifying scenario coverage, guiding the agent’s steps.

• Greater analytical and domain knowledge needed to shape AI’s inputs/requirements.

How GoTestPro Can Compete

Real-Time DOM Analysis

• Must adopt an AI agent approach, not just static code generation.

• Ensures truly valid scripts in dynamic or complex UIs.

Multi-Format Requirement Ingestion

• Accepting PDFs, images, JIRA, etc. helps produce robust coverage.

• Handling partial or ambiguous specs can be a major differentiator.

Deep Integration with BDD

• Using Gherkin or a similar format supporting simpler test readability.

• Facilitates collaboration between dev, QA, and non-technical stakeholders.

Data Privacy & On-Prem

• Offering local or on-prem solutions addresses corporate security concerns.

• AI model-agnostic design (e.g., custom LLM vs. GPT-4) for enterprise trust.

Active Human-in-the-Loop

• Provide an interactive chat or collaboration model to fix AI missteps.

• Position AI as a partner, not a replacement, to gain user adoption.

Additional Important Points

AI Maturity

• Tools are still evolving; early adopters see big time savings but must guide AI carefully.

Comparison to Translation Industry

• Just like machine translation didn’t eliminate human translators, AI-based test creation won’t remove testers. It augments them.

Test Data & Negative Cases

• Next step includes generating boundary/negative test data automatically.

Sign-Up and Beta

• Autify Genesis is in a limited release, with sign-up available.

• Community feedback crucial to refine multi-format requirement analysis and coverage.

GenAI to Build Test Cases & Playwright Scripts - Demo Screenshots:

Open

Open

Open

Open

Open

Question and Answer:

How does AI generate test cases and Playwright scripts?

• Answer: AI analyzes requirements (PDFs, JIRA tickets, Figma mockups) to generate Gherkin-based test cases. It then converts these into executable Playwright scripts by interacting with the application in real-time to identify accurate locators and workflows.

2. Can AI handle dynamic UI elements (e.g., data grids, charts)?

• Answer: Yes, but it requires careful design. For dynamic elements (e.g., sorting data grids), AI can use visual recognition or specific locators (e.g., data-test-id) to ensure stability. No-code tools may leverage screenshots for interaction.

3. How does AI improve test coverage and efficiency?

• Answer: AI automates scenario creation, reduces manual effort by 55%, and accelerates test scripting (6.6x faster for test cases, 99x faster for Playwright code). It also identifies gaps in requirements.

4. Does AI replace manual testers or automation engineers?

• Answer: No. AI acts as a "partner" handling repetitive tasks (e.g., drafting test cases, coding) while humans review, refine, and oversee strategic decisions.

5. How accurate are AI-generated test scripts?

• Answer: ~95% accuracy for test cases, but human review is needed. Playwright scripts are validated in real-time against the application, unlike static ChatGPT outputs.

6. Can AI update tests for UI changes automatically?

• Answer: Yes. AI regenerates scripts when requirements or UIs change. For minor changes, self-healing (classical ML) adjusts locators; major changes require re-generation.

7. How does AI handle test data generation?

• Answer: AI analyzes data definitions and application screens to create varied test data patterns, ensuring coverage of edge cases and input scenarios.

8. What AI models power the tool (e.g., ChatGPT, custom LLMs)?

• Answer: Model-agnostic (supports GPT-4, Claude, etc.). Enterprises can use internal LLMs (e.g., internal GPT) for security-sensitive environments.

9. Is the tool secure for internal/private applications?

• Answer: Yes. A desktop version supports local/on-prem use, avoiding cloud storage of sensitive data. VPN/private network testing is also supported.

10. Why use Gherkin for test cases?

• Answer: Gherkin bridges non-technical and technical stakeholders. AI generates abstract scenarios, reducing maintenance overhead compared to ad-hoc formats.

11. Does it support API or mobile testing?

• Answer:

  • API: Possible (Playwright supports APIs) but not a current focus.

  • Mobile: Planned integration with existing no-code mobile testing platforms.

12. Can AI detect ambiguous/missing requirements?

• Answer: Future updates will flag gaps in requirements by cross-referencing generated test cases with input docs (e.g., Figma, PDFs).

13. How customizable are AI-generated outputs?

• Answer: Full control, users can edit Gherkin scenarios, Playwright code, and locators. Model behavior customization requires vendor support.

14. What’s the biggest challenge with AI test automation?

• Answer: Balancing accuracy with dynamic UIs (e.g., random IDs) and ensuring context-rich inputs (e.g., supplemental docs) for reliable outputs.