Understanding Playwright's Selector Detection and Locator Generation

1. How Playwright Detects Selectors

Playwright uses a sophisticated approach to detect and generate selectors:

1.1 Event Interception

• Intercepts DOM events (clicks, inputs, etc.) on the page

1.2 Element Identification

• Identifies the target element when an event occurs

1.3 Selector Generation Process

 a. Unique Attributes: Checks for id, data-testid, or aria-label

 b. Text Content: Uses text content for elements like buttons or links

c. Role-based Selectors: Generates selectors based on ARIA roles

 d. CSS Selectors: Falls back to CSS selectors if needed

1.4 Selector Optimization

• Optimizes the generated selector for uniqueness and stability

1.5 Special Handling

• Handles elements inside iframes and Shadow DOM

1.6 Key Components

• Implemented in files like:

  • packages\playwright-core\src\server\injected\consoleApi.ts

  • packages\playwright-core\src\server\injected\recorder\recorder.ts

  • packages\playwright-core\src\server\injected\selectorGenerator.ts

  • packages\playwright-core\src\server\injected\injectedScript.ts

Playwright's Locator Generation: Integration and Complexity

It's important to note that we cannot easily extract Playwright's locator generation functionality for use in a custom recorder. Here's why:

2.1 Tight Integration

• Playwright's core InjectedScript, which handles locator generation, is deeply integrated with other Playwright components

2.2 Complex Dependencies

• Relies on various parts of Playwright's codebase, including DOM utilities and internal APIs

2.3 Context Awareness

• Requires context from the entire page, including iframes and shadow DOMs

2.4 Evolving Implementation

• Frequently updated and optimized, making a separate copy hard to maintain

2.5 Internal APIs

• Uses non-public APIs that may change without notice

2.6 Performance Optimizations

• Includes optimizations dependent on Playwright's internal workings

2.7 Browser-Specific Logic

• May contain browser-specific code paths

how to use playwright libraries

Playwright is primarily designed for web automation and testing, but it can be used in creative ways to build various record and playback applications. Here's a brief overview of how to use Playwright, followed by some ideas for record and playback applications.

1. Desktop App for record and playback user interaction for web apps.

https://github.com/elastic/synthetics-recorder/tree/main

Above recorder is designed by elastic search team of their particular use case.

Note that the above recorder is in a Tech Preview phase at the moment, and not supported.

The idea is we can look into their source code and we can build some thing similar like mabl.

2. Playwright-CRX ( chrome Extension ).

https://github.com/ruifigueira/playwright-crx

This package contains the Chrome Extensions flavor of the Playwright library.

3. Existing chrome recorder

Using existing recorder we can generate playwright steps using playwright locator generator.

below are some files which are responsible for selector generation

Implemented in files like:

• packages\playwright-core\src\server\injected\consoleApi.ts

• packages\playwright-core\src\server\injected\recorder\recorder.ts

• packages\playwright-core\src\server\injected\selectorGenerator.ts

• packages\playwright-core\src\server\injected\injectedScript.ts

It's not an easy job. A deep dive is necessary to fully understand their locator generation engine

Here's a concise breakdown of the pros and cons for implementing a Playwright script recorder as a Chrome extension versus a desktop app:

Chrome Extension:

Pros:

1. Easy distribution through Chrome Web Store

2. Seamless integration with Chrome browser

3. Cross-platform compatibility (works on any OS that runs Chrome)

4. Automatic updates

Cons:

1. Limited access to system resources

2. Restricted to Chrome browser only

3. Potential performance limitations

4. Subject to Chrome's extension policies and restrictions

Desktop App:

Pros:

1. Full access to system resources

2. Can work with multiple browsers

3. More flexibility in design and functionality

4. Potentially better performance

Cons:

1. More complex distribution and installation process

2. Requires separate builds for different operating systems

3. Manual update process for users

4. May require additional security measures

Conclusion

While we can't use the full Playwright library directly in Chrome extensions due to its Node.js dependency, we've found a solution in Playwright CRX. This flavor of Playwright is specifically designed for Chrome extensions, allowing us to leverage much of Playwright's functionality in a browser-compatible format.

We're planning to use specific JavaScript files from Playwright CRX that generate locators and action code snippets. This approach gives us access to Playwright's powerful selector generation capabilities and automation features, tailored for use within Chrome extensions.