Let's explore several options for bulding a test automation framework in all layers.
Alternatives
For CI/CD Integration and setting up pipelines, there are several options. Jenkins, Github Actions and Gitlab Runner are the most commonly used platforms. For automating tests, depending on the programming language, there are multiple ways to choose. Focusing on JS, Jest, Mocha and Jasmine are possibly the best options. Mocha can be used with Chai assertion library, where the other two already have assertion pacakges inside. Besides automating tests with Selenium on top of one of these, an E2E test framework can be choosen. The most powerful candidates are Playwright, Cypress and TestCafe. A table summarizing alternatives would look like:
E2E Test Frameworks
To be able to see the capabilities and how convenient each of them, I implement the same scenario with several JS based E2E frameworks. After the scenario is automated, can be executed via:
Playwright : playwright test testffolder --repeat-each=15
TestCafe : testcafe chrome :headless testffolder
Nightwatch : nightwatch -e chrome –headless ./nightwatch/specs/*.js
Cypress : cypress open --browser chrome run --spec "**/SearchButton.spec.js"
Selenium + Jest : jest --config=selenium/tests/config.prod.js
TestCafe : testcafe chrome :headless testffolder
Nightwatch : nightwatch -e chrome –headless ./nightwatch/specs/*.js
Cypress : cypress open --browser chrome run --spec "**/SearchButton.spec.js"
Selenium + Jest : jest --config=selenium/tests/config.prod.js
Pro's and con's:
Playwright : (+) has device emulators
Playwright : (+) multiple & parallel runs
Playwright : (+) fast: using Browser content rather than browser itself (background: Playwright uses a WebSocket connection rather than the WebDriver API and HTTP.)
Cypress : (-) does not support safari -> webkit support is just introduced but with Playwright
Cypress : (+) has dashboard
Cypress : (+) supports tags
Nightwatch : (+) Fast
Nightwatch : (+) Understandable
Nightwatch : (-) Poor documentation
Selenium : (-) Interceptions not easy
Metacafe : (-) To type in text field, clicks
Playwright : (+) multiple & parallel runs
Playwright : (+) fast: using Browser content rather than browser itself (background: Playwright uses a WebSocket connection rather than the WebDriver API and HTTP.)
Cypress : (-) does not support safari -> webkit support is just introduced but with Playwright
Cypress : (+) has dashboard
Cypress : (+) supports tags
Nightwatch : (+) Fast
Nightwatch : (+) Understandable
Nightwatch : (-) Poor documentation
Selenium : (-) Interceptions not easy
Metacafe : (-) To type in text field, clicks
Execution durations are (in msec):
Overall, for me Cypress's dashboard is the biggest weapon for them. If Safari automation is needed, there are two options: Playwright and TestCafe. Not only for this, but for ease of use and performance, Playwright is my favorite. Then, Cypress in case of no Safari need, otherwise TestCafe.
Snapshots from test classes in each framework:
