While the code-based web application testing framework had significantly advanced over the past years, and both the competition between the leading frameworks as well as the richness of offering these frameworks provide grew, we have also experienced another interesting...
While the code-based web application testing framework had significantly advanced over the past years, and both the competition between the leading frameworks as well as the richness of offering these frameworks provide grew, we have also experienced another interesting trend in this landscape – codeless capabilities.
Out of many solutions, I have decided to look into the rising Cypress Studio (Beta) and Playwright CodeGen tools to review their low-code abilities. Note that there are more mature commercial solutions in the market like Perfecto Codeless and Testim, however, here we will focus on the 2 open-source frameworks.
Before diving into these tools, here are few of the capabilities that I would expect to find in a complete low code/no-code test automation solution
Coverage of most testing types including functional, performance, accessibility, APIs, and more.Parallel and scalable testing abilities like we have with Selenium Grid within cloud providers or in the open-source.Ability to import code module into the codeless frameworks and export codeless created script into code like JavaScript.Test reporting and analysis with rich artifacts and with abilities to slide and dice test dataIntegrability into DevOps and CI/CD tools like Jenkins, GitHub Actions, and moreEase of use and quick ramp up with the toolStrong community, support, and documentationSince both Cypress and Playwright are emerging frameworks, both cannot cover all of the above capabilities, however, they can provide some impressive test creation features. Note that the 2 frameworks are different by definition.
Cypress is a JavaScript only framework that runs inside the browsers with rich debugging and functionalities. Playwright is a chrome debugger protocol (CDP) framework that is led by Microsoft and through its architecture can support more language bindings and more browsers including WebKit Safari that isn’t supported yet by Cypress.
Playwright CodeGen (Test Generator)
Once you install Playwright Node library following the below commands, by definition you get the CodeGen library.
npm i -D @playwright/test
By running the following command from your local Playwright installation folder within your IDE terminal (in my case VS Code), you launch the test generator tool and can record any action on your web application that in parallel generates JavaScript test code.
npx playwright codegen http://continuoustesting.dev
npx playwright install
As you can see in the above recorded test code, it navigates to my blog webpage and performs some basic navigation and clicks on different links and menus. The generated code can than be exported and saved in different language bindings including the native Playwright Test, Java, JavaScript, Python, and C#.
Another cool feature of the Playwright CodeGen tool is the Explore button. This capability enables developers to mouse hover the web application under tests and inspect the elements via the Playwright Selector. When selecting an element from the tool you can than copy it into your IDE test code and proceed with your test creation. In addition, you can record, than stop, and continue recording few times as long as the tool is still running. Lastly, you can do basic debugging of your recorded test from the Playwright Inspector CodeGen tool through stepping over, execution stop and more.
Cypress Studio (Beta)
Like with Playwright, the codeless recorder tool is built into Cypress, but needs to be enabled through a command added to the cypress.json configuration file. Note again, this tool is still in an experimental stage and in beta as this blog is being written.
To start using the tool, simply enable it by adding the following line to your cypress.json file:
{ "experimentalStudio": true }
From this moment, the tool is enabled, and when you run Cypress in GUI mode through
npx cypress open
You can than edit an existing JavaScript test spec file and click on the red recording button to inject recorded test code into your script, or build a whole new project. Cypress documentation recommends cloning their built-in GitHub repository and using the following JavaScript test spec as a ‘getting started’ project (https://github.com/cypress-io/cypress-realworld-app/blob/develop/cypress/tests/demo/cypress-studio.spec.ts)
As you can see from the above screenshot, once the Cypress Studio is enabled, editing a test step triggers the Cypress Studio button where you can add new commands to you test. As of now, 5 main test commands are supported by the tool – Check, Click, Select, Type, and Uncheck.
As mentioned above, you can create a brand new test through the recorder or add new code into an existing spec file.
I ran the tool and added a simple exploratory test into the built-in test spec called todo.spec.js that comes with the Cypress open-source framework, below is the generated JavaScript code.
Key Takeaways
It is clear from my quick overview of both codeless tools that they are only in the beginning and that there is potential to grow and add more of the above core capabilities of a fully ready test automation framework. Since these two tools are provided for free and in early stages, we shouldn’t complain, but rather embrace, and provide feedback to the owners of these frameworks to improve.
Advantages of Playwright CodeGen:
More language bindings to export the created codeMore browsers to run the tests onThe Explore object spy is quite easy to useTool is separated from the basic code-based GUIDisadvantages of Playwright CodeGen:
Hard to debug testsLimited test creation of the tool, no automated assertionsNo documentation and examples to get started withAdvantages of Cypress Studio
Easy to use from the Cypress GUI toolFriendly debugging toolRicher test creation abilitiesGood documentation and sample project to help ramp upDisadvantages of Cypress Studio
Limited test creation and assertion within the toolOnly exports to JavaScriptNo support for all browsers (Safari WebKit)Bottom Line
The 2 tools are a great start and an optimistic horizon for codeless tools complementing code based tools. Both are still growing and there are many gaps to close before they can be self-served by front-end developers and SDETs, however, they provide a sing for the future of intelligent test automation and this is promising. We should continue following the evolution of these tools including the newly Selenium IDE within Selenium 4 and continuously provide feedback to the communities to grow and build these tools right.
