Nočnica Mellifera

Frequently in support conversations and posts on Playwright forums, a problem has come up that’s a little bit hard to describe, but comes down to synchronous testing: developers writing a series of Playwright tests that operate on the assumption that one of the tests will either run first or run last, and perform the function of a setup and cleanup script.

I’ve seen posts on the Playwright subreddit where someone tried adding hard waits 😓 to make one test take a huge amount of time, or others where they added waits to all the other tests to make one run first. I want to talk about why we might get into this situation, and some best practices to get out of it.

The Problem: we want ‘synchronous’ tests, but they're not a great idea

There are a number of good reasons why we’d want to designate ‘synchrony’ in our tests.

Note: for the sake of my tortured grammar throughout the rest of this article, I’m going to refer to wanting one test to run before everything else for setup. Just know that everything below applies equally well to wanting to run one test last to do cleanup.

We might want to have authentication completed before we do next steps. Or check an API endpoint for configuration before moving on to run our tests. Perhaps we want to get extremely fancy and create a kind of dynamic test, where the setup gets a list of random subdomains to test, and the test suite can be altered dynamically by changing that list in one place. Neat stuff!

More generally, a single starter/setup test is tempting as it helps us follow the ‘don’t repeat yourself’ (DRY) principle of coding: we don’t want everyone writing every test to do the same setup steps, let’s run it once and save time and effort, and put all that setup in one place where it’s easier to maintain and update.

At Checkly, and in the broader Playwright community, it’s always a safe bet to assume that your test coverage is going to expand. Even with a set number of API routes, pages, and paths to check, the detail level of testing will increase over time. The complexity of things we’re monitoring will go up, and the overall number of tests will increase. This is a good thing: it means better monitors giving more accurate results, and as configuration improves with experience, it means better test coverage with fewer false alarms.

But this is the downfall of synchronous tests: the more we force our tests to run one at a time, the longer we’re making our tests take.

This horse race will take less than two minutes, how long will it take if each horse has to wait for the horse before it to finish?

With asynchronous, parallel testing, adding a test might not increase total runtime of the test suite at all. Resource use will be higher the more tests we have, but the total time to run all tests will still be near the time it takes for the slowest test to complete.

In this example doubling the number of tests didn’t increase total test time.

Picture a horse race, with each horse forced to wait for the previous horse to finish. Not only with this take longer, it means that every additional horse increases the total time by its run time.

In synchronous ‘one at a time’ scheduling, doubling the number of tests triples the overall runtime.

Note that when running Playwright scripts locally you do have some control over how your individual spec files will be run, whether in parallel or in sequence, however this doesn’t mean that relying on one-at-a-time testing is best practices. The disadvantages of sequential test running in general are worth considering, specifically in the Checkly system, trying to run one test before everything else is going to mean fighting against a system optimized to get you results quickly and consistently. Checkly runs each check from its own container which is stood up for this purpose, this ensures a secure and consistent platform for each test, but since each test isn’t being run off the same machine, it’s much more difficult to ensure which test will run first. To try to force synchrony is to fight against the very system that alerts you to downtime in the fastest possible time, and therefore the system that ensures you meet your SLAs.

The solution: alternatives to synchronous tests

All right, if you’ve read this far you’re probably starting to be convinced that we need something better than running tests in a set order. What’s the alternative? The good news is that there’s no benefit of synchronous tests that you can’t get via another route.

Playwright fixtures to keep your code DRY

One common reason for synchronous tests is the desire to do setup steps just once and then set a variable to hold necessary details for all other tests, meaning less repeated code. This again runs into the high security and isolation of the Checkly system, where checks running separately won’t share the same context, but it is a laudable goal to not re-write code all over the place. The solution is to use test fixtures to make repeated code available in multiple places. Test fixtures can help you make single calls like user.login.verify(userID) to do complex login steps with all the advantages of a single dependency: easier updates, easier onboarding of engineers, and more maintainable code.

To start using fixtures in Playwright, first, extend the test object provided by Playwright to create a custom fixture. For example, if you need a login step for multiple tests, define a fixture that handles the login process. Use test.extend to set up the fixture, adding your logic (like navigating to a page and filling in credentials) and passing the result to your tests with use. Then, replace repetitive code in your tests with the custom fixture. For instance, instead of writing login steps in every test, simply call the fixture. The whole fixture process is on our docs page. This keeps your code clean, reusable, and easy to maintain.

Check out Stefan’s video of test fixtures in action:

How to parameterize and configure your custom Playwright fixtures

Global beforeEach and afterEach hooks to set up and clean up

Checkly supports hooks to allow you to run code before and after each check, this is especially useful if the action you’re taking is stateful like setting up a user or making changes to a demo user account with each check. For example, use test.beforeEach to navigate to a specific URL or log in before every test, and test.afterEach to clean up resources. Check out Stefan’s video demonstrating how to add use these hooks.

How to combine POMs (Page Object Models) with Playwright Fixtures for better developer experience

Combining tests and adding test steps for documentation

During a recent session of our weekly Kick-start webinar, an engineer asked a question that boiled down to:

Is it better to write a bunch of small tests that each check one thing, or one huge test that simulates every step of a long user journey?

And I realized there was no single right answer to this question! In the context of trying to not repeat code, there can be real benefit to combining tests into a single spec file. Further, if any part of a user journey is critical enough that its failure cuts off everything that comes later, for example if ecommerce users couldn’t add items to their cart, it probably makes sense for a ‘user checkout’ test to include the browsing and ‘add to cart’ steps. The big advantage of many small tests, in a context where each step still relies upon the last, is documentation. It’s a lot more useful to get an alert ‘failure on add-to-cart step’ than ‘failure in web shop’. However, if you want more clear testing reports, it’s not necessary to decompose all your tests into small pieces. Just use test steps!

Test steps are a documentation-only change to your tests that help you see what part of a test failed. In this example where we navigate a few pages and check for a link, test steps define two separate parts to this test:

This makes it easier to read the report on a larger and more complex tests.

Conclusions: embracing parallel testing while keeping DRY

Playwright is a framework created for coders, with an eye to creating consistent and maintainable scripts. As such, once we start wrestling with the framework and forcing behaviors for which there’s no existing support, we’ve probably lost the plot. With fixtures and each hooks you can get the same behavior you’re looking for with running a single test before the others, with another one after everything else has run. Finally, consider combining inter-dependent tests into a single test, with test steps to document which part of a test failed.

If you’d like to go further, explore our Learn Playwright site, or join our community Slack to meet a community of like-minded PWT testers.

Detect

Communicate

Resolve

Monitoring as Code

Developers

Resources

Community

Why you shouldn't run tests sequentially