Nočnica Mellifera

Playwright isn’t a testing framework. Sure it’s got assertions, scripted behaviors, even controls over environments. But testing isn’t Playwright’s only purpose. Playwright is an automation tool. It can carry out any browser-based action consistently, and carry out instructions robustly. Locators for buttons and other elements aren’t visual or CSS class-based, but based on ARIA role, and even small styling changes won’t make the scripted action fail. Of course, all this is useful if you’re using Playwright to run tests.

As an automation framework, Playwright is a bit like a loyal robot that works tirelessly to follow your exact orders. And when things go wrong, it sometimes seems that the fault isn’t with the robot but the nature of our instructions. Sometimes our instructions are too vague, sometimes too restrictive, but there are a small subset of instructions that are almost never a good idea to use. One of those is page.waitForTimeout(), which results in brittle tests, slow tests, and a frustrating Playwright experience.

The function page.waitForTimeout() tells the current script to wait for a perfectly fixed length of time. The result is a ‘hard wait’ that can’t be interrupted. Here’s why hard waits are almost always a bad idea.

The problem: hard waits are bad in either case

Generally page.waitForTimeout() is used in a script like the following (in pseudocode):

Forcing a wait seems like a sort of ‘safety’ making sure that you definitely allow time for an element to load before checking for it. Let’s talk about how this is an antipattern if the element ever takes longer than our timeout to load. Ironically, this behavior is also not ideal if the element loads much faster than expected, which will be explored next.

Even in testing, a false positive is a problem, but when the power of Playwright is harnessed by Checkly to make production monitors, false positives can be expecially frustrating, let’s imagine a scenario where we added a hard wait to our script, and an element took very slightly longer than the scripted amount of time to load.

This is the dreaded ‘flaky test’ that ruins the lives of so many SREs and Ops engineers. A test that raises a line when anything looks a bit off, often returning to normal on its own moments later. People get woken up, have to start an investigation, only to find that their monitoring tool has gone green with no human intervention. After a few such incidents, alerts get muted, phone ringers are turned off, and eventually a system meant to detect major outages is now being ignored.

Now you might ask, isn’t this a problem with any kind of assertion, or script, that it will failure if there’s ever a minor infraction? This is the best part of best-practices Playwright: automatic retries. A huge class of Playwright assertions are auto-retrying assertions, that will automatically retry a check if it fails.

Hard waits also fail if an element loads quickly

While the failure above is frustrating and all-too-common, there’s another silent problem with hard waits: wasted time.

The auto-retrying methods mentioned above start trying to find an element almost instantly, and as soon as the element is available they can move on to the next step. With a hard wait, the script cannot continue before the time has elapsed.

It might seem minor that our Checkly monitor has to wait a few seconds even after an element has loaded, but minutes are made of seconds, and 60 minutes make an hour. Let’s do some quick math in a scenario:

Dave is a diligent app developer who wants to thoroughly test his web UI. He also knows that sometimes the app UI needs a few seconds to load, so he adds await page.waitForTimeout(5000) before he checks for each element. There are five elements in the page, so he adds five hard waits. Now he knows he doesn’t want any false alarms, so he adds a linear retry policy: if a test fails, it will retry after 1 minute, and if that fails it will try again after two minutes.

Seems pretty reasonable, right? But would you believe we’ve just set ourselves up to never meet our SLA? Think about it: Meeting SLA means fixing all issues in time, meaning we’ve got a time budget for issue resolution, in other words we need to keep our mean time to resolution (MTTR) low to meet SLA.

With a time budget of about 10 minutes per incident, this Checkly-detected failure is going to meet SLA

With added hard waits, we’ve just added 30 seconds at least to the execution of the tests, but that delay will repeat for each retry. When the content fails to load, the test will still wait 30 extra seconds, a total of 1.5 minutes, additionally.

This fix took more than 10 minutes to deploy, breaking SLA

Every incident now brings us closer to SLA violation. That’s not to consider the fact that less reliable tests means the chance goes up that a critical alert will be ignored, since they’re often false alarms when using hard waits.

The solution: there’s always a better way

This article has made only brief reference to the right ways to wait for a page element, but the good news is we have a whole page on waiting and timeouts in our documentation. We’ve also got a list of all the assertions that automatically retry, causing more consistent tests that also run faster. Once in a while even these standard methods don’t work, but good news I recently wrote a guide specifically on what to do when auto-waiting isn’t working as normal.

Conclusions: it’s only good to be flaky if you’re a pie crust

I hope that this article has convinced you that it’s not a good idea to add hard waits to your Playwright tests. We can all write code that doesn’t perfectly follow best practices, and if there are some await page.waitForTimeout() calls in your currently running code, it’s probably not a concern until you need to reduce flakiness or speed up your tests. However I do hope you’ll remove hard waits from your “quiver” of standard tools when writing tests.

If you’d like to learn more, join our slack community to talk with other Playwright developers working at the highest level. Happy testing!

Why you should never use page.waitForTimeout() in Playwright

Detect

Communicate

Resolve

Monitoring as Code

Developers

Resources

Community

Why you should never use page.waitForTimeout() in Playwright