Table of contents

Playwright Assertions - Types & Best Practices

Playwright is more than a testing framework. With its generalized tool at simulating user behavior in a browser (or via an API), playwright is better defined as a web automation framework. You can use Playwright to scrape websites, automate form submissions, or any other time it would be helpful to replace a human web user with a robot. For most users, Playwright is intimately associated with end-to-end tests, and for these we must stop our automated actions and check to see if everything is working right.

To make an automation into a test, we have assertions. When a hard assertion fails, the execution of a Playwright test stops, and following steps are not run.

Playwright offers a variety of assertion types to accommodate different testing scenarios, and there are two broad categories for flexible, Web-first assertions and rigid, brittle, non-retrying assertions. Web-first assertions return promises, since they will return when they are succesful or time out, where as non-retrying assertions throw their response immediately.

Auto-retrying assertions automatically recheck conditions until they pass or a timeout is reached. They both make a test more stable, and can make tests execute more quickly, and in the language of game theory should be ‘strictly better’ than non-retrying assertions.

Non-retrying assertions evaluate conditions only once. They are used when you expect a condition to be deterministic and want to avoid any retry logic.

This makes the most sense when using Playwright to run single end-to-end tests in a controlled environment. If you’re using Playwright to run on a cadence and monitor a production system, for example, with Checkly, you generally want to stick with the auto-retrying assertions.

Negating matchers allow you to assert that a condition does not hold. For example, you can use toBeFalsy or not.toContain to make negative assertions (as will be explained later, if you’re checking an element directly, probably the latter of these two options will work better as it implements Playwright’s auto-retries).

Soft assertions collect all failures within a block before failing the test, rather than stopping at the first failure. This approach is useful for gathering more complete information during test runs. Soft assertions are a critical component of production monitoring, as when measuring things like load times, a failing response time may not mean that we want to stop the check entirely. Any Playwright test with failing soft assertions will be listed as failed, but the test will still run to completion (or the next hard assertion that fails).

Checkly can use soft assertions as part of a ‘degraded’ state for checks. If a check fails soft assertions and there is a call to markCheckAsDegraded, the check will be placed in a ‘yellow light’ category of degraded rather than failing. Instead of categorizing the check as a total failure the ‘degraded’ state signals that the check encountered issues but still completed execution. Degraded checks have different notification policies. For example: you might have degraded checks report to Slack rather than sending a notification that wakes people up.

This is useful for distinguishing between partial service disruptions and complete outages, offering teams a more precise understanding of performance and reliability. If your Playwright test is running on Checkly, the check will enter a ‘degraded’ state when some soft assertions fail, or if you call markCheckAsDegraded in its execution.

import { test, expect } from "@playwright/test"
import { getAPIResponseTime, markCheckAsDegraded } from "@checkly/playwright-helpers"

const baseUrl = "https://api.spacexdata.com/v3"

test("SpaceX-API Dragon Capsules & Next Launch", async ({ request }) => {
  await test.step("get all capsules", async () => {
    const response = await request.get(`${baseUrl}/dragons`)

    // Hard assertion for a 200 status code, this passes!
    expect(response).toBeOK()
    
    // Soft assertion of a very fast response time
    expect.soft(getAPIResponseTime(response), 'GET /dragons too slow').toBeLessThanOrEqual(200)

    return response.json()
  })

  // Mark as degraded if we got any errors. 
  if (test.info().errors.length) {
    markCheckAsDegraded('Check degraded due to soft assertion failure.')
  }
})

In this case if the response has a 200 status code but the API response time is more than 200 seconds, the status of this check will be degraded and the slow api response time will be included as a failed step. If the response came back as a non-200 status (indicating an error) then the check would be ‘failed,’ whether or not markCheckAsDegraded was called. You can configure alert channels to notify you when a check has degraded.

While Playwright has almost every scenario for testing supported in its core methods, it remains quite customizable, even allowing custom assertions.

• expect.configure: Configures custom options for assertions.
• expect.poll: Polls a condition until it passes or a timeout is reached, useful for dynamic content.
• expect.toPass: Runs a block of assertions until all pass, often used for testing flaky or complex scenarios.

Soft assertions are helpful for tests where understanding multiple failures is beneficial, such as end-to-end tests with a wide range of checks. Hard assertions are more suitable for critical conditions where a single failure should immediately terminate the test.

Follow these steps to get up and running with Playwright Assertions:

1. Install Playwright:npm init playwright@latest.
2. Start writing tests with various assertion types to validate application behavior.

Custom matchers can be added using expect.extend, allowing developers to define more readable or application-specific assertions.

Here’s a quick example of how to create a custom matcher for checking specific conditions:

expect.extend({
  toBeWithinRange(received, floor, ceiling) {
    const pass = received >= floor && received <= ceiling;
    if (pass) {
      return {
        message: () => `expected ${received} not to be within range ${floor} - ${ceiling}`,
        pass: true,
      };
    } else {
      return {
        message: () => `expected ${received} to be within range ${floor} - ${ceiling}`,
        pass: false,
      };
    }
  },
});

// Usage
expect(100).toBeWithinRange(90, 110);

This can be especially useful if we’re doing complex assertions, or specialized parsing of responses.

Understanding how to diagnose assertion failures can save significant debugging time.

Hard waits should be avoided. A ‘hard wait’ refers to giving an exact period of time before going on to the next step of a test. Generally using code like:

await page.waitForTimeout(3000);

When you have page components you want to ensure are loading quickly, or you have page components that you know take some time to load, it’s tempting to reach for a hard wait, however, there are better ways to accomplish this functionality, like the soft assertions and auto-retry assertions listed above.

After that fixed period, if the next step doesn’t work, the test will fail. Hard waits are the most common cause of unreliable or ‘flaky’ tests with Playwright. They’re also inefficient, if the element we’re checking for becomes available before the hard wait time has elapsed, the test runner still has to wait to the end of the wait time. Read more about this in our page on waiting in Playwright.

While a hard wait is problematic since it doesn’t benefit from Playwright’s auto-waiting, an improperly structured assertion doesn’t include waiting at all. By wrapping the await inside the expect() instead of vice versa, and by using a non-web-first assertion, we get no waiting. In this case we’re checking that there is some text inside an element.

// expects the text to be visible instantly
expect(await targetPage.locatorToSomeElement.innerText()).toBeTruthy();

Use web-first assertions such as toBeEmpty() instead.

// uses auto-waiting, defaulting to five seconds max
await expect(targetPage.locatorToSomeElement).not.toBeEmpty();

Playwright provides debug tools such as page.pause() and verbose logging to inspect what happens when a test fails. You might also consider a visual like the Playwright VS Code extension to add a visual debugger when writing or debugging tests. Read more about debugging Playwright scripts.

Flaky tests are those that fail intermittently, often due to timing issues or unpredictable environment factors. You can use expect.poll and .toPass adjust timeouts to stabilize flaky tests. To identify flaky tests you can use --repeat-each=100 to really hammer a test and see if it passes constantly.

Implementing best practices can make your tests more reliable and easier to maintain.

• Test Granularity: Structure tests to cover distinct, isolated behaviors, making debugging easier. The choice between one ‘super test’ and 5 or 10 smaller tests should always go with the more granular option. Since tests can be run in parallel, it’s likely that better test granularity will also improve test execution time.
• Create Concise Statements: Keep assertions simple and focused. Asserting that five different page components is an indicator that you’re testing multiple ideas with one test, and want to consider either breaking up your test or should at least have multiple steps.
• Use Descriptive Messages: Adding clear messages to assertions helps identify failures quickly. It’s also good to use test steps to add more description to the exact phase where the test failed. Read further to see how to add test steps programatically with Typescript decorators.
• Parameterize Assertions: Use variables for assertion parameters to improve test readability and maintainability.
• Leverage Libraries: Utilize existing libraries for common assertions where possible. Note that if you’re running tests on a cadence, a full set of libraries may not be available in your execution environment.
• Be Specific: Make sure your assertions are as targeted as possible to avoid false positives. Bet on auto-waiting and web-first assertions. Even a change from checking if a text label has the needed text with .toContain() rather than toBe() can improve the reliability of a page monitor.
• Leverage Built-in Timeout: Use Playwright’s built-in timeout features to handle flaky conditions gracefully. Avoid hard waits whenever possible.

Checkly offers tools for continuous monitoring using Playwright scripts. With Checkly, you use the power of Playwright to effectively simulate a real user’s behavior with your site or service, and a can get alerts of problems before your users have noticed.

By using the right types of assertions, debugging tools, and best practices, you can ensure your tests are robust and efficient. In general, my direct experience has served to emphasize the wisdom of the Playwright project’s recommendations: web first assertions, dependent on web-first selectors.

For assertions, web-first, auto-retrying assertions are the preferred choice for most scenarios, as they leverage Playwright’s built-in waiting mechanisms, making tests more resilient and less prone to flakiness. These assertions continuously check conditions until they pass or reach a timeout, which can help streamline test execution. Examples like toBeVisible or toContainText illustrate how these assertions can improve test stability. On the other hand, non-retrying assertions evaluate conditions just once and should be reserved for scenarios where retries are unnecessary or could introduce ambiguity. Anything other than web-first assertions will lead to brittle tests with more false positives.

Soft assertions also add value by collecting all failures within a block before stopping the test, offering a complete picture of what went wrong. They are especially useful in production monitoring scenarios where partial failures should not disrupt the entire check, as seen in the Checkly example using markCheckAsDegraded. The ability to use soft assertions alongside the concept of a degraded state provides a nuanced way to monitor service health without causing false alarms.

I’ll again recommend avoiding some common pitfalls, such as relying on hard waits, which can introduce flakiness and inefficiency into your tests. Using auto-waiting mechanisms and structuring assertions properly can make a significant difference. Debugging tools like page.pause() and verbose logging, combined with visual debugging options, can be valuable in diagnosing test failures.

Follow best practices now for a better testing and monitoring experience later. These include using concise, focused assertions, adding descriptive messages for better debugging, and leveraging built-in timeouts and parameterization for clarity and maintainability. Granular test design also matters as a way to improve both parallel execution efficiency and debugging ease. Avoiding complex, monolithic tests in favor of smaller, well-defined ones can make your testing framework more manageable and robust.