Writing Your First Visual Regression Check in Playwright

Visual regression testing ensures that your web application looks as expected and that any visual changes are intentional. These tests amount to comparing two screenshots and looking for pixels that are different. With Playwright, you can achieve this with just a few lines of JavaScript. Let's walk through the process using a simple example. Once we’ve done a visual regression test start to finish in Playwright, we’ll show how you can add Checkly tools to create visual regression monitors.

Setting Up Playwright

First, you'll need to set up your Playwright environment. Create a screenshot.spec file and require the necessary modules from Playwright's test runner. This includes test and expect, which you'll use to define and run your tests.

Writing the Test Case

Here's a step-by-step guide to writing a test case that checks the visual integrity of a web page, specifically a page about raccoons:

1 — Define the Test Case: Start by defining an asynchronous test function. This function will contain the steps to navigate to the page and take a screenshot.

import { test, expect } from '@playwright/test';

test('raccoon page looks good', async ({ page }) => {
    await page.goto('<http://localhost>');
    const screenshot = await page.screenshot();
    expect(screenshot).toMatchSnapshot('raccoon.png');
});

2 — Run the Test: Open your terminal and run the test using the command:

npx playwright test

Initially, the test will fail because there is no baseline snapshot to compare against

3 — Create a Snapshot: To create a baseline snapshot, update the snapshots in your terminal:

npx playwright test --update-snapshots

This will create the raccoon.png snapshot file. Run the test again to ensure it passes:

npx playwright test

Introducing Visual Changes

To see how Playwright handles visual regressions, make a visual change to your test page. For example, modify some CSS or HTML elements to alter the appearance of the page.

Detecting Screenshot Changes

Run the test again after making the changes:

npx playwright test

The test should now fail, indicating that there was a visual regression. Playwright will generate a test results image highlighting the changed pixels, making it easy to identify what has changed.

To see this code in action, check out Stefan’s video below:

Considerations with visual regression testing

While visual regression tests can be enormously helpful for finding pixel-level changes to your site. This can help find subtle problems like font rendering, spacing changes and other hard-to-see problems. Visual regression testing can also detect problems where the interface looks quite different visually despite having all the same elements. A few considerations when using these tests:

Configure thresholds for failure

The maxDiffPixels setting lets you decide how far the current snapshot can vary from the baseline.

import { test, expect } from '@playwright/test';

test('example test', async ({ page }) => {
  await page.goto('https://playwright.dev');
  await expect(page).toHaveScreenshot({ maxDiffPixels: 100 });
});

You can also configure this setting globally in your playwright config file. See all the option for visual comparison in the Playwright documentation for snapshot tests.

Screenshot tests aren’t portable!

Slight changes in the rendering engine of your browser will cause visual shifts, so your playwright visual regression tests will fail when run on another machine. For long-term use it’s best to run these tests from a virtual machine with a standard container image. If you’re looking at using visual regression checks at a larger scale, consider using Checkly’s visual regression monitoring tools with Playwright.

Going Further with Checkly and Screenshot Monitoring

For continuous monitoring of your site, with alerts when visual comparison fails and easy viewer to show visual differences, considering running your Playwright code on Checkly.

Viewing the results when a visual regression check on Checkly fails.

With all Playwright checks running from Checkly-managed virtual machines, there’s no issue with rendering differences between check runs.

Since Checkly fully supports Playwright, the process of writing page monitors is just like writing Playwright to run off your own workstation.

Even better, with the Checkly CLI your whole team can run, deploy, and manage your monitors as part of a shared repository right next to your application code. This Monitoring as Code (MaC) model means everyone can take part in making sure your service is reliable and surprise-free.

To see more about how to use Checkly for visual regression monitoring, check out our documentation.