Integrating Accessibility Checks in Playwright Tests with Checkly

Ensuring your web application is accessible is not just about compliance; it's about inclusivity. Tools like Google Chrome's Lighthouse provide a starting point for accessibility checks. but integrating these checks into your development workflow can significantly enhance the quality of your product. This post explores how to perform automated accessibility checks using Playwright and Checkly, leveraging the power of the axe-core library.

Understanding the Foundation

Lighthouse uses the axe-core library for its accessibility audits, a powerful tool for identifying accessibility issues. By inspecting the accessibility violation details provided by Lighthouse, developers can gain insights into specific issues, such as elements not following a sequentially descending order.

What is Axe?

Axe is a comprehensive accessibility testing engine designed for websites and HTML-based user interfaces. By facilitating the automation of accessibility testing alongside routine functional testing, Axe ensures that accessibility becomes an integral part of the development process rather than an afterthought.

The foundational belief behind Axe is that the web should be an accessible and inclusive space for everyone. This vision can only be realized when developers are equipped and encouraged to take proactive steps towards implementing accessibility testing and adopting accessible coding practices from the outset.

Automated accessibility testing empowers each dev to take responsibility for most accessibility issues without needing an outer loop of testing. It eliminates the need for specialized expertise in the initial stages of testing, thereby allowing development teams to allocate expert resources more efficiently, focusing on addressing complex accessibility issues that automated testing cannot resolve.

It’s important to note that Axe doesn’t intend for its engine to be used to replace human testing for accessibility. Automated testing isn’t able to tell you that you have zero accessibility issues on your page. By embedding accessibility testing into the early stages of development, Axe helps in identifying and mitigating accessibility issues when they are easiest and least expensive to fix. This proactive approach not only saves time and resources but also significantly reduces frustration among development teams.

Setting Up Playwright with axe-core

To integrate axe-core into Playwright tests, you first need to understand how to implement axe-core checks manually. This involves including the axe.min.js library in your project and utilizing the run function to execute accessibility checks. However, a more streamlined approach involves using the axe-core/playwright package, which abstracts the low-level axe calls and ensures checks run within iframes.

Configuring Playwright Tests for Accessibility

Implementing accessibility checks in Playwright tests is straightforward with the axe-core/playwright package. After adding the package to your project, you can use the AxeBuilder class to initialize and run the accessibility checks. The process involves creating an instance of AxeBuilder, passing the Playwright page object, and calling the analyze function to execute the checks. The results, which include details about any violations, can then be logged or asserted within your tests.

The following check expects Axe’s list of accessibility violations to have a length of zero:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('homepage', () => {
  test('should not have any automatically detectable accessibility issues', async ({ page }) => {
    await page.goto('https://www.checklyhq.com/welcome');

    const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); 

    expect(accessibilityScanResults.violations).toHaveLength(0);
  });
});

Enhancing Test Readability and Configuration

Initially, a large number of calls will show up as top-level steps from the test code above:

To improve the readability of your test reports, consider grouping low-level page.evaluate calls under a test step. This approach not only makes the reports more readable but also allows for better organization of your tests.

test.describe('homepage', () => {
  test('should not have any automatically detectable accessibility issues', async ({ page }) => {
    await page.goto('https://www.checklyhq.com/welcome');
    await test.step('check page accessibility', async () => {
      const accessibilityScanResults = await new AxeBuilder({ page }).analyze();
      expect(accessibilityScanResults.violations).toHaveLength(0);
    })
  });
});

The result is a much cleaner report with the calls nested under a readable step name:

Additionally, configuring the AxeBuilder to match Lighthouse's settings—focusing on specific WCAG standards and rules—ensures consistency between your Playwright tests and Lighthouse reports.

Integrating with Checkly for Synthetic Monitoring

With Checkly's support for axe-core/playwright in our 2024.02 runtime, you can extend accessibility checks to synthetic monitoring. This integration allows for continuous monitoring of your web application's accessibility, ensuring that new or existing content does not introduce accessibility issues.

If you’ve been using Checkly for a while, you’ll want to update your runtime to make sure these checks are supported. See ‘how runtimes work’ in our docs site. If you’re running an older version of the Checkly runtime, the accessibility check code above will get you an INFO - Error: No tests found message.

Conclusion — Accessibility means ongoing work.

Integrating accessibility checks into your Playwright tests and synthetic monitoring with Checkly is a powerful strategy for maintaining high standards of web accessibility. By leveraging axe-core and the axe-core/playwright package, developers can automate the detection of accessibility issues, ensuring their web applications are inclusive and compliant. Remember, accessibility is not a one-time task, but an ongoing commitment to delivering quality web experiences for all users.

I'll also emphasize again that the purpose of automated accessibility checks is not to replace human a11y auditors, but rather to shorten feedback time for easy-to-detect issues and free up human auditors to find more complex issues.

To see accessibility checks in action, check out Stefan's demonstration video:

Join the Checkly Community

Are you interested in continually monitoring your site's accessibility? Have you implemented checks with tools other than Playwright and Axe? We'd love to hear about in the Checkly community slack, join us and say hi!