BlogAdvent of Monitoring, Day 5
(Updated: )

The Advent of Monitoring, Day 5: Dealing With Third-Party Dependencies Causing False Positives for Synthetics

Javier Pérez Anaya

Frontend Engineer

Synthetic Monitoring
Playwright

Share on social

This is the fifth part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

When we’re testing our apps, it's a big headache to simulate what the user goes through while steering clear of the more problematic parts of those processes. These parts, often external and beyond our control and responsibility, are usually not the focus of our testing.

Think external services, third-party modules, or APIs. Relying on these unpredictable elements for our tests is a no-go. Nor do we want to rework our tests to check internal implementations just to dodge these issues. It’s a well-known fact that this leads to fragile, high-maintenance tests.

So, how do we tackle this challenge?

We’ll adopt a user-centric testing approach. For unit tests, this means faking or mocking our API calls or libraries. This is something you can easily do with modern testing tools like Jest or Vitest. But then, there’s the question of end-to-end (E2E) testing and monitoring our live code or feature branches.

Let’s use our own situation as a case study.

Right now, we’ve got about 40 checks running on both monitoring and feature branches. Most are solid, but we hit a snag: several checks were acting up intermittently. After digging around, we realized that some of our marketing tools were quietly experiencing outages or slowdowns. This was causing our app’s load times to be unpredictable, or, in some cases, they wouldn’t load at all.

We weighed two options:

  1. Implement a special query parameter to bypass the loading of marketing libraries.
  2. Adopt Playwright test route mocking.

Eventually, we decided to go with option 2, as it mirrored what users actually experience in our app.

Having settled on route mocking, we traced the problem to a specific pixel from a well-known job networking site failing to load. Initially, we contemplated solving the immediate issue by mocking this specific request. However, we ultimately chose to preemptively avoid potential external request failures by mocking them all. We achieved this by using regex matching during the page load phase of our checks.

We declared a few regex for all the libraries that might pop up during the app load:

const HOTJAR_REGEX = /https:\/\/(.*)hotjar\.(com|io)(.*)/i
const FEATUREBASE_REGEX = /https:\/\/(.*)featurebase\.app(.*)/i
const GOOGLE_REGEX = /https:\/\/(.*)(googletagmanager\.com|google\.com\/pagead|googleadds\.g\.doubleclick\.net)(.*)/i
const INTERCOM_REGEX = /https:\/\/(.*)(intercomcdn\.com|intercom\.io)(.*)/i
const LINKEDIN_REGEX = /https:\/\/(.*)(licdn\.com|px\.ads\.linkedin\.com)(.*)/i
const REDDIT_REGEX = /https:\/\/(.*)(alb\.reddit\.com|redditstatic\.com\/ads)(.*)/i
const TWITTER_REGEX = /https:\/\/(.*)(ads-twitter\.com|t\.co|analytics\.twitter\.com)(.*)/i

const thirdPartyUrls = [
  HOTJAR_REGEX,
  FEATUREBASE_REGEX,
  GOOGLE_REGEX,
  INTERCOM_REGEX,
  LINKEDIN_REGEX,
  REDDIT_REGEX,
  TWITTER_REGEX,
]

Then, we set up the route interception to avoid loading the problematic bits:

await this.page.route('**/*', (route) => {
  if (thirdPartyUrls.some(regex => regex.test(route.request().url()))) {
    route.fulfill({
      status: 200,
    })
  } else {
    route.continue()
  }
})

It’s important to remember that this method isn’t perfect. It does need some upkeep, particularly in terms of keeping the regex expressions up-to-date. However, as long as the external libraries in the app remain relatively unchanged, this maintenance effort is pretty manageable.

Ever since we rolled out this fix a few months ago, the change has been night and day. We went from occasional false positives to having a spotless record. A big shoutout to Playwright and its versatile mocking capabilities!

Before we wrap up, there’s one hiccup worth mentioning: the route.continue() and route.fulfill() steps end up in the Playwright report as actual test steps. This clutters the report with extra details that don’t really add much value. We’re still brainstorming ways to tackle this little snag, which has been reported to Playwright team a few times, so stay tuned for the part II of this story!

Share on social