How (not) to test signup and keep your CEO happy

One of Checkly's strengths is the capability to monitor key transactions on your site. It'd be missed opportunity if we didn't reuse it to monitor our own product! But for some important flows that comes with a couple of pitfalls. In this post, we'll take a closer look at how we monitor one of our top key flows: signup.

The signup flow

First, a quick overview of the steps we want to perform. Here we are looking at a user signing up using email and password.

The user should:

1. Go to the signup page
2. Fill in the email/password fields
3. Click Signup

In addition, we'll want to remove the created account:

1. Go to the account deletion section of the settings page
2. Click the delete account button
3. Type in the account name to confirm the intent
4. Click the delete account button in the modal window

This is how the full script will look like when using Playwright:

const { chromium } = require('playwright');
const expect = require('expect')
const uuid = require('uuid')

// We need to construct a unique user for each test run. This is done by generating an id and appending to the email 
const id = uuid.v1()

const browser = await chromium.launch()
const page = await browser.newPage()

await page.setViewportSize({ width: 1200, height: 1000 })
await page.goto(process.env.ENVIRONMENT_URL || 'https://app.checklyhq.com?utm_source=monitoring', { 'waitUntil': 'networkidle' });

await page.click('.auth0-lock-tabs a')

// Fill in the email, password. Click the Signup button 
await page.click('[aria-label="Email"]')
await page.type('[aria-label="Email"]', `racoon+${id}@checklyhq.com`)

await page.click('[aria-label="Password"]')
await page.type('[aria-label="Password"]', id)

await page.waitForSelector('[aria-label="Sign Up"]')

await Promise.all([
  page.click('[aria-label="Sign Up"]'),
  page.waitForNavigation()
])

// The user would land on the onboarding flow. We need to skip that
await page.click('[data-test-id="onboarding-skip"]')

// At this point the user is signed up. Now we need to delete the account. We are navigating to the account deletion section.
await page.waitForSelector('[data-test-id="navbard-menu-trigger"]')
try { await page.click('[data-test-id="onboarding-nudger-close"]') } catch (e) { }
await page.click('[data-test-id="navbard-menu-trigger"]')

await page.click('[href="/settings/account/general"]')

await page.click('[data-test-id="delete-account-button"]')

// Here, we need to confirm the account deletion by typing the account name (which is the user email by default)
await page.click('[data-test-id="delete-account-name-input"]')
await page.type('[data-test-id="delete-account-name-input"]', `racoon+${id}@checklyhq.com`)

await page.waitForSelector('[data-test-id="delete-account-modal-btn"]')

await page.click('[data-test-id="delete-account-modal-btn"]')

// Waiting for the account deletion confirmation text... 
const title = await page.textContent('[data-test-id="account-deleted-title"]');

await expect(title).toEqual('Thanks for using Checkly. Goodbye 🖖')

await page.waitForNavigation()
await page.waitForNavigation()

const targetUrl = await page.url()

expect(targetUrl).toEqual('https://www.checklyhq.com/')
await browser.close()

Keep the data clean!

Alright, we've created our check and tested it on the staging environment. Everything works great so we enable it for production, high-five ourselves for doing a great job, and move on to other tasks. A couple of hours pass, and suddenly we notice a little message from our CEO on Slack. Oops!

We at Checkly care a lot about product and marketing metrics. We have more than 450 paying customers and thousands of engineering teams who depend on us getting things right. So interpreting data, KPIs and input from various channels like our public roadmap is vital in keeping us connected to our users (so is talking to customers, one of our core values). Essentially, all this input comes together to inform us and enable us to identify and prioritize the next steps and future initiatives. Knowing this, it is no surprise that clean data is mission-critical.

Filtering out the analytics tools

Hannes' message reminded us we also need to filter out the check from the tracking tools. To do that, we'll need to alter the application code. This will vary depending on the choice of the analytics tools, but this is how we do it in our Vue frontend app where we use Segment as the events broker.

function isInternalEmail (email = '') {
  const domain = email.split('@')[1]
  return /checklyhq/.test(domain)
}
export const identify = (user) => {
  if (isInternalEmail(user?.email)) {
    return
  }
  const { id, email, name, created_at: createdAt } = user
  window.analytics.identify(id, {
    name,
    email,
    createdAt,
  })
}

As you can see, we just skip the call to the analytics when the user is in our email domain. Other analytics methods (page, group, track in this case) are wrapped in a similar fashion.

There's one more thing. If you look at the URL we target with the E2E check, you'll notice it has the utm_source="monitoring" query param added. This will allow you to filter out traffic from your analytics solution of choice, like Google Analytics or Plausible. You can read more about it in this guide.

Tweaking to your needs

As you just saw, setting up proper E2E monitoring of the important flows is more about making sure that you don't mess up the Really Vital Metrics™ or flood your DB with ghost accounts.

What we described here is not a one-size-fits-all solution of course:

• You may find out that your check takes too long and times out. If that's the case you could think about having a check for an account creation where the account deletion is handled via an API call (axios is available in the script runtime).
• In case there are several checks that would use the account creation flow, you could think of introducing reusability with code snippets.

We hope this little guide will help you get set up monitoring your signups... without getting the CEO involved!

Happy E2E monitoring!