Manel Fourati

This is the seventh part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

At Checkly, we manage various scheduled jobs, some of which play a crucial role in our application's functionality, and others exist to support different teams within Checkly.

In this article, we'll break down how we use our own Heartbeat monitoring feature to monitor these scheduled jobs and catch issues before they reach out users. 

We maintain a group of jobs that serve different purposes. Two example categories are:

Infrastructure Jobs (Essential for the App)

Sending Emails to Customers: Notifications for quota warnings, expired plans, weekly digests, etc.

Database Cleanup: Tasks such as revoking support access or removing unused domains of deleted dashboards.

Managing Database Partitions: Ensuring optimized and efficient database organization.

To empower the management and sales teams to monitor feature performance and make data-driven decisions, we've built a data lake that combines different data for ease of querying and access. To keep this data lake up-to-date, we created jobs that fetch information from different sources, including internal databases and third-party services.

Frequently fetching data from write-intensive tables.

Daily scheduled data imports from Stripe.

Data transformation jobs that aggregate and insert data into different tables.

Before implementing a monitoring solution, we would occasionally receive reports from the affected teams giving notice that the data is unavailable, inaccurate, or incomplete. This could potentially lead to decisions based on outdated information.

Therefore, it became evident that we needed a way to monitor these jobs in order to mitigate any potential issues as soon as a job run fails.

Upon reaching the beta phase of the Heartbeat feature, we implemented checks to keep tabs on jobs that were previously not monitored. We also migrated some previously monitored jobs with Healthchecks.io to use Heartbeats as a unified service.

With Monitoring as Code, we created Heartbeat checks with the Checkly CLI. An example of these checks looks like:

The alert channel, period and grace units are chosen depending on the job frequency and urgency.

Once checks are deployed, we receive a ping URL to call after each job run. For backend jobs on Heroku, we update the Run Command, including a Curl to the ping URL. For data lake jobs using AWS Glue, Python scripts are updated to make a simple request to the ping URL.

Once the check is set up, any missing pings (therefore failed job runs) on the expected interval will trigger a notification to the specified alert channel.


This is an example of an alert received on our data lake Slack channel when a Glue job failed.

By implementing Heartbeat checks to monitor our CRON jobs, we are now able to promptly handle any failed job runs, making sure our systems run as expected and all teams can work with up-to-date, trustworthy data.

To learn more, check our documentation or blog post.

The Advent of Monitoring, Day 7: A Peek Into Our Job Monitoring Strategy With Heartbeat Checks

Detect

Communicate

Resolve

Monitoring as Code

Developers

Resources

Community

The Advent of Monitoring, Day 7: A Peek Into Our Job Monitoring Strategy With Heartbeat Checks