Tim Nolet

Today, a customer reported that on public dashboards, the contents of any browser check script are part of the API response that supplies the dashboard with data.

Admittedly, this was by design not due to a bug or hack: In our thinking, browser check scripts should not contain sensitive data. Instead, secrets should be stored in the environment variables which are encrypted on disk and in flight.

However, this design was too naïve and did not take into account that users might "hard code" sensitive data in their scripts. Also, the script data did not add anything to the dashboards and should have been scrubbed in the first place.

For this reason, we consider this part of the security incident as in both cases unauthorized users could access this data.

If you showed browser checks on a public dashboard and you stored any confidential information directly in your browser check script, please take the following steps:

Rotate or change any passwords, block users or take any other measure to ensure the secrets used in your browser scripts are no longer valuable.

Remove the secrets in your script and stored them as global environment variables or environment variables directly in your browser check.

Please see the relevant docs on our docs site for help on how to achieve this.

Similar to the parent incident, we have no reason to believe any check data was actually leaked.

No user data, emails, passwords or other Personally Identifiable Information was compromised.

No credit card data or other payment information was compromised.

The mitigating steps to not have similar issues in the future are still relevant and listed below in the What now? paragraph.

Today, from 16:17 till 18:21, users were able to see other users' disabled checks in their dashboard when setting a specific dashboard filter combination.

When inspecting the JSON payload, they could have read specific parts of check data. Specifically, if a browser check script contained any non-encrypted user credentials for test users, these might have been compromised.

No user data, emails, passwords or other Personally Identifiable Information was compromised. No credit card data or other payment information was compromised. No encrypted variables, headers or other check fields were compromised.

We have emailed all users that have been impacted purely as a precaution. We have no indication any data was actually leaked.

At 17:30 CET I was made aware of security flaw in Checkly. This flaw allowed users to see disabled checks in their dashboard when configuring the dashboard filters to show disabled checks and non-disabled checks.

From 17:30 to roughly 17:55 I chatted on Intercom with the user reporting the issue to pin point the origin.

At first I suspected some caching issue with our authentication provider Auth0, but I quickly discarded this as the behavior was 100% reproducible on a local dev machine.

At 18:00 the issue became clear. It was a logic bug in the endpoint that servers the dashboard. Each query in fired on the database is accompanied by an accountId.

This accountId determines which checks belong to which account. However, in one part of the query — where we determine the filter for disabled and other checks — a wrongly nested OR statement effectively negated the crucial WHERE accountId clause.

A patch was made, tested and rolled out. Our deployment log shows it went live at 18:21 CET. Both the reporter and I confirmed the patch correctly addressed the issue.

This issue has the following impact characteristics

It was live from 16:17 to 18:21 CET on 29-10-2020.

It only impacted one API endpoint that populated the dashboard.

It could only leak information on disabled checks, specifically the browser check scripts which contain free form data. Users might enter test credentials there, although Checkly has many features to protect these credentials and store them separately and encrypted.

Only authenticated users could potentially see this data.

No account data was impacted. No username, email addresses, passwords, credit card data etc. Checkly does not store any of these types of data. We use 3rd party services like Stripe and Auth0 for this.

I have an exact overview of which checks for which account potentially could have been visible. I've emailed these users and advised them to take action.

The bug was pretty terrible because it was a logic bug. Not a hack or undocumented feature. This has an upside and a downside.

The upside: The fix was trivial. Our authentication is not broken as this
The downside: It shows we need to integrate authentication, accounts and how accounts interact with endpoints in a more robust way.

For this reason, I'll take the following actions

Move any and all account "pinning" outside of the general application logic. This way any queries or other code cannot interfere with "the accounting" take place on another abstraction level.

Do a security assessment with a pro security consultant / ethical hacker.

Move as quickly as possible to a code review practice.

Keep lines open for security bugs using a bug bounty program.

For any questions, please reach out to me on tim [at] checklyhq.com

Post mortem: Checkly security incident [updated]

Detect

Communicate

Resolve

Monitoring as Code

Developers

Resources

Community

Post mortem: Checkly security incident [updated]