Changing assertions for API monitoring checks

Table of contents

What is changing?
This is how it will work
This will break
This will all not break
What should I do?

Update: this has been released as per 13:00 CET on 11 November. We delayed it a week due to some rescheduling of other items on our roadmap.

This post is to give you a heads up of a change we will release on Tuesday November 3rd that could impact a tiny percentage of your Checkly API monitoring.

Take 5 minutes and read this through. There is a (very) large chance this will not impact you, but we are being extra careful here.

What is changing?

In a nutshell:

API check assertions on JSON bodies will apply to *each element* in a response array instead of to the array as a whole. This enables using JSON path expressions to check all array values.

If you are not asserting JSON bodies, please skip all of this and enjoy your day.
If you are confused, let's look at some examples...

This is how it will work

Just to be 100% clear: this describes the new / upcoming situation. Let's say your API returns a JSON array as a response, i.e.

[
  {
	"customerId": 1234,
	"name": "John Doe"
  },
  {
	"customerId": 5678,
	"name": "Kate Doe"	
  }
]

You want to make sure that each customer name has the word "Doe" in it. You can now check this with the $[*] JSON path operator and a simple Contains comparison

This will check if each name attribute in the array will contain the word "Doe". In this case the assertion will pass ✅

Let's extend the example with the active boolean attribute.

[
   {
      "customerId":1234,
      "name":"John Doe",
      "active":true
   },
   {
      "customerId":5678,
      "name":"Kate Doe",
      "active":false
   }
]

Say we want to assert that every customer in the array is active with the following assertion:

Again, this will check if each active attribute in the array is true and will therefore fail ❌ because Kate Doe is not active.

This will break

We've identified one specific corner case that will break under this new regime. Look at the following payload. We've added the tags array to each customer.

[
   {
      "customerId":1234,
      "name":"John Doe",
      "tags": ["one_tag", "another_one"]
   },
   {
      "customerId":5678,
      "name":"Kate Doe",
      "tags": ["i_love_tags"]
   }
]

We want to check if the word "tag" is part of each tags array. We use the following assertion:

In the current / old situation, this assertion will pass ✅ as JSON path will yield the string [one_tag, another_one, i_love_tags] and clearly the word tag is in there. Twice even!
In the upcoming / new situation, this will fail ❌ as we check each item in this array separately and clearly the word another_one does not contain the word tag.

If you have such an assertion, I would advise to just remove it now and set a short reminder for yourself to add a similar one after 2 November.

This will all not break

JSON responses with arrays can already be asserted in many ways using JSON path queries. All of the examples below will not break.

The reason is that the JSON path queries below all yield exactly one item from the array. The moment our comparison operator comes into play, we are not "looking at" an array anymore.

Targeting a specific item by index

In the first example below we check if the first item in our result array has a property title:

In the next example we pick the last item in the array and check if the customerId property has the value 123abc:

In this example we pick the item with index value 4. This is the 5th item as array indexes start at 0. We then assert that the responseTime property is less than 2000.

Targeting the length of an array

In the last example we check if the returned array has more than 10 items.

What should I do?

Check if you are using API checks that assert JSON response arrays "as a whole".
If so, just remove them for now and set a short reminder to add a similar assertions after 2 November.

In the worst case, you will be alerted of a failing check on 2 November and the result will clearly state what is wrong.

For any follow up questions, don't hesitate to reach out to support using the chat bubble!