Nočnica Mellifera

How much do Synthetics matter to your team? I think they matter a whole lot.

Back when I was a freelance developer, I doubled my annual income with synthetics. Working mainly in database optimizations, I would finish out a contract and leave a synthetic monitor running at a very low frequency on their service. When I saw a pattern of slower performance, I knew it was time to hit the team lead-up to ask if I could help. I only reached out when my work was needed, and booked more contracts as a result.

What amazes me about synthetics is how teams of every size benefit from them: From the huge enterprises that need some way to test end-to-end from around the world, to the fractional CTO monitoring more than one tiny project.

Synthetics are critical to almost every team. You need to monitor your systems from outside because it’s the only way to truly know whether your service is available for your users. But every team worth its salt also thinks about the costs of any new tool. Why, then, do so many SaaS APM providers price synthetics like it’s an expensive add-on rather than a core part of Observability?

I’d like to talk about how Datadog has chosen to price synthetics so high that you can’t really do the kind of synthetic coverage you need to be effective.

Datadog proudly claims that you only pay $12 for 1,000 synthetic browser checks. And that sounds like a lot of checks for just twelve dollars! Even in my worst weeks as a dev, I don’t feel like I went to the production site and clicked around to test something more than 1,000 times in a single week. But let’s break down that number and see what it might really cost us. There’s probably not one single URL that our business rests on, and once we start adding a few more, the number can quickly grow. But giving ourselves 4 routes for our marketing site, 8 for basic user actions, and another 4 for signup and other funnel activities, we’ll say we’ve got 16 total to keep track of.

Since we’ve all gotten the dreaded incident where “the site is down but only in Europe,” we know we can’t check from just one region, but again, we want conservative numbers, so 4 should be plenty.

So from that comfortable $12, we’re now spending more than a team member’s salary on synthetic checks, and this is really the bare minimum.

Right now, this checks some basic user actions but doesn’t try valid and invalid inputs. It doesn’t simulate page errors or check the performance of large requests vs. small requests (e.g., loading a friends list of 10,000 users vs. a friends list of 80). Any of these additional requirements can double your synthetics bill.

Oh, and one last piece of bad news about Datadog’s 1,000 checks for $12. That’s their lowest price, not what you’ll actually pay. $12 is the minimum and includes an annual commitment. For monthly or pay-as-you-go plans, you’ll spend $15–18. That means you’ll be spending more than $12,000 a month just to know if parts of your site are available or not.

The result, inevitably, will be that Operations will end up telling all other teams, ‘Don’t add any more synthetic checks; we have to remove some before we can monitor more.’ And the risk increases because your users will be the first to know about an outage.

Back in the day, synthetic checks were called ‘pingers’ and were an often-overlooked part of monitoring. In those days, a basic monitor just checked for a 200 status, and a more advanced one could do string matching. Clever SREs configured their pingers to match text that loaded last on a page, giving them some insight into how the site was performing. An API check, which is much cheaper from Datadog than the browser checks mentioned above ($5 for 10,000 checks with an annual commitment), is essentially one of those more basic pingers. They’re not really a substitute for full browser checks.

With Checkly, our fully-featured playwright tests let you check page functions, take actions, and test behavior based on programmatic values. You can even monitor the visual appearance of your web app with snapshot testing. None of that works with an API check like that offered by Datadog.

In Datadog’s ideal use case, you’d use an API check to ping every single API endpoint, of which there will be any from a few dozen to several hundred. To do things right, you should also send some invalid requests or otherwise check the endpoint’s response to multiple inputs. Suddenly, Datadog’s generous $5 per 10k runs seems less appealing. 

Again, this $5 per 10,000 price is for an annual commitment, so you’ll be locked into spending over $66,000 annually for API monitoring.

These prices are not so large that a medium-sized team should balk, but obviously, a startup can’t afford $60k per year for just one type of monitoring. It also means that even with basic monitoring, your budget will be sorely stretched.

Two ways to drastically improve your Synthetics bill

Look, I’m not going to write a whole paragraph when a sentence will do: Checkly’s pricing is roughly half that of Datadog. Step one to paying less is to pay less. Simple as.

At Checkly we started with synthetics as the core tool for observability. The promise of a system totally independent of your own stack, that runs whether or not your router, queues, and every ounce of computing are on fire, is the future of observability. As such, our tooling reflects how fully we’re invested in synthetics.

Unlike Datadog, we have full Playwright support, capture trace files and videos from our testing sessions, and even have a Playwright Ambassador on the team. We also let you create tests from the CLI, next to your application code in your repository, or with our Web UI. How can better tooling for tests lower your costs? By increasing the chances that you’re performing the right tests in the first place.

With Checkly’s flexible system for writing, deploying, and running tests you can put the power of synthetics in the hands of the people who rely on these tests the most - and deploy them from their code repository. Front-end-focused devs can write TypeScript tests comfortably, while backend developers can write and deploy tests without ever leaving their trusted CLI. Finally, team leads, sales, and everyone else can see results within the Checkly UI and even create code-free tests as needed.

By democratizing access to testing, you’re lessening the need to ‘just test everything in every way' and increasing the chance that every test will monitor a critical issue.

Conclusions: Synthetics matter; do them right

Synthetics monitoring plays a pivotal role in ensuring the reliability and performance of your applications, regardless of your team's size or the scale of your projects.

As demonstrated, the cost of synthetics monitoring can quickly add up, especially with SaaS APM providers that price synthetics as an expensive add-on. Datadog, for example, may initially seem affordable at $12 for 1,000 synthetic browser checks. However, when you consider real-world requirements, such as monitoring multiple routes, checking from different regions, and ensuring frequent checks to detect issues promptly, the costs can escalate significantly.

Synthetic monitoring goes beyond simple API checks, especially regarding browser checks. It involves simulating user actions, testing behavior, and validating inputs, all of which contribute to a comprehensive understanding of your application's health. Datadog's pricing for API checks may appear attractive, but when you factor in the need to cover numerous endpoints with various inputs, the expenses can quickly mount.

Synthetics monitoring is not just a nice-to-have; it's a critical component of ensuring your applications perform optimally. By embracing cost-effective solutions and implementing comprehensive testing strategies, you can harness the power of synthetic monitoring to enhance your team's observability and deliver a better experience to your users.

I hope you’re inspired to join the Checkly Slack and talk about how you can go from ‘no one in Germany can log in’ to fixing problems before the users even know something is wrong.

The real costs of Datadog Synthetics monitoring

Detect

Communicate

Resolve

Monitoring as Code

Developers

Resources

Community

The real costs of Datadog Synthetics monitoring