When adding new Checks in Checkly a number of locations are available to check your endpoints from multiple locations around the world. For most use cases this is more than enough to ensure your resources are online.
However, these locations are outside of your network and are unable to check on resources deployed more securely inside your private network.
To help get around this, Checkly offers a Private Locations agent, allowing you to deploy an agent inside of your network and use it as a trusted location to perform your checks.
What is a Private Location?
A Private Location in Checkly is similar to existing public locations but it exists in your own infrastructure, on-premise or in the cloud. Agents are stateless, meaning you don't have to worry about managing the agent in any way. Just one agent running in your own infrastructure is enough to get started with Private locations.
This presents an opportunity to access private resources that you couldn't otherwise monitor directly from Checkly, including commonly used resources such as AWS RDS, Elasticsearch or Elasticache, which if deployed correctly are available internally and not publicly accessible.
Using Private Locations also provides you with reduced latency so your checks will respond a little bit faster each time they run.
Installing and running the Checkly agent
Running the Checkly agent locally is straightforward, as Checkly provides a docker image to run. You can get your Private Location key from Checkly first and run the container as follows:
docker run -e API_KEY="pl_...." -d ghcr.io/checkly/agent:latest
One private location in the same region as your existing infrastructure is enough to get started. By comparison, the Datadog agent requires you to install the agent on each individual instance to monitor it.
There are a number of ways to deploy the Checkly container agent within AWS. Any service capable of running a container such as Fargate or EKS hosted kubernetes will work. In this post we'll keep it simple and install docker on a small EC2 instance to run the container and we will create it using Terraform.
The full source code for this project is available here: https://github.com/gordonmurray/terraform_aws_checkly_private_location
It is likely you have some infrastructure in place already that you'd like to begin monitoring, however for completeness this project includes all of the necessary resources to create a working infrastructure on AWS which includes a VPC, subnets, routing tables and more to provide us with resources to check on with Checkly.
Using a Terraform block to create an EC2 instance to use as a Checkly location
Instead of looking into each individual resource Terraform is creating in this project we will focus on the EC2 instance that terraform is creating to become a Checkly private location.
#Create an EC2 instance to use as a Checkly location
resource "aws_instance" "checkly" {
ami = "ami-0333305f9719618c7" # Ubuntu 22.04
instance_type = "t3.nano"
key_name = aws_key_pair.pem-key.key_name
subnet_id = aws_subnet.subnet-1a.id
vpc_security_group_ids = [aws_security_group.webserver.id]
// ...
user_data = data.template_file.checkly_user_data.rendered
}
This terraform block is creating a small EC2 instance. The Checkly agent doesn't need a lot of resources to work so a small instance is enough and keeps costs low.
Terraform is supplying some user data to the EC2 instance. This user data is a script with steps to install docker and the Checkly agent when the EC2 instance starts up for the first time.
If you have terraform installed and if you have updated the variables file to add your own Checkly API key and location key, you should be able to run Terraform to create the instance and other resources. In a few moments when the EC2 instance is running and the docker container has started you'll have a working private location visible in your Checkly account.
If you create or update any existing checks in your Checkly account, this private location will now be available to you to use as an additional location to perform your checks from.
Using the Checkly Terraform Provider
Thanks to a Checkly Terraform Provider, we can further use Terraform to add some new checks in our account to check on some private resources such as an RDS instance that Checkly would ordinarily not be able to reach.
The project includes the necessary resources to create an RDS database instance and also an Elasticache Redis instance to use as private resources we can check on.
We can now use Terraform to create the instances as well as create the checks to monitor them also which can be a great time saver in day to day infrastructure development.
resource "checkly_check" "internal-api-check" {
name = "Health check"
type = "API"
// …
private_locations = [
"private-location"
]
request {
URL = "http://${aws_instance.checkly.private_ip}:8080/healthz"
follow_redirects = true
body_type = "JSON"
method = "GET"
assertion {
source = "JSON_BODY"
property = "$.results[1].found"
comparison = "EQUALS"
target = "PONG"
}
}
In this resource the two main areas to look at are the private locations block and the request block.
The private locations block needs to hold the name of the new private location you created in your Checkly account.
The request block provides Checkly with an endpoint to test and a couple of assertions to tell it what to expect back from the endpoint it is testing, such as a HTTP status code or some desired text in the body of the response.
Using Goss to create an endpoint for monitoring
The URL that is being monitored in the check is the most interesting part. It is set up to perform a GET request to the private IP address of the private instance we have created, like a regular API check. The items we want to check such as an RDS and Elasticache do not have an API endpoint we can check directly. Instead we use a tool called Goss. It is a Go application that can run commands such as pinging an Elasticache node and return the results in the format of an API that Checkly can perform requests against.
In this example Checkly is looking at http://xxxxx:8080/healthz. This is the Goss endpoint showing results from 2 checks, one for the RDS instance and one for the Elasticache instance. The Goss configuration to check both resources is a short yaml file:
command:
rds:
exec: mysql -h ${rds_host} -u ${rds_username} -p${rds_password} -N -e "select 1;"
exit-status: 0
stdout:
- 1
stderr: []
timeout: 10000
redis:
exec: redis-cli -h ${redis_host} -r 1 ping
exit-status: 0
stdout:
- PONG
stderr: []s
timeout: 10000You will notice some variables in this file. One of the advantages of using Terraform to create resources—as well as creating the checks in Checkly—is that Terraform can pass the resource endpoints such as an RDS DNS address in to the template that becomes a yaml file for Goss to use.
This Goss configuration pings the Elasticache node and runs a MySQL query to the RDS instance. Checkly looks at the results and checks that the expected results are in place. In response to a Ping command to Redis, we expect a ‘PONG’ response in return, and in response to running a query on the RDS instance we expect back a ‘1’ response which Checkly can check.
We now have Checkly monitoring our RDS and Elasticache instances, through the private location. We can take one final optional step and use Terraform to create a nice dashboard to visualize the results of our checks which we can share internally with our team or with stakeholders. Your dashboard will be accessible at http://{your_unique_custom _URL}.checkly-dashboards.com.
resource "checkly_dashboard" "dashboard_1" {
custom_URL = "aws-checkly-private-location"
custom_domain = "www.domain.com"
header = "AWS Infrastructure dashboard"
refresh_rate = 60
paginate = false
pagination_rate = 30
hide_tags = false
width = "FULL"
tags = [
"private"
]
}Summary
In this post we've used Terraform to create a Checkly Private Location in our own infrastructure. We then continued to use Terraform to create checks and Dashboard monitoring resources within our Private network which would normally be out of Checkly’s reach.
The combination of Terraform, Checkly private locations, and dashboards within your Checkly account can make monitoring your infrastructure an appealing and cost effective monitoring solution.