🤖 Meet OnCall AI, our observability copilot that makes troubleshooting easy. Read announcement.

Skip to content
Engineering

Introducing Edge Delta Terraform Provider: Manage Hundreds of Agents in a Single Terraform Config

Mar 21, 2022 / 9 minute read

Edge Delta’s new Terraform provider allows you to create and manage hundreds of agents with ease

.

Edge Delta features, like agents and monitors, enrich your observability pipeline. It does so by analyzing your data as it’s created at the source and surfacing insights, such as changes in your systems and anomalous behavior.

As customers go from monitoring multiple data sources to managing hundreds, we’ve received feedback to automate the configurations of agents, monitors, and other features. As a result, we’ve come up with a solution that’s easy to use and developer-friendly: a Terraform provider for Edge Delta resources!

We present you terraform-provider-edgedelta. After considering and trying different solutions, we found that Terraform's HCL provides an excellent declarative language to define various Edge Delta resources clearly and precisely in a single configuration file. It also supports iterative constructs and templates, which helps you avoid lengthy, repetitive configuration files declaring hundreds of resources quite similar to each other.

At the time being, Edge Delta Terraform provider supports the following resources:

  • Agent configuration (edgedelta_config)

  • - Monitor (edgedelta_monitor)

This post will go over the current features of terraform-provider-edgedelta with step-by-step examples for edgedelta_config and edgedelta_monitor resources.

Setting Up the Environment

Before using the provider, you need to have Terraform CLI installed. If you don't, you can install it following the steps here.

The setup is pretty straightforward, we just need to initialize a .tf config file using provider "edgedelta/edgedelta":

terraform {
  required_providers {
    edgedelta = {
      source  = "edgedelta/edgedelta"
      version = "0.0.5"
    }
  }
}

variable "ED_API_TOKEN" {
  type = string
}

provider "edgedelta" {
  org_id             = ""
  api_secret         = var.ED_API_TOKEN
}

We can save this config file to edgedelta.tf and run terraform init to initialize the provider:

terraform init

Before we start, we need to provide some credentials to the provider. We can do this by setting the ED_API_TOKEN variable in the environment:

export TF_VAR_ED_API_TOKEN=

If you don't have an API token, you can create one from the global settings. Here, you can find the API token under the "Token" section. Click on the "Create Token" button and give your token a name. Be sure to add "Agent Configurations" and "Monitors" permissions to your token. You can then use the token in the environment variable TF_VAR_ED_API_TOKEN.

Creating an Agent Configuration

Let's start by defining a new agent configuration called new_config:

resource "edgedelta_config" "new_config" {
  config_content = file("new_config.yml")
}

Here, edgedelta_config is the type of the resource we're defining. We have also used config_content to specify the content of the agent configuration, and the file function to read the configuration content from a YAML file. This is a Terraform built-in function that allows us to read files from the disk.

Let's also put the configuration content in a file called new_config.yml:

version: v2

outputs:
  streams:
    - name: cloudwatch-test-integration
      type: cloudwatch
      region: us-west-2
      log_group_name: /ecs/microservice
      log_stream_name: test-stream

Here, we defined a dummy agent configuration using Edge Delta's Amazon CloudWatch log stream integration. This configuration will be used to create a new agent.

We are using the Amazon CloudWatch integration just for the sake of example. You can use any other integration.

To see the configuration ID that will be generated for the new agent configuration, we can define an output variable:

output "new_config_id" {
  value              = edgedelta_config.new_config.id
  description        = "The config ID created for the edgedelta_config.new_config instance"
}

To put the configuration all together, we now have:

terraform {
  required_providers {
    edgedelta = {
      source  = "edgedelta/edgedelta"
      version = "0.0.5"
    }
  }
}

variable "ED_API_TOKEN" {
  type = string
}

provider "edgedelta" {
  org_id             = ""
  api_secret         = var.ED_API_TOKEN
}

output "new_config_id" {
  value              = edgedelta_config.new_config.id
  description        = "The config ID created for the edgedelta_config.new_config instance"
}

resource "edgedelta_config" "new_config" {
  config_content = file("new_config.yml")
}

To see the actions that Terraform will apply using the configuration we have just created, we can use terraform plan:

terraform plan

This will create a plan that Terraform will execute to create the new agent configuration. We can see that the new agent configuration will be created and the output variable new_config_id will be set to the ID of the new agent configuration:

Terraform will perform the following actions:

  # edgedelta_config.new_config will be created
  + resource "edgedelta_config" "new_config" {
      + config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
                region: us-west-2
                log_group_name: /ecs/microservice
                log_stream_name: test-stream
        EOT
      + id             = (known after apply)
      + tag            = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + new_config_id = (known after apply)

Now we can run terraform apply to create the new agent configuration:

terraform apply

After typing yes and the command finishes, we can see the new agent configuration:

edgedelta_config.new_config: Creating...
edgedelta_config.new_config: Creation complete after 2s [id=11111111-2222-3333-4444-555555555555]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

new_config_id = "11111111-2222-3333-4444-555555555555"

We can also navigate to app.edgedelta.com/agent-settings to see the new agent configuration:

.

Updating the Agent Configuration

Now that we have created a new agent configuration, we can update it by changing the config_content attribute. To do this, we just need to update the configuration content in the new_config.yml file. Let's change the region attribute to us-west-1 and apply terraform plan:

terraform plan

The plan shows that the agent configuration will be updated:

Terraform will perform the following actions:

  # edgedelta_config.new_config will be updated in-place
  ~ resource "edgedelta_config" "new_config" {
      ~ config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
          -     region: us-west-2
          +     region: us-west-1
                log_group_name: /ecs/microservice
                log_stream_name: test-stream
        EOT
        id             = "11111111-2222-3333-4444-555555555555"
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

As we can see, Terraform will update the agent configuration by only changing the region attribute to us-west-1. Now we can run terraform apply to update the agent configuration:

terraform apply
edgedelta_config.new_config: Modifying... [id=11111111-2222-3333-4444-555555555555]
edgedelta_config.new_config: Modifications complete after 1s [id=11111111-2222-3333-4444-555555555555]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

new_config_id = "11111111-2222-3333-4444-555555555555"

The agent configuration is now updated on the Edge Delta side and the new region is us-west-1.

Deleting the Agent Configuration

If we want to delete the agent configuration, we can use terraform destroy. This will delete the agent configuration:

terraform destroy

After typing yes, we can see that the agent configuration is deleted:

Terraform will perform the following actions:

  # edgedelta_config.new_config will be destroyed
  - resource "edgedelta_config" "new_config" {
      - config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
                region: us-west-1
                log_group_name: /ecs/microservice
                log_stream_name: test-stream
        EOT -> null
      - id             = "11111111-2222-3333-4444-555555555555" -> null
      - tag            = "Edge" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - new_config_id = "11111111-2222-3333-4444-555555555555" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

edgedelta_config.new_config: Destroying... [id=11111111-2222-3333-4444-555555555555]
edgedelta_config.new_config: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.

Creating a Monitor Resource

Just like the agent configuration, we can create monitors using the edgedelta_monitor resource in Terraform. In Edge Delta we have 4 types of monitors: correlated-signal, mettric-alert, pattern-check and pattern-skyline. In this example we will create a pattern-check monitor.

You can find the full list of monitor types in the Edge Delta documentation.

Let's create a monitor resource named pattern_check_monitor in our Terraform config. The resource definition is as follows:

output "monitor_id" {
    value = edgedelta_monitor.pattern_check_monitor.id
}

resource "edgedelta_monitor" "pattern_check_monitor" {
    name     = "pattern-check-example-monitor"
    type     = "pattern-check"
    enabled  = true
    payload  = file("payload.json")
    creator  = "creator@email.domain"
}

And also provide the payload file payload.json in the same directory as the Terraform config file. The payload file is as follows:

{
  "merge_level": "none",
  "mail_recipients": "test1@example.org",
  "triggers": ["slack-trigger"],
  "suppression_window": "12h",
  "timezone": "Europe/London",
  "trigger_template": ""
}

To create the monitor, run terraform apply:

terraform apply

Which outputs the following:

Terraform will perform the following actions:

  # edgedelta_monitor.pattern_check_monitor will be created
  + resource "edgedelta_monitor" "pattern_check_monitor" {
      + creator = "creator@email.domain"
      + enabled = true
      + id      = (known after apply)
      + name    = "pattern-check-example-monitor"
      + payload = jsonencode(
            {
              + mail_recipients    = "test1@example.org"
              + merge_level        = "none"
              + suppression_window = "12h"
              + timezone           = "Europe/London"
              + trigger_template   = ""
              + triggers           = [
                  + "slack-trigger",
                ]
            }
        )
      + type    = "pattern-check"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + monitor_id = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

edgedelta_monitor.pattern_check_monitor: Creating...
edgedelta_monitor.pattern_check_monitor: Creation complete after 2s [id=222222-3333-4444-5555-666666666666]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

monitor_id = "222222-3333-4444-5555-666666666666"

Updating a Monitor Resource

After making changes to the monitor definition in the Terraform configuration, we can update the monitor by running terraform apply.

Let's update the payload of the pattern-check monitor we have created above and change the timezone to America/Los_Angeles:

{
  "merge_level": "none",
  "mail_recipients": "test1@example.org",
  "triggers": ["slack-trigger"],
  "suppression_window": "12h",
  "timezone": "America/Los_Angeles",
  "trigger_template": ""
}

Now we can run terraform apply to update the monitor:

terraform apply

We can see that the monitor is updated successfully in the output:

Terraform will perform the following actions:

  # edgedelta_monitor.pattern_check_monitor will be updated in-place
  ~ resource "edgedelta_monitor" "pattern_check_monitor" {
        id      = "222222-3333-4444-5555-666666666666"
        name    = "pattern-check-example-monitor"
      ~ payload = jsonencode(
          ~ {
              ~ timezone           = "Europe/London" -> "America/Los_Angeles"
                # (5 unchanged elements hidden)
            }
        )
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

edgedelta_monitor.pattern_check_monitor: Modifying... [id=222222-3333-4444-5555-666666666666]
edgedelta_monitor.pattern_check_monitor: Modifications complete after 2s [id=222222-3333-4444-5555-666666666666]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

monitor_id = "222222-3333-4444-5555-666666666666"

Deleting a Monitor Resource

To delete a monitor, we can run terraform destroy:

terraform destroy

This will delete our existing monitor resource with ID 222222-3333-4444-5555-666666666666:

Terraform will perform the following actions:

  # edgedelta_monitor.pattern_check_monitor will be destroyed
  - resource "edgedelta_monitor" "pattern_check_monitor" {
      - creator = "creator@email.domain" -> null
      - enabled = true -> null
      - id      = "222222-3333-4444-5555-666666666666" -> null
      - name    = "pattern-check-example-monitor" -> null
      - payload = jsonencode(
            {
              - mail_recipients    = "test1@example.org"
              - merge_level        = "none"
              - suppression_window = "12h"
              - timezone           = "America/Los_Angeles"
              - trigger_template   = ""
              - triggers           = [
                  - "slack-trigger",
                ]
            }
        ) -> null
      - type    = "pattern-check" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - monitor_id = "222222-3333-4444-5555-666666666666" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

edgedelta_monitor.pattern_check_monitor: Destroying... [id=222222-3333-4444-5555-666666666666]
edgedelta_monitor.pattern_check_monitor: Destruction complete after 2s

Destroy complete! Resources: 1 destroyed.

As we can see, the monitor is deleted successfully in the output.

Using Templates to Create Multiple Resources

One of the most useful features of the provider is the ability to create multiple resources at once. To do this, we can use the templatefile function and for_each construct in Terraform. templatefile function allows us to use a template file to create multiple resources. The template file for this example should be in YAML format. Let's use config.yml.tpl as the template file:

version: v2

outputs:
  streams:
    - name: cloudwatch-test-integration
      type: cloudwatch
      region: ${region}
      log_group_name: /ecs/microservice
      log_stream_name: ${log_stream_name}

Here we are using the ${region} and ${log_stream_name} variables to replace the values in the template file. We can use the for_each construct to iterate over a list of values:

output "multi_conf_ids" {
  value = {
    for k, v in edgedelta_config.multi_confs : k => v.id
  }
  description        = "The config ID created for the edgedelta_config.new_config instance"
}

variable "config_instances" {
    type = map(map(string))
    default = {
        "instance_1" = {
            "region"           = "us-west-1",
            "log_stream_name"  = "test-stream-1"
        },
        "instance_2" = {
            "region"           = "us-west-1",
            "log_stream_name"  = "test-stream-2"
        },
        "instance_3" = {
            "region"           = "us-west-2",
            "log_stream_name"  = "test-stream-3"
        },
        "instance_4" = {
            "region"           = "us-west-2",
            "log_stream_name"  = "test-stream-4"
        },
    }
}

resource "edgedelta_config" "multi_confs" {
    for_each = var.config_instances

    config_content = templatefile("${path.module}/config.yml.tpl", {
        region = each.value["region"]
        log_stream_name = each.value["log_stream_name"]
    })
}

The for_each construct will iterate over the values in the config_instances variable and replace the ${region} and ${log_stream_name} variables with the values from the current iteration. The templatefile function will use the template file to create the configuration content for each iteration.

To try this out, we can run terraform plan:

terraform plan

This will list the four new agent configurations that will be created:

Terraform will perform the following actions:

  # edgedelta_config.multi_confs["instance_1"] will be created
  + resource "edgedelta_config" "multi_confs" {
      + config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
                region: us-west-1
                log_group_name: /ecs/microservice
                log_stream_name: test-stream-1
        EOT
      + id             = (known after apply)
      + tag            = (known after apply)
    }

  # edgedelta_config.multi_confs["instance_2"] will be created
  + resource "edgedelta_config" "multi_confs" {
      + config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
                region: us-west-1
                log_group_name: /ecs/microservice
                log_stream_name: test-stream-2
        EOT
      + id             = (known after apply)
      + tag            = (known after apply)
    }

  # edgedelta_config.multi_confs["instance_3"] will be created
  + resource "edgedelta_config" "multi_confs" {
      + config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
                region: us-west-2
                log_group_name: /ecs/microservice
                log_stream_name: test-stream-3
        EOT
      + id             = (known after apply)
      + tag            = (known after apply)
    }

  # edgedelta_config.multi_confs["instance_4"] will be created
  + resource "edgedelta_config" "multi_confs" {
      + config_content = <<-EOT
            version: v2

            outputs:
              streams:
              - name: cloudwatch-test-integration
                type: cloudwatch
                region: us-west-2
                log_group_name: /ecs/microservice
                log_stream_name: test-stream-4
        EOT
      + id             = (known after apply)
      + tag            = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + multi_conf_ids = {
      + instance_1 = (known after apply)
      + instance_2 = (known after apply)
      + instance_3 = (known after apply)
      + instance_4 = (known after apply)
    }

Now we can run terraform apply to create the new agent configurations:

terraform apply

When the terraform apply command is run, the provider will create the new agent configurations:

edgedelta_config.multi_confs["instance_2"]: Creating...
edgedelta_config.multi_confs["instance_1"]: Creating...
edgedelta_config.multi_confs["instance_3"]: Creating...
edgedelta_config.multi_confs["instance_4"]: Creating...
edgedelta_config.multi_confs["instance_3"]: Creation complete after 1s [id=333333-4444-5555-6666-777777777777]
edgedelta_config.multi_confs["instance_4"]: Creation complete after 1s [id=444444-5555-6666-7777-888888888888]
edgedelta_config.multi_confs["instance_2"]: Creation complete after 2s [id=222222-3333-4444-5555-666666666666]
edgedelta_config.multi_confs["instance_1"]: Creation complete after 3s [id=111111-2222-3333-4444-555555555555]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

multi_conf_ids = {
  "instance_1" = "111111-2222-3333-4444-555555555555"
  "instance_2" = "222222-3333-4444-5555-666666666666"
  "instance_3" = "333333-4444-5555-6666-777777777777"
  "instance_4" = "444444-5555-6666-7777-888888888888"
}

Similar to the previous example, we can use terraform destroy to delete the agent configurations:

terraform destroy

As the previous example, the provider will delete the agent configurations:

edgedelta_config.multi_confs["instance_1"]: Destroying... [id=111111-2222-3333-4444-555555555555]
edgedelta_config.multi_confs["instance_2"]: Destroying... [id=222222-3333-4444-5555-666666666666]
edgedelta_config.multi_confs["instance_4"]: Destroying... [id=444444-5555-6666-7777-888888888888]
edgedelta_config.multi_confs["instance_3"]: Destroying... [id=333333-4444-5555-6666-777777777777]
edgedelta_config.multi_confs["instance_1"]: Destruction complete after 0s
edgedelta_config.multi_confs["instance_4"]: Destruction complete after 0s
edgedelta_config.multi_confs["instance_3"]: Destruction complete after 0s
edgedelta_config.multi_confs["instance_2"]: Destruction complete after 0s

Destroy complete! Resources: 4 destroyed.

Importing Existing Resources

If you already have agent configuration or monitor resources on the Edge Delta side, you can import it into Terraform using the terraform import command. This will import the agent configuration/monitor into the Terraform state file. After importing to the state file, you can use the terraform show command to see the existing resource in HCL format, which can be used to add the resource to your Terraform configuration.

Let's assume we have a monitor with ID 11111111-2222-3333-4444-555555555555 and payload same as payload.json from the example in [creating a monitor resource](#creating-a-monitor-resource) section above, on the Edge Delta side. We can import it into Terraform, to a monitor resource named imported_monitor in our Terraform config using the terraform import command. Let the following be the resource definition we will use to import the monitor:

resource "edgedelta_monitor" "imported_monitor" {

}

Simply run the following command to import the monitor:

terraform import edgedelta_monitor.imported_monitor 11111111-2222-3333-4444-555555555555

Which outputs the following:

edgedelta_monitor.imported_monitor: Importing from ID "11111111-2222-3333-4444-555555555555"...
edgedelta_monitor.imported_monitor: Import prepared!
  Prepared edgedelta_monitor for import
edgedelta_monitor.imported_monitor: Refreshing state... [id=11111111-2222-3333-4444-555555555555]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Now we can use the terraform show command to see the imported monitor in HCL format:

terraform show

The output of terraform show is shown below:

# edgedelta_monitor.imported_monitor:
resource "edgedelta_monitor" "imported_monitor" {
    creator    = "mail@email.domain"
    enabled    = true
    id         = "11111111-2222-3333-4444-555555555555"
    monitor_id = "11111111-2222-3333-4444-555555555555"
    name       = "pattern-check-monitor-remote"
    payload    = jsonencode(
        {
            mail_recipients    = "test1@example.org"
            merge_level        = "none"
            suppression_window = "12h"
            timezone           = "Europe/London"
            trigger_template   = ""
            triggers           = [
                "slack-trigger",
            ]
        }
    )
    type       = "pattern-check"
}

We can directly add this output to our Terraform configuration file. Now that we have imported the monitor, we can use the terraform apply command to synchronize the monitor with the remote one. After that, we can do changes to the resource and run terraform apply, or terraform destroy to delete the resource.

Importing Multiple Resources

Edge Delta Terraform provider also supports importing multiple resources at once. To do this, we can use the same importing command by specifying comma-separated resource IDs. The example command below will import the agent configurations with IDs 11111111, 22222222 and 33333333:

terraform import edgedelta_config.imported_config 11111111,22222222,33333333

Which outputs the following:

edgedelta_config.imported_config: Import prepared!
  Prepared edgedelta_config for import
  Prepared edgedelta_config for import
  Prepared edgedelta_config for import
edgedelta_config.imported_config: Refreshing state... [id=11111111]
edgedelta_config.imported_config-1: Refreshing state... [id=22222222]
edgedelta_config.imported_config-2: Refreshing state... [id=33333333]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Importing All Resources of a Resource Type

In another use case, we can import all the specific type of resources by using "*" as the resource ID. The example command below will import all the monitors we have remotely on the Edge Delta side:

terraform import edgedelta_monitor.imported_monitor "*"

As in the previous example, remote resources (monitors) will be imported to the state file with names imported_monitor-1, imported_monitor-2, etc.

Conclusion

Now that you’ve seen how the Edge Delta Terraform provider works, you’re ready to start using it. We hope that by automating configurations at scale, you can more gain visibility into every data source in a frictionless manner.

.

Ozan Sazak

Software Engineer

Filed Under

Share

Stay in Touch

Sign up for our newsletter to be the first to know about new articles.