【Terraform】Terraform ～ AWS CloudWatch ～

Terraform

■ はじめに

https://dk521123.hatenablog.com/entry/2023/05/14/122215

のトラブルシュートのために、
AWS CloudWatchで出力することを学んだので、メモ。

目次

【１】公式ドキュメント
　１）Resource: aws_cloudwatch_log_group
【２】使い方
【３】サンプル
　１）For MSK Cluster
　２）For MSK Connector

【１】公式ドキュメント

１）Resource: aws_cloudwatch_log_group

* AWS CloudWatch Log Group の作成

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group

【２】使い方

[1] 「Resource: aws_cloudwatch_log_group」を使ってロググループを定義
[2] [1] をそれぞれのAWSサービスリソースの定義に沿って付与する

【３】サンプル

１）For MSK Cluster

# AWS CloudWatch
resource "aws_cloudwatch_log_group" "demo_cloudwatch_log_for_msk_cluster" {
  name = "/aws/msk/cluster"
  # 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731,
  # 1096, 1827, 2192, 2557, 2922, 3288, 3653, and 0
  retention_in_days = 7

  tags = {
    Environment = "dev"
    Application = "serviceA"
  }
}

# If you want other codes, see https://dk521123.hatenablog.com/entry/2023/05/14/122215

# MSK cluster
resource "aws_msk_cluster" "demo_msk_cluster" {
  cluster_name           = "demo-msk-cluster"
  kafka_version          = "2.7.1"
  number_of_broker_nodes = 3

  broker_node_group_info {
    instance_type = "kafka.m5.4xlarge"
    client_subnets = [
      aws_subnet.subnet_az1.id,
      aws_subnet.subnet_az2.id,
      aws_subnet.subnet_az3.id,
    ]
    storage_info {
      ebs_storage_info {
        provisioned_throughput {
          enabled           = true
          volume_throughput = 250
        }
        volume_size = 1000
      }
    }
    security_groups = [aws_security_group.demo_msk_sg.id]
  }

  # To set timeout (Default is 20min)
  timeouts {
    create = "1h"
    update = "1h"
    delete = "1h"
  }

  # ★ここに注目★
  # For logging
  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_cluster#logging_info  
  logging_info {
    # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_cluster#broker_logs  
    broker_logs {
      # 
      cloudwatch_logs {
        enabled   = true
        log_group = aws_cloudwatch_log_group.demo_cloudwatch_log_for_msk_cluster.name
      }
      #firehose {
      #  enabled         = true
      #  delivery_stream = aws_kinesis_firehose_delivery_stream.test_stream.name
      #}
      #s3 {
      #  enabled = true
      #  bucket  = aws_s3_bucket.bucket.id
      #  prefix  = "logs/msk-cluster-"
      #}
    }
  }
}

２）For MSK Connector

# AWS CloudWatch
resource "aws_cloudwatch_log_group" "demo_cloudwatch_log_for_msk_connector" {
  name = "/aws/msk/connector"
  # 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731,
  # 1096, 1827, 2192, 2557, 2922, 3288, 3653, and 0
  retention_in_days = 7

  tags = {
    Environment = "dev"
    Application = "serviceA"
  }
}

# If you want other codes, see https://dk521123.hatenablog.com/entry/2023/05/14/122215

resource "aws_mskconnect_connector" "demo_mskconnect_connector" {
  name = "demo-msk-connect"

  kafkaconnect_version = "2.7.1"

  capacity {
    autoscaling {
      mcu_count        = 1
      min_worker_count = 1
      max_worker_count = 2

      scale_in_policy {
        cpu_utilization_percentage = 20
      }

      scale_out_policy {
        cpu_utilization_percentage = 80
      }
    }
  }

  # Connector Conguration
  connector_configuration = {
    "name" = "demo-msk-connect"
    "connector.class" = "com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkConnector"
    "tasks.max"       = "3"
    "topics"          = "demo-topic"
  }

  kafka_cluster {
    apache_kafka_cluster {
      bootstrap_servers = aws_msk_cluster.demo_msk_cluster.bootstrap_brokers_tls

      vpc {
        security_groups = [aws_security_group.demo_msk_sg.id]
        subnets = [aws_subnet.subnet_az1.id, subnet_az2.example2.id, subnet_az2.example3.id]
      }
    }
  }

  kafka_cluster_client_authentication {
    authentication_type = "NONE"
  }

  # https://docs.aws.amazon.com/ja_jp/MSKC/latest/mskc/API_KafkaClusterEncryptionInTransit.html
  # Valid Values: PLAINTEXT | TLS
  kafka_cluster_encryption_in_transit {
    encryption_type = "TLS"
  }

  plugin {
    custom_plugin {
      arn      = aws_mskconnect_custom_plugin.demo_msk_connect_custom_plugin.arn
      revision = aws_mskconnect_custom_plugin.demo_msk_connect_custom_plugin.latest_revision
    }
  }

  service_execution_role_arn = aws_iam_role.demo_msk_connector_service_execution_role.arn

  # To set timeout (Default is 20min)
  timeouts {
    create = "1h"
    update = "1h"
    delete = "1h"
  }

  # ★ここに注目★
  # For logging
  # See https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_connector#log_delivery  
  log_delivery {
    # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_connector#cloudwatch_logs-configuration-block  
    worker_log_delivery {
       cloudwatch_logs {
         # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_connector#cloudwatch_logs-configuration-block  
         enabled = true
         log_group = aws_cloudwatch_log_group.demo_cloudwatch_log_for_msk_connector.name
       }
    }
  }
}

関連記事

Terraform ～ 環境構築編 ～
https://dk521123.hatenablog.com/entry/2023/04/05/000224
Terraform ～ 入門編 ～
https://dk521123.hatenablog.com/entry/2019/12/09/222057
Terraform ～ 基本編 ～
https://dk521123.hatenablog.com/entry/2023/05/03/000000
Terraform ～ AWS IAM ～
https://dk521123.hatenablog.com/entry/2023/04/12/214311
Terraform ～ AWS S3 ～
https://dk521123.hatenablog.com/entry/2023/04/09/104204
Terraform ～ AWS MSK ～
https://dk521123.hatenablog.com/entry/2023/05/14/122215