■ はじめに

Amazon Managed Workflows for Apache Airflow (MWAA) を
Terraform で作る。

なお、MWAA および Airflow については、以下の関連記事を参照のこと。

MWAA ～入門編～
https://dk521123.hatenablog.com/entry/2021/09/29/131101
Apache Airflow ～基礎知識編～
https://dk521123.hatenablog.com/entry/2021/09/28/135510

【１】公式ドキュメント

１）Resource

Resource: aws_mwaa_environment
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mwaa_environment

２）Module

source = "aws-ia/mwaa/aws"
https://registry.terraform.io/modules/aws-ia/mwaa/aws/latest
source = "idealo/mwaa/aws"
https://registry.terraform.io/modules/idealo/mwaa/aws/latest

【２】サンプル

main.tf

# ---------------------------------------------------------------------------------------------------------------------
# MWAA Environment
# ---------------------------------------------------------------------------------------------------------------------
resource "aws_mwaa_environment" "mwaa_demo" {
  name = "demo-mwaa"
  execution_role_arn = "arn:aws:iam::${local.account_id}:role/xxxxxxxx"

  environment_class = "mw1.small"
  min_workers = 1
  max_workers = 1
  webserver_access_mode = "PUBLIC_ONLY"

  source_bucket_arn               = local.source_bucket_arn
  dag_s3_path                      = "xxxx/dags"

#  plugins_s3_object_version = "latest"
#  plugins_s3_path                  = "xxxx/dags/plugins/plugins.zip"
#  requirements_s3_path             = "latest"
#  requirements_s3_object_version   = "xxxx/requirements/requirements.txt"
#  startup_script_s3_path           = "latest"
#  startup_script_s3_object_version = "xxxx/startup/startup.sh"

  airflow_configuration_options = {
    "core.default_task_retries" = 1
    "core.parallelism"          = 1
  }

  # kms_key           = "your-kms-key"

  tags = merge(local.common_tags, {"Name" = "demo-mwaa" })

  network_configuration {
    security_group_ids = [aws_security_group.demo_mwaa_sg.id]
    subnet_ids = [
      "subnet-xxxxxxxx1",
      "subnet-xxxxxxxx2"
    ]
  }

  logging_configuration {
    dag_processing_logs {
      enabled   = true
      log_level = local.logging_level
    }
    scheduler_logs {
      enabled   = true
      log_level = local.logging_level
    }
    task_logs {
      enabled   = true
      log_level = local.logging_level
    }
    webserver_logs {
      enabled   = true
      log_level = local.logging_level
    }
    worker_logs {
      enabled   = true
      log_level = local.logging_level
    }
  }

  lifecycle {
    ignore_changes = [
      plugins_s3_object_version,
      requirements_s3_object_version
    ]
  }
  depends_on = [
    aws_security_group.demo_mwaa_sg
  ]
}

resource "aws_security_group" "demo_mwaa_sg" {
  name = "demo-mwaa-sg"
  description = "This is for Demo MWAZ security group"

  vpc_id = local.vpc_id
  ingress {
    description = "This is for Demo"
    from_port = 0
    to_port = 0
    protocol = "-1"
    self = true
  }
  egress {
    description = "This is for Demo"
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  tags = merge(local.common_tags, {"Name" = "demo-mwaa-sg" })
}

locals.tf

data "aws_caller_identity" "current" {}

locals {
  source_bucket_arn = "arn:aws:s3::::your-s3-buket"
  account_id = data.aws_caller_identity.current.account_id
  vpc_id = "vpc-xxxxxxxxx"
  logging_level = "DEBUG"
  common_tags = {
    Env = "sandbox"
  }
}

backend.tf

terraform {
  backend "s3" {
    bucket  = "your-s3-bucket"
    key     = "tf/dev/terraform.tfstate"
    region  = "us-west-2"
  }

provider.tf

# Configure the AWS Provider
provider "aws" {
  region = "us-west-2"
}

【３】MWAA あれこれ

１）Airflow variablesを設定するには

* スタートアップスクリプトを使用して、設定するのもあり

https://docs.aws.amazon.com/ja_jp/mwaa/latest/userguide/using-startup-script.html

参考文献

https://dev.classmethod.jp/articles/mwaa_tf/

プログラムの超個人的なメモ

Memo for Programming.

【Terraform】Terraform ～ AWS MWAA ～

■ はじめに

目次

【１】公式ドキュメント

１）Resource

２）Module

【２】サンプル

【３】MWAA あれこれ

１）Airflow variablesを設定するには

参考文献

関連記事