【Terraform】Terraform ～ AWS Glue ～

Terraform

■ はじめに

https://dk521123.hatenablog.com/entry/2023/04/05/000224

の続き。

今回は、AWS Glue に関して、少しずつだが、調べた内容を纏めていく

目次

【１】公式ドキュメント
【２】サンプル
　例１：1Workflow-1Job-1StartTrigger
　例２：Scheduled Trigger
【３】Tips
　１）タグ
【４】AWSへのデプロイ手順

【１】公式ドキュメント

* AWS Glue に関する記述は、以下。

Glue Job
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_job
Glue Trigger
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_trigger
Glue Workflow
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_workflow

【２】サンプル

例１：1Workflow-1Job-1StartTrigger

# [0]-1 To define setting values
locals {
  env = "dev" # or stage or prod
  glue_job_role_arn = "arn:aws:iam:00000000:role/your-glue-job-role-arn" # Real ARN
  glue_job_script_location = "s3://your-bucket-name/demo/glue/hello.scala"
  glue_job_script_class = "com.demo.Hello"
  glue_job_script_jar_location = "s3://your-bucket-name/demo/glue/utils.jar"
  glue_job_templocation = "s3://your-bucket-name/temp/"
}

# [0]-2：To specify AWS
provider "aws" {
  region = "us-west-2"
  default_tags {
    tags = {
      env = local.env
    }
  }
}

# [1] To create Glue workflow
resource "aws_glue_workflow" "demo-glue-workflow" {
  name = "demo-${local.env}-glue-workflow"
  description = "This is a sample Glue workflow"
  tags = {
    name = "demo-${local.env}-glue-workflow"
  }
}

# [2] To create Glue Job
resource "aws_glue_job" "aws_job1" {
  name = "demo-${local.env}-glue-job1"
  description = "This is a sample Glue job"
  role_arn = local.glue_role_arn
  glue_version = "4.0"
  max_retries = local.env == "dev" ? 0 : 3
  timeout = local.env == "dev" ? 60 : 2880
  worker_type = local.env == "dev" ? Standard : G.2X
  number_of_workers = local.env == "dev" ? 3 : 10

  command {
    name = glueetl
    script_location = local.glue_job_script_location
  }
  connections = [ "your-glue-connection" ]

  # Glue Job parameters
  default_arguments = {
    "--job-language" = "scala"
    "--class" = local.glue_job_script_class
    "--extra-jars" = local.glue_job_script_jar_location
    "--TempDir" = local.glue_job_templocation
  }

  tags = {
    name = "demo-${local.env}-glue-job1"
  }
}

# [3] To create Glue start trigger
resource "aws_glue_trigger" "demo-glue-start-trigger" {
  name = "demo-${local.env}-start-trigger"
  description = "This is a sample Glue start trigger"
  type = "ON_DEMAND"
  workflow_name = aws_glue_workflow.demo-glue-workflow.name

  actions {
    job_name = aws_glue_job.aws_job1.name
  }
  tags = {
    name = "demo-${local.env}-start-trigger"
  }
}

例２：Scheduled Trigger

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/glue_trigger#scheduled-trigger

resource "aws_glue_trigger" "demo-trigger-scheduled-start" {
  name     = "demo-trigger-scheduled-start"
  description = "This is a sample Glue start trigger"
  schedule = "cron(15 12 * * ? *)"
  type = "SCHEDULED"
  enabled = local.env == "prod" ? true : false

  actions {
    job_name = aws_glue_job.aws_job1.name
  }
}

【３】Tips

１）タグ

* タグ管理って意外と面倒くさいが、
　以下のサイトでデフォルト的に指定できることを知った

https://dev.classmethod.jp/articles/terraform-aws-provider-default-tags/

【４】AWSへのデプロイ手順

* 以下の関連記事を参照のこと

Terraform ～ AWS S3 ～
https://dk521123.hatenablog.com/entry/2023/04/09/104204

参考文献

https://zenn.dev/pageo/articles/e4fc7c6a58188c

関連記事

Terraform ～ 環境構築編 ～
https://dk521123.hatenablog.com/entry/2023/04/05/000224
Terraform ～ AWS S3 ～
https://dk521123.hatenablog.com/entry/2023/04/09/104204
Terraform ～ AWS IAM ～
https://dk521123.hatenablog.com/entry/2023/04/12/214311
Terraform ～ AWS Secrets Manager ～
https://dk521123.hatenablog.com/entry/2023/04/11/152801
Terraform ～ Docker ～
https://dk521123.hatenablog.com/entry/2023/04/10/193239
AWS Glue ～ Scalaでの実装 ～
https://dk521123.hatenablog.com/entry/2023/03/17/000000
AWS Glue ～ 入門編 ～
https://dk521123.hatenablog.com/entry/2019/10/01/221926