◾️はじめに

https://dk521123.hatenablog.com/entry/2025/12/19/233012

の続き。

以下のようなサービス構成を考えたので、
Terraformを使って、以下の構成ができるように組んでいく
（Glue Jobの正式な内容は、別途作成する）

Terraformも大分久しぶりに触るので、リハビリしながらやっていく

サービス構成

Items	Choose
IaC	Terraform
Cloud	AWS
Platform	AWS Glue
Language	Python
Tool Library	Python SDV (Synthetic Data Vault)

【１】フォルダ構成
【２】サンプル
　０）環境ごとの設定ファイル
　１）variables.tf
　２）provider.tf
　３）main.tf
　４）modules/glue/main.tf
　５）modules/glue/variables.tf
　６）glue_jobs/synthetic_files_generator.py
【３】デプロイ
　１）DEV環境へのコマンド例

【１】フォルダ構成

└── modules
    ├── glue
        ├── main.tf
        └── variables.tf
└── glue_jobs
    └──  synthetic_files_generator.py << ここは今は適当
├── main.tf
├── provider.tf
├── dev.tfbackend
├── provider.tf
└── variables.tf

１）前提条件

* 今回は、以下が既存のものを使う（variables.tfで入力させる）
 + s3
 + IAM

【２】サンプル

０）環境ごとの設定ファイル

* いちいち実行ごとに入力するのは面倒なので、設定ファイル化しておく

dev.tfbackend

bucket = "your-s3-bucket"
key = "tf-for-synthtic/dev/terraform.tfstate"
region = "us-west-2"

dev.tfvars

env = "dev"

# For Glue
glue_role_arn = "arn:aws:iam:00000000:role/your-glue-job-role-arn"
glue_job_script_source = "glue_jobs/synthetic_files_generator.py"
glue_job_script_location_bucket = "your-bucket-name"
glue_job_script_location_key = "demo/glue_jobs/synthetic_files_generator.py"
glue_job_connections = ["your-glue-connection"]

１）variables.tf

variable "aws_region" {
  description = "AWS region to deploy resources"
  type = string
  default = "us-west-2"
}

variable "env" {
  description = "Environment"
  type = string
  default = "dev"

  validation {
    condition     = contains(["dev", "stage", "prod"], var.env)
    error_message = "Only dev / stage / prod"
  }
}

variable "project_name" {
  description = "Project name"
  type = string
  default = "synthetic_files"
}

variable "glue_role_arn" {
  description = "AWS Glue job role ARN"
  type = string
}

variable "glue_version" {
  description = "AWS Glue job version"
  type = string
  default = "5.0"
}

variable "glue_job_script_source" {
  description = "Python script path of AWS Glue job script"
  type = string
  default = "glue_jobs/synthetic_files_generator.py"
}

variable "glue_job_script_location_bucket" {
  description = "s3 bucket of AWS Glue job script"
  type = string
}

variable "glue_job_script_location_key" {
  description = "s3 bucket key of AWS Glue job script"
  type = string
}

variable "glue_job_connections" {
  description = "AWS Glue job connections"
  type = list(string)
}

２）provider.tf

# Configure the AWS Provider
provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      env = var.env
    }
  }
}

３）main.tf

module "demo_generate_synthetic_data_glue_module" {
  source = "./modules/glue"
  env = var.env
  glue_role_arn = var.glue_role_arn
  glue_job_script_source = var.glue_job_script_source
  glue_job_script_location_bucket = var.glue_job_script_location_bucket
  glue_job_script_location_key = var.glue_job_script_location_key
  glue_job_connections = var.glue_job_connections
}

４）modules/glue/main.tf

# [1] To create Glue workflow
resource "aws_glue_workflow" "demo_generate_synthetic_data_workflow" {
  name = "demo-${var.env}-generate-synthetic-data-workflow"
  description = "This is the glue workflow to generate synthetic dada."
  tags = {
    name = "demo-${var.env}-generate-synthetic-data-workflow"
  }
}

# [2-1] To upload Glue script to s3
resource "aws_s3_object" "clanyan_source" {
  source = var.glue_job_script_source
  bucket = var.glue_job_script_location_bucket
  key    = var.glue_job_script_location_key
  etag   = filemd5(var.glue_job_script_source)
}

# [2-2] To create Glue Job
resource "aws_glue_job" "demo_generate_synthetic_data_job" {
  name = "demo-${var.env}-generate-synthetic-data-job"
  description = "This is the glue job to generate synthetic dada."
  role_arn = var.glue_role_arn
  glue_version = var.glue_version
  max_retries = var.env == "dev" ? 0 : 3
  timeout = var.env == "dev" ? 60 : 2880
  worker_type = var.env == "dev" ? "G.2X" : "G.4X"
  number_of_workers = var.env == "dev" ? 2 : 10

  command {
    name = "glueetl"
    script_location = "s3://${var.glue_job_script_location_bucket}${var.glue_job_script_location_key}"
  }
  connections = var.glue_job_connections

  # Glue Job parameters
  default_arguments = {
    "--job-language" = "python"

    # 今回はなくていいけど、将来的には必要
    # "--additional-python-modules" = "sdv==1.32.0"

    "--TempDir"          = "s3://${var.glue_job_script_location_bucket}/temp/"
    "--continuous-log-logGroup"          = "/aws-glue/jobs"
    "--enable-continuous-cloudwatch-log" = "true"
    "--enable-continuous-log-filter"     = "true"
    "--enable-metrics"                   = ""
    "--enable-auto-scaling"              = "true"
  }

  tags = {
    name = "demo-${var.env}-generate-synthetic-data-job"
  }
}

# [3] To create Glue start trigger
resource "aws_glue_trigger" "demo_generate_synthetic_data_start_trigger" {
  name = "demo-${var.env}-generate-synthetic-data-start-trigger"
  description = "This is the glue start trigger to generate synthetic dada."
  type = "ON_DEMAND"
  workflow_name = aws_glue_workflow.demo_generate_synthetic_data_workflow.name

  actions {
    job_name = aws_glue_job.demo_generate_synthetic_data_job.name
  }
  tags = {
    name = "demo-${var.env}-generate-synthetic-data-start-trigger"
  }
}

５）modules/glue/variables.tf

variable "env" {
  description = "Environment"
  type = string
}

variable "glue_role_arn" {
  description = "AWS Glue job role ARN"
  type = string
}

variable "glue_version" {
  description = "AWS Glue job version"
  type = string
  default = "5.0"
}

variable "glue_job_script_source" {
  description = "Python script path of AWS Glue job script"
  type = string
  default = "glue_jobs/synthetic_files_generator.py"
}

variable "glue_job_script_location_bucket" {
  description = "s3 bucket of AWS Glue job script"
  type = string
}

variable "glue_job_script_location_key" {
  description = "s3 bucket key of AWS Glue job script"
  type = string
}

variable "glue_job_connections" {
  description = "AWS Glue job connections"
  type = list(string)
}

６）glue_jobs/synthetic_files_generator.py

# ひとまず、今回は、Hello worldを実行させるだけ
import sys
import boto3
from awsglue.utils import getResolvedOptions

glue_client = boto3.client("glue")
args = getResolvedOptions(sys.argv, ['WORKFLOW_NAME', 'WORKFLOW_RUN_ID'])
workflow_name = args['WORKFLOW_NAME']
workflow_run_id = args['WORKFLOW_RUN_ID']
workflow_params = glue_client.get_workflow_run_properties(
  Name=workflow_name, RunId=workflow_run_id)["RunProperties"]

# 設定
workflow_params = { 'key_1': 'hello_world' }
glue_client.put_workflow_run_properties(
  Name=workflow_name, RunId=workflow_run_id, RunProperties=workflow_params)
print("Done")

【３】デプロイ

１）DEV環境へのコマンド例

# 初期化
terraform init -backend-config=dev.tfbackend

# チェック
terraform plan -var-file=dev.tfvars

# デプロイ
terraform apply -var-file=dev.tfvars --auto-approve

ランキング参加中

プログラミング

プログラムの超個人的なメモ

Memo for Programming.

【Terraform】AWS上に合成データ生成システムを構築する〜インフラ編〜

◾️はじめに

目次

【１】フォルダ構成

１）前提条件

【２】サンプル

０）環境ごとの設定ファイル

１）variables.tf

２）provider.tf

３）main.tf

４）modules/glue/main.tf

５）modules/glue/variables.tf

６）glue_jobs/synthetic_files_generator.py

【３】デプロイ

１）DEV環境へのコマンド例

関連記事