【Pulumi】Pulumi ~ AWS Glue のデプロイ ~

■ はじめに

 以下の関連記事で、
Boto3を使って、AWS Glue componentを作成したが

https://dk521123.hatenablog.com/entry/2019/10/14/000000
https://dk521123.hatenablog.com/entry/2021/04/16/135558
https://dk521123.hatenablog.com/entry/2020/01/29/224525

Pulumi のよる AWS Glue component のデプロイできるらしいので
勉強しておく。

目次

【0】Pulumi のよる Glueデプロイ
【1】Job
【2】Trigger
【3】Crawler
【4】Workflow
【5】Connection

【0】Pulumi のよる Glueデプロイ

* 基本「aws.glue.<GlueComponent>」で、できる感じ
 => 思ったより、種類があってびっくりした。。。
* API仕様にサンプルもあり充実しているから、
 PulumiとAWS のそれぞれのAPI仕様を参照して作ればよさそう。

https://www.pulumi.com/registry/packages/aws/api-docs/glue/

【1】Job

* aws.glue.Job を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/job/

サンプル

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.glue.Job("demo-pyspark-job", {

    roleArn: aws_iam_role.example.arn,
    glueVersion: '3.0',
    command: {
        name: 'glueetl',
        scriptLocation: `s3://${aws_s3_bucket.example.bucket}/example.py`,
        pythonVersion: '3',
    },
    defaultArguments: {
       '--job-language': 'python',
       '--TempDir': 'python',
       '--job-bookmark-option': 'job-bookmark-disable',
    },
});

【2】Trigger

* aws.glue.Trigger を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/trigger/

サンプル

例1:type: "SCHEDULED"(開始トリガー)

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const demoTrigger = new aws.glue.Trigger("demo-trigger", {
    schedule: "cron(15 12 * * ? *)",
    type: "SCHEDULED",
    enabled: false,
    actions: [{
        jobName: aws_glue_job.example.name,
        arguments: {"--extra-files": "s3", "--hello": "world"},
    }],
    tags: {
        Name: "demo-trigger",
    }
});

例2:ON_DEMAND / CONDITIONAL

* 後述「【4】Workflow」のサンプルを参照のこと

【3】Crawler

* aws.glue.Crawler を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/crawler/

サンプル

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.glue.Crawler("demo-crawler", {
    databaseName: aws_glue_catalog_database.example.name,
    role: aws_iam_role.example.arn,
    s3Targets: [{
        path: `s3://your-s3-bucket/xxx/xxx/`,
    }],
});

【4】Workflow

* aws.glue.Workflow を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/workflow/

サンプル

例1:Workflow と それに紐づくTriggerの作成

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Workflow 作成
const demoWorkflow = new aws.glue.Workflow("demo-workflow", {});

// Trigger作成
// 開始Trigger作成
const demoTriggerStart = new aws.glue.Trigger("demo-trigger-start", {
    type: "ON_DEMAND",
    workflowName: demoWorkflow.name,
    actions: [{
        jobName: "demo-pyspark-job",
    }],
});
// 成功時のTrigger作成
const demoTriggerSuccess = new aws.glue.Trigger("demo-trigger-success", {
    type: "CONDITIONAL",
    workflowName: demoWorkflow.name,
    predicate: {
        conditions: [{
            jobName: "demo-pyspark-job",
            state: "SUCCEEDED",
        }],
    },
    actions: [{
        jobName: "demo-another-job",
    }],
});

例2:apply() を使用した場合

* apply() については、以下の関連記事を参照のこと

https://dk521123.hatenablog.com/entry/2022/03/18/115954

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Workflow 作成
const demoWorkflow = new aws.glue.Workflow("demo-workflow", {});

// Trigger作成
demoWorkflow.name.apply((workflowName) => {
  // 開始Trigger作成
  const demoTriggerStart = new aws.glue.Trigger("demo-trigger-start", {
      type: "ON_DEMAND",
      workflowName: workflowName,
      actions: [{
          jobName: "demo-pyspark-job",
      }],
  },
  // こんな感じで、親子関係を設定できる(pulumi up/preview時に階層表示になる)
  { parent: demoWorkflow });

  // 成功時のTrigger作成
  const demoTriggerSuccess = new aws.glue.Trigger("demo-trigger-success", {
      type: "CONDITIONAL",
      workflowName: workflowName,
      predicate: {
          conditions: [{
              jobName: "demo-pyspark-job",
              state: "SUCCEEDED",
          }],
      },
      actions: [{
          jobName: "demo-another-job",
      }],
  },
  { parent: demoWorkflow });
})

【5】Connection

* aws.glue.Connection を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/connection/

サンプル

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.glue.Connection("demo-connection", {
    connectionType: "JDBC",
    connectionProperties: {
        JDBC_CONNECTION_URL: "jdbc:mysql://example.com/exampledatabase",
        PASSWORD: "examplepassword",
        USERNAME: "exampleusername",
    },
    physicalConnectionRequirements: {
        availabilityZone: aws_subnet.example.availability_zone,
        securityGroupIdLists: [aws_security_group.example.id],
        subnetId: aws_subnet.example.id,
    },
});

関連記事

Pulumi ~ 基礎知識編 ~
https://dk521123.hatenablog.com/entry/2021/10/23/025230
Pulumi ~ 入門編 / Hello World in Local/k8s
https://dk521123.hatenablog.com/entry/2022/03/07/233752
Pulumi ~ 入門編 / Hello World in AWS
https://dk521123.hatenablog.com/entry/2022/03/11/184041
Pulumi ~ 基本編 / Output
https://dk521123.hatenablog.com/entry/2022/03/18/115954
Pulumi ~ AWS S3 / KMS のデプロイ ~
https://dk521123.hatenablog.com/entry/2022/03/03/095415
Pulumi ~ AWSリソース情報を取得する ~
https://dk521123.hatenablog.com/entry/2022/03/22/212828
AWS Glue ~ Boto3 / 入門編 ~
https://dk521123.hatenablog.com/entry/2019/10/14/000000
AWS Glue ~ Boto3 / クローラ編 ~
https://dk521123.hatenablog.com/entry/2021/04/16/135558
AWS Glue ~ Boto3 / Glue connection編 ~
https://dk521123.hatenablog.com/entry/2020/01/29/224525
AWS Glue ~ 入門編 ~
https://dk521123.hatenablog.com/entry/2019/10/01/221926