■ はじめに

　以下の関連記事で、
Boto3を使って、AWS Glue componentを作成したが

https://dk521123.hatenablog.com/entry/2019/10/14/000000
https://dk521123.hatenablog.com/entry/2021/04/16/135558
https://dk521123.hatenablog.com/entry/2020/01/29/224525

Pulumi のよる AWS Glue component のデプロイできるらしいので
勉強しておく。

【０】Pulumi のよる Glueデプロイ

* 基本「aws.glue.<GlueComponent>」で、できる感じ
 => 思ったより、種類があってびっくりした。。。
* API仕様にサンプルもあり充実しているから、
　PulumiとAWS のそれぞれのAPI仕様を参照して作ればよさそう。

https://www.pulumi.com/registry/packages/aws/api-docs/glue/

【１】Job

* aws.glue.Job を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/job/

サンプル

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.glue.Job("demo-pyspark-job", {

    roleArn: aws_iam_role.example.arn,
    glueVersion: '3.0',
    command: {
        name: 'glueetl',
        scriptLocation: `s3://${aws_s3_bucket.example.bucket}/example.py`,
        pythonVersion: '3',
    },
    defaultArguments: {
       '--job-language': 'python',
       '--TempDir': 'python',
       '--job-bookmark-option': 'job-bookmark-disable',
    },
});

【２】Trigger

* aws.glue.Trigger を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/trigger/

サンプル

例１：type: "SCHEDULED"（開始トリガー）

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const demoTrigger = new aws.glue.Trigger("demo-trigger", {
    schedule: "cron(15 12 * * ? *)",
    type: "SCHEDULED",
    enabled: false,
    actions: [{
        jobName: aws_glue_job.example.name,
        arguments: {
          "--extra-files": "s3://your-s3-bucket/test/test1.yaml,s3://your-s3-bucket/test/test2.yaml",
          "--hello": "world"
        },
    }],
    tags: {
        Name: "demo-trigger",
    }
});

例２：ON_DEMAND / CONDITIONAL

* 後述「【４】Workflow」のサンプルを参照のこと

【３】Crawler

* aws.glue.Crawler を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/crawler/

サンプル

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.glue.Crawler("demo-crawler", {
    databaseName: aws_glue_catalog_database.example.name,
    role: aws_iam_role.example.arn,
    s3Targets: [{
        path: `s3://your-s3-bucket/xxx/xxx/`,
    }],
});

【４】Workflow

* aws.glue.Workflow を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/workflow/

サンプル

例１：Workflow とそれに紐づくTriggerの作成

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Workflow 作成
const demoWorkflow = new aws.glue.Workflow("demo-workflow", {});

// Trigger作成
// 開始Trigger作成
const demoTriggerStart = new aws.glue.Trigger("demo-trigger-start", {
    type: "ON_DEMAND",
    workflowName: demoWorkflow.name,
    actions: [{
        jobName: "demo-pyspark-job",
    }],
});
// 成功時のTrigger作成
const demoTriggerSuccess = new aws.glue.Trigger("demo-trigger-success", {
    type: "CONDITIONAL",
    workflowName: demoWorkflow.name,
    predicate: {
        conditions: [{
            jobName: "demo-pyspark-job",
            state: "SUCCEEDED",
        }],
    },
    actions: [{
        jobName: "demo-another-job",
    }],
});

例２：apply() を使用した場合

* apply() については、以下の関連記事を参照のこと

https://dk521123.hatenablog.com/entry/2022/03/18/115954

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Workflow 作成
const demoWorkflow = new aws.glue.Workflow("demo-workflow", {});

// Trigger作成
demoWorkflow.name.apply((workflowName) => {
  // 開始Trigger作成
  const demoTriggerStart = new aws.glue.Trigger("demo-trigger-start", {
      type: "ON_DEMAND",
      workflowName: workflowName,
      actions: [{
          jobName: "demo-pyspark-job",
      }],
  },
  // こんな感じで、親子関係を設定できる（pulumi up/preview時に階層表示になる）
  { parent: demoWorkflow });

  // 成功時のTrigger作成
  const demoTriggerSuccess = new aws.glue.Trigger("demo-trigger-success", {
      type: "CONDITIONAL",
      workflowName: workflowName,
      predicate: {
          conditions: [{
              jobName: "demo-pyspark-job",
              state: "SUCCEEDED",
          }],
      },
      actions: [{
          jobName: "demo-another-job",
      }],
  },
  { parent: demoWorkflow });
})

【５】Connection

* aws.glue.Connection を使う

https://www.pulumi.com/registry/packages/aws/api-docs/glue/connection/

サンプル

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.glue.Connection("demo-connection", {
    connectionType: "JDBC",
    connectionProperties: {
        JDBC_CONNECTION_URL: "jdbc:mysql://example.com/exampledatabase",
        PASSWORD: "examplepassword",
        USERNAME: "exampleusername",
    },
    physicalConnectionRequirements: {
        availabilityZone: aws_subnet.example.availability_zone,
        securityGroupIdLists: [aws_security_group.example.id],
        subnetId: aws_subnet.example.id,
    },
});

プログラムの超個人的なメモ

Memo for Programming.

【Pulumi】Pulumi ～ AWS Glue のデプロイ～

■ はじめに

目次

【０】Pulumi のよる Glueデプロイ

【１】Job

サンプル

【２】Trigger

サンプル

【３】Crawler

サンプル

【４】Workflow

サンプル

【５】Connection

サンプル

関連記事