■ はじめに

https://dk521123.hatenablog.com/entry/2023/12/17/000000

で、Apache Airflow の Variable について扱った。

今回は、AWS MWAAでの  Variable について扱う。

【１】公式ドキュメントのサンプル

まずは、公式ドキュメントのサンプル を見てみる

１）前提条件

* AWS CLI
* jq

jq コマンド～コマンドで JSON を扱う～
https://dk521123.hatenablog.com/entry/2020/02/01/000000

２）サンプル

https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html#example-airflow-cli-commands-bash

# See https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html#example-airflow-cli-commands-bash  
#
# Set, get or delete an Apache Airflow v2 variable
# You can use the following sample code to set, get or delete a variable
# in the format of 
# <script> <mwaa env name> get | set | delete <variable> <variable value> </variable> </variable>

[ $# -eq 0 ] && echo "Usage: $0 MWAA environment name " && exit

if [[ $2 == "" ]]; then
    dag="variables list"

elif  [ $2 == "get" ] ||  [ $2 == "delete" ] ||  [ $2 == "set" ]; then
    dag="variables $2 $3 $4 $5"

else
    echo "Not a valid command"
    exit 1
fi

CLI_JSON=$(aws mwaa --region $AWS_REGION create-cli-token --name $1) \
    && CLI_TOKEN=$(echo $CLI_JSON | jq -r '.CliToken') \
    && WEB_SERVER_HOSTNAME=$(echo $CLI_JSON | jq -r '.WebServerHostname') \
    && CLI_RESULTS=$(curl --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \
    --header "Authorization: Bearer $CLI_TOKEN" \
    --header "Content-Type: text/plain" \
    --data-raw "$dag" ) \
    && echo "Output:" \
    && echo $CLI_RESULTS | jq -r '.stdout' | base64 --decode \
    && echo "Errors:" \
    && echo $CLI_RESULTS | jq -r '.stderr' | base64 --decode

【２】AWS CLI

-- 該当部分抜粋
aws mwaa --region $AWS_REGION create-cli-token --name $1

１）aws mwaa create-cli-token

create-cli-tokenのコマンドを使用して
 Apache Airflow CLI トークンを作成

https://docs.aws.amazon.com/cli/latest/reference/mwaa/create-cli-token.html

# YOUR_ENVIRONMENT_NAME: 
aws mwaa create-cli-token --name YOUR_ENVIRONMENT_NAME

Response
https://docs.aws.amazon.com/mwaa/latest/API/API_CreateCliToken.html#API_CreateCliToken_ResponseSyntax

HTTP/1.1 200
Content-type: application/json

{
   "CliToken": "string",
   "WebServerHostname": "string"
}

補足：AWS CLI v2 のインストール

* AWS CLI v1 でもできるらしいが、念のため、v2 の設定方法も調べてみた

AWS CLI v2 インストール方法 for Linux
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

# Step1: Uninstall awscli v1
sudo yum remove awscli

# Step2: Download and install awscli v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

【３】REST API

-- 該当部分抜粋
-- $dag = e.g. variables set KEY1 VALUE1
curl --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \
    --header "Authorization: Bearer $CLI_TOKEN" \
    --header "Content-Type: text/plain" \
    --data-raw "$dag"

１）For variables

* 通常の Airflow の REST API との違う。
 => ってゆーかー、REST APIというより、REST+Airflow CLI で操作する感じ？

https://docs.aws.amazon.com/mwaa/latest/userguide/call-mwaa-apis-cli.html#create-cli-token-curl

curl --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \
  --header "Authorization: Bearer $CLI_TOKEN" \
  --header "Content-Type: text/plain" \
  --data-raw "＜ここにAirflow CLIをはいる＞"

例：variables set
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#variables

curl --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \
  --header "Authorization: Bearer $CLI_TOKEN" \
  --header "Content-Type: text/plain" \
  --data-raw "variables set KEY1 VALUE1"

参考文献

https://medium.com/dnx-labs/how-to-use-apache-airflow-cli-with-amazon-mwaa-a773edfa0a45

プログラムの超個人的なメモ

Memo for Programming.

【Airflow】MWAA ～ Variable ～

■ はじめに

目次

【１】公式ドキュメントのサンプル

１）前提条件

２）サンプル

【２】AWS CLI

１）aws mwaa create-cli-token

補足：AWS CLI v2 のインストール

【３】REST API

１）For variables

参考文献

関連記事