■ はじめに

MWAA Local で AWSに繋げるようにする必要ができたのでメモ。
あと、個人的なことで作業が簡単になるように
MWAA Local をちょこっと改造する

【１】MWAA Local で AWSに繋げるようにするには

* AWS 接続関連の環境変数（AWS_ACCESS_KEY_IDなど）を追加する

Case１：docker/config/.env.localrunner に追加

* コメントのExample ... を読むと分かる通り、こちらに追加するのが良さそう

https://github.com/aws/aws-mwaa-local-runner/blob/b240667eeb10eb27182a064c015c0e90db35ad01/docker/config/.env.localrunner

# Any environment variables set in this .env.localrunner file will be set in local-runner on start
# Example environment variables using temporary security credentials
# AWS_ACCESS_KEY_ID=XXXXXXXXXX
# AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYY
# AWS_SESSION_TOKEN=ZZZZZZZZZZ 

# to change default password you'll need to delete the db-data folder (when running locally)
DEFAULT_PASSWORD="test"
S3_DAGS_PATH=""
S3_PLUGINS_PATH=""
S3_REQUIREMENTS_PATH=""

Case２：docker-compose-local.yml の「environment」配下に追加

* docker-compose-local.yml の「environment」配下に
　AWS 接続関連の環境変数（AWS_ACCESS_KEY_IDなど）を追加する

https://github.com/aws/aws-mwaa-local-runner/blob/b240667eeb10eb27182a064c015c0e90db35ad01/docker/docker-compose-local.yml#L23

        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            # ★追加★
            - AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
            - AWS_SECRET_ACCESS_KEY=ASIAIOSFODNN7EXAMPLE
            - AWS_SESSION_TOKEN=AQoDYXdzEJr...PxRfiCYEXAMPLEKEY
        logging:

【２】MWAA Local 開始時に環境変数を書き換える

この記事の本題。

https://github.com/aws/aws-mwaa-local-runner#local-runner

新規シェル「set-aws-env.sh」を追加し
./set-aws-env.sh "<環境変数>"
で置き換えられるように修正。

docker/config/.env.localrunner
https://github.com/aws/aws-mwaa-local-runner/blob/b240667eeb10eb27182a064c015c0e90db35ad01/docker/config/.env.localrunner

# Any environment variables set in this .env.localrunner file will be set in local-runner on start
# Example environment variables using temporary security credentials
# AWS_ACCESS_KEY_ID=XXXXXXXXXX
# AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYY
# AWS_SESSION_TOKEN=ZZZZZZZZZZ 

# to change default password you'll need to delete the db-data folder (when running locally)
DEFAULT_PASSWORD="test"
S3_DAGS_PATH=""
S3_PLUGINS_PATH=""
S3_REQUIREMENTS_PATH=""

# Add
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=

set-aws-env.sh
https://github.com/aws/aws-mwaa-local-runner/blob/v2.6.3/mwaa-local-env#L59

# Custom
replace_aws_environmental_variables() {
  # e.g. "export AWS_ACCESS_KEY_ID=[XXX];export AWS_SECRET_ACCESS_KEY=[XXX];export AWS_SESSION_TOKEN=[XXX]”
  aws_env="${1}"

  AWS_ACCESS_KEY_ID=""
  AWS_SECRET_ACCESS_KEY=""
  AWS_SESSION_TOKEN=""

  # Parse AWS environmental variables
  if [ -n "$aws_env" ]; then
    #AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE;AWS_SECRET_ACCESS_KEY=ASIAIOSFODNN7EXAMPLE;AWS_SESSION_TOKEN=AQoDYXdzEJr...PxRfiCYEXAMPLEKEY
    aws_env_without_export=${aws_env//export /}
    echo "[DEBUG] aws_env_without_export = $aws_env_without_export"

    aws_params=(${aws_env_without_export//;/ })
    echo "[DEBUG] aws_params = $aws_params"

    for aws_param in "${aws_params[@]}"
    do
      key=`echo "${aws_param}"| awk -F'=' '{print $1}'`
      case $key in
      AWS_ACCESS_KEY_ID)
        AWS_ACCESS_KEY_ID=`echo "${aws_param}"| awk -F'=' '{print $2}'`
        ;;
      AWS_SECRET_ACCESS_KEY)
        AWS_SECRET_ACCESS_KEY=`echo "${aws_param}"| awk -F'=' '{print $2}'`
        ;;
      AWS_SESSION_TOKEN)
        AWS_SESSION_TOKEN=`echo "${aws_param}"| awk -F'=' '{print $2}'`
        ;;
      *)
        echo "[WARN] key_value=${key_value}"
        ;;
      esac
    done
  else
    echo "No AWS param. So clear the AWS parameters."
  fi

  # Replace the docker compose file
  rm -f temp.env

  while IFS= read -r line; do
    case $line in
      *AWS_ACCESS_KEY_ID=*)
        echo "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" >> temp.env
        ;;
      *AWS_SECRET_ACCESS_KEY=*)
        echo "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" >> temp.env
        ;;
      *AWS_SESSION_TOKEN=*)
        echo "AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}" >> temp.env
        ;;
      *)
        echo "$line" >> temp.env
        ;;
    esac
  done < docker/config/.env.localrunner

  cp temp.env docker/config/.env.localrunner
  rm -f temp.env
}

# Custom
replace_aws_environmental_variables "$1"

使い方

# 環境変数に置き換える
./set-aws-env.sh "export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE;export AWS_SECRET_ACCESS_KEY=ASIAIOSFODNN7EXAMPLE;export AWS_SESSION_TOKEN=AQoDYXdzEJr...PxRfiCYEXAMPLEKEY”

# 環境変数をクリアにする
./set-aws-env.sh

MWAA Local ～環境構築編～
https://dk521123.hatenablog.com/entry/2021/11/05/233309
MWAA Local ～環境構築編 / Docker compose ～
https://dk521123.hatenablog.com/entry/2021/11/07/132014
MWAA Local ～入門編～
https://dk521123.hatenablog.com/entry/2023/10/21/233404
Apache Airflow ～入門編～
https://dk521123.hatenablog.com/entry/2021/07/24/233012

シェル

シェル～文字列置換～
https://dk521123.hatenablog.com/entry/2023/10/25/000000
シェル～文字列抽出あれこれ～
https://dk521123.hatenablog.com/entry/2021/08/03/160901
シェルで split するには
https://dk521123.hatenablog.com/entry/2021/09/02/000000

プログラムの超個人的なメモ

Memo for Programming.

【Airflow】MWAA Local ～ AWSに繋げるようにするには～

■ はじめに

目次

【１】MWAA Local で AWSに繋げるようにするには

Case１：docker/config/.env.localrunner に追加

Case２：docker-compose-local.yml の「environment」配下に追加

【２】MWAA Local 開始時に環境変数を書き換える

使い方

関連記事

シェル