Skip to main content

Pulumi

This page provides introduction to Pulumi.

warning

These are my personal notes and not an official guide for creating AWS resources using Pulumi. Following these instructions may incur AWS costs.

Overview​

Pulumi is an open-source infrastructure-as-code (IaC) platform that enables developers to define, deploy, and manage cloud resources using general-purpose programming languages like Python, TypeScript, Go etc.

Terminology​

Below are some of the terminologies:

  • Resource: Any object or thing managed by pulumi.
  • Project: Project is folder containing pulumi code. It contains pulumi yaml file.
  • Program: Program exists inside of project and manages the resources.
  • Stack: Configured instance of pulumi program. You can apply same pulumi program to different environments using different configurations.
  • Provider: Provider interacts with an infrastructure on pulumi's behalf. It has two components, language SDK and plugins.
  • Output: Output is special value which is returned to you by infrastructure.
  • State: State is where pulumi stores the results of it's execution.

Installing Pulumi​

brew install pulumi

Logging In​

  • Logging in sets up your state.
  • Logging in is a global operation.
  • When logging in, you select your backend.
# to login into the local file system
# this command will make pulumi store the state in local file system
pulumi login file://

Bootstrap Project​

Click here to see pulumi-examples project on github.

# create a directory
mkdir pulumi-examples
cd pulumi-examples

# run pulumi new & select aws-python project
pulumi new
# o/p:
# Project name (pulumi-examples):  
# Project description (A minimal AWS Python Pulumi program):  
# Created project 'pulumi-examples'

# Stack name (dev):  
# Enter your passphrase to protect config/secrets:  
# Re-enter your passphrase to confirm:  
# Created stack 'dev'

# The toolchain to use for installing dependencies and running the program pip
# The AWS region to deploy into (aws:region) (us-east-1):  
# Saved config

# Installing dependencies...

Setup Dev Stack in LocalStack​

Start LocalStack by following steps here.

Update the Pulumi.dev.yaml configuration file for the dev stack to point AWS service endpoints to localstack as below:

config:
  aws:accessKey: test
  aws:s3UsePathStyle: "true"
  aws:secretKey: test
  aws:skipCredentialsValidation: "true"
  aws:skipRequestingAccountId: "true"
  aws:endpoints:
    - accessanalyzer: http://localhost:4566
    - account: http://localhost:4566
    - acm: http://localhost:4566
    - acmpca: http://localhost:4566
    - amg: http://localhost:4566
    - amp: http://localhost:4566
    - amplify: http://localhost:4566
    - apigateway: http://localhost:4566
    - apigatewayv2: http://localhost:4566
    - appautoscaling: http://localhost:4566
    - appconfig: http://localhost:4566
    - appfabric: http://localhost:4566
    - appflow: http://localhost:4566
    - appintegrations: http://localhost:4566
    - appintegrationsservice: http://localhost:4566
    - applicationautoscaling: http://localhost:4566
    - applicationinsights: http://localhost:4566
    - appmesh: http://localhost:4566
    - appregistry: http://localhost:4566
    - apprunner: http://localhost:4566
    - appstream: http://localhost:4566
    - appsync: http://localhost:4566
    - athena: http://localhost:4566
    - auditmanager: http://localhost:4566
    - autoscaling: http://localhost:4566
    - autoscalingplans: http://localhost:4566
    - backup: http://localhost:4566
    - batch: http://localhost:4566
    - beanstalk: http://localhost:4566
    - bedrock: http://localhost:4566
    - bedrockagent: http://localhost:4566
    - budgets: http://localhost:4566
    - ce: http://localhost:4566
    - chime: http://localhost:4566
    - chimesdkmediapipelines: http://localhost:4566
    - chimesdkvoice: http://localhost:4566
    - cleanrooms: http://localhost:4566
    - cloud9: http://localhost:4566
    - cloudcontrol: http://localhost:4566
    - cloudcontrolapi: http://localhost:4566
    - cloudformation: http://localhost:4566
    - cloudfront: http://localhost:4566
    - cloudfrontkeyvaluestore: http://localhost:4566
    - cloudhsm: http://localhost:4566
    - cloudhsmv2: http://localhost:4566
    - cloudsearch: http://localhost:4566
    - cloudtrail: http://localhost:4566
    - cloudwatch: http://localhost:4566
    - cloudwatchevents: http://localhost:4566
    - cloudwatchevidently: http://localhost:4566
    - cloudwatchlog: http://localhost:4566
    - cloudwatchlogs: http://localhost:4566
    - cloudwatchobservabilityaccessmanager: http://localhost:4566
    - cloudwatchrum: http://localhost:4566
    - codeartifact: http://localhost:4566
    - codebuild: http://localhost:4566
    - codecatalyst: http://localhost:4566
    - codecommit: http://localhost:4566
    - codedeploy: http://localhost:4566
    - codeguruprofiler: http://localhost:4566
    - codegurureviewer: http://localhost:4566
    - codepipeline: http://localhost:4566
    - codestarconnections: http://localhost:4566
    - codestarnotifications: http://localhost:4566
    - cognitoidentity: http://localhost:4566
    - cognitoidentityprovider: http://localhost:4566
    - cognitoidp: http://localhost:4566
    - comprehend: http://localhost:4566
    - computeoptimizer: http://localhost:4566
    - config: http://localhost:4566
    - configservice: http://localhost:4566
    - connect: http://localhost:4566
    - connectcases: http://localhost:4566
    - controltower: http://localhost:4566
    - costandusagereportservice: http://localhost:4566
    - costexplorer: http://localhost:4566
    - costoptimizationhub: http://localhost:4566
    - cur: http://localhost:4566
    - customerprofiles: http://localhost:4566
    - databasemigration: http://localhost:4566
    - databasemigrationservice: http://localhost:4566
    - dataexchange: http://localhost:4566
    - datapipeline: http://localhost:4566
    - datasync: http://localhost:4566
    - dax: http://localhost:4566
    - deploy: http://localhost:4566
    - detective: http://localhost:4566
    - devicefarm: http://localhost:4566
    - directconnect: http://localhost:4566
    - directoryservice: http://localhost:4566
    - dlm: http://localhost:4566
    - dms: http://localhost:4566
    - docdb: http://localhost:4566
    - docdbelastic: http://localhost:4566
    - ds: http://localhost:4566
    - dynamodb: http://localhost:4566
    - ec2: http://localhost:4566
    - ecr: http://localhost:4566
    - ecrpublic: http://localhost:4566
    - ecs: http://localhost:4566
    - efs: http://localhost:4566
    - eks: http://localhost:4566
    - elasticache: http://localhost:4566
    - elasticbeanstalk: http://localhost:4566
    - elasticloadbalancing: http://localhost:4566
    - elasticloadbalancingv2: http://localhost:4566
    - elasticsearch: http://localhost:4566
    - elasticsearchservice: http://localhost:4566
    - elastictranscoder: http://localhost:4566
    - elb: http://localhost:4566
    - elbv2: http://localhost:4566
    - emr: http://localhost:4566
    - emrcontainers: http://localhost:4566
    - emrserverless: http://localhost:4566
    - es: http://localhost:4566
    - eventbridge: http://localhost:4566
    - events: http://localhost:4566
    - evidently: http://localhost:4566
    - finspace: http://localhost:4566
    - firehose: http://localhost:4566
    - fis: http://localhost:4566
    - fms: http://localhost:4566
    - fsx: http://localhost:4566
    - gamelift: http://localhost:4566
    - glacier: http://localhost:4566
    - globalaccelerator: http://localhost:4566
    - glue: http://localhost:4566
    - grafana: http://localhost:4566
    - greengrass: http://localhost:4566
    - groundstation: http://localhost:4566
    - guardduty: http://localhost:4566
    - healthlake: http://localhost:4566
    - iam: http://localhost:4566
    - identitystore: http://localhost:4566
    - imagebuilder: http://localhost:4566
    - inspector: http://localhost:4566
    - inspector2: http://localhost:4566
    - inspectorv2: http://localhost:4566
    - internetmonitor: http://localhost:4566
    - iot: http://localhost:4566
    - iotanalytics: http://localhost:4566
    - iotevents: http://localhost:4566
    - ivs: http://localhost:4566
    - ivschat: http://localhost:4566
    - kafka: http://localhost:4566
    - kafkaconnect: http://localhost:4566
    - kendra: http://localhost:4566
    - keyspaces: http://localhost:4566
    - kinesis: http://localhost:4566
    - kinesisanalytics: http://localhost:4566
    - kinesisanalyticsv2: http://localhost:4566
    - kinesisvideo: http://localhost:4566
    - kms: http://localhost:4566
    - lakeformation: http://localhost:4566
    - lambda: http://localhost:4566
    - launchwizard: http://localhost:4566
    - lex: http://localhost:4566
    - lexmodelbuilding: http://localhost:4566
    - lexmodelbuildingservice: http://localhost:4566
    - lexmodels: http://localhost:4566
    - lexmodelsv2: http://localhost:4566
    - lexv2models: http://localhost:4566
    - licensemanager: http://localhost:4566
    - lightsail: http://localhost:4566
    - location: http://localhost:4566
    - locationservice: http://localhost:4566
    - logs: http://localhost:4566
    - lookoutmetrics: http://localhost:4566
    - m2: http://localhost:4566
    - macie2: http://localhost:4566
    - managedgrafana: http://localhost:4566
    - mediaconnect: http://localhost:4566
    - mediaconvert: http://localhost:4566
    - medialive: http://localhost:4566
    - mediapackage: http://localhost:4566
    - mediapackagev2: http://localhost:4566
    - mediastore: http://localhost:4566
    - memorydb: http://localhost:4566
    - mq: http://localhost:4566
    - msk: http://localhost:4566
    - mwaa: http://localhost:4566
    - neptune: http://localhost:4566
    - networkfirewall: http://localhost:4566
    - networkmanager: http://localhost:4566
    - oam: http://localhost:4566
    - opensearch: http://localhost:4566
    - opensearchingestion: http://localhost:4566
    - opensearchserverless: http://localhost:4566
    - opensearchservice: http://localhost:4566
    - opsworks: http://localhost:4566
    - organizations: http://localhost:4566
    - osis: http://localhost:4566
    - outposts: http://localhost:4566
    - pcaconnectorad: http://localhost:4566
    - pinpoint: http://localhost:4566
    - pipes: http://localhost:4566
    - polly: http://localhost:4566
    - pricing: http://localhost:4566
    - prometheus: http://localhost:4566
    - prometheusservice: http://localhost:4566
    - qbusiness: http://localhost:4566
    - qldb: http://localhost:4566
    - quicksight: http://localhost:4566
    - ram: http://localhost:4566
    - rbin: http://localhost:4566
    - rds: http://localhost:4566
    - recyclebin: http://localhost:4566
    - redshift: http://localhost:4566
    - redshiftdata: http://localhost:4566
    - redshiftdataapiservice: http://localhost:4566
    - redshiftserverless: http://localhost:4566
    - rekognition: http://localhost:4566
    - resourceexplorer2: http://localhost:4566
    - resourcegroups: http://localhost:4566
    - resourcegroupstagging: http://localhost:4566
    - resourcegroupstaggingapi: http://localhost:4566
    - rolesanywhere: http://localhost:4566
    - route53: http://localhost:4566
    - route53domains: http://localhost:4566
    - route53recoverycontrolconfig: http://localhost:4566
    - route53recoveryreadiness: http://localhost:4566
    - route53resolver: http://localhost:4566
    - rum: http://localhost:4566
    - s3: http://localhost:4566
    - s3api: http://localhost:4566
    - s3control: http://localhost:4566
    - s3outposts: http://localhost:4566
    - sagemaker: http://localhost:4566
    - scheduler: http://localhost:4566
    - schemas: http://localhost:4566
    - sdb: http://localhost:4566
    - secretsmanager: http://localhost:4566
    - securityhub: http://localhost:4566
    - securitylake: http://localhost:4566
    - serverlessapplicationrepository: http://localhost:4566
    - serverlessapprepo: http://localhost:4566
    - serverlessrepo: http://localhost:4566
    - servicecatalog: http://localhost:4566
    - servicecatalogappregistry: http://localhost:4566
    - servicediscovery: http://localhost:4566
    - servicequotas: http://localhost:4566
    - ses: http://localhost:4566
    - sesv2: http://localhost:4566
    - sfn: http://localhost:4566
    - shield: http://localhost:4566
    - signer: http://localhost:4566
    - simpledb: http://localhost:4566
    - sns: http://localhost:4566
    - sqs: http://localhost:4566
    - ssm: http://localhost:4566
    - ssmcontacts: http://localhost:4566
    - ssmincidents: http://localhost:4566
    - ssmsap: http://localhost:4566
    - sso: http://localhost:4566
    - ssoadmin: http://localhost:4566
    - stepfunctions: http://localhost:4566
    - storagegateway: http://localhost:4566
    - sts: http://localhost:4566
    - swf: http://localhost:4566
    - synthetics: http://localhost:4566
    - timestreamwrite: http://localhost:4566
    - transcribe: http://localhost:4566
    - transcribeservice: http://localhost:4566
    - transfer: http://localhost:4566
    - verifiedpermissions: http://localhost:4566
    - vpclattice: http://localhost:4566
    - waf: http://localhost:4566
    - wafregional: http://localhost:4566
    - wafv2: http://localhost:4566
    - wellarchitected: http://localhost:4566
    - worklink: http://localhost:4566
    - workspaces: http://localhost:4566
    - xray: http://localhost:4566

Run Dev Stack in LocalStack​

# selecting pulumi dev stack
pulumi stack select dev

# previewing pulumi stack
pulumi preview
# o/p:
#      Type                 Name                 Plan       
#  +   pulumi:pulumi:Stack  pulumi-examples-dev  create     
# +   └─ aws:s3:BucketV2   my-bucket            create     
# 
# Outputs:
#     bucket_name: output<string>
# 
# Resources:
#     + 2 to create

# running pulumi dev stack, which will create infrastrcture in LocalStack
pulumi up
# o/p:
#      Type                 Name                 Plan       
#  +   pulumi:pulumi:Stack  pulumi-examples-dev  create     
#  +   └─ aws:s3:BucketV2   my-bucket            create     
# 
# Outputs:
#     bucket_name: output<string>
# 
# Resources:
#     + 2 to create
# 
# Do you want to perform this update? yes
# Updating (dev):
#      Type                 Name                 Status              
#  +   pulumi:pulumi:Stack  pulumi-examples-dev  created (12s)       
#  +   └─ aws:s3:BucketV2   my-bucket            created (0.15s)     
# 
# Outputs:
#     bucket_name: "my-bucket-4dbacbf"
# 
# Resources:
#     + 2 created
# 
# Duration: 14s
info

Since we are using a local backend, the state of the infrastructure created above will be stored on your local machine in the .pulumi folder. Be sure to add .pulumi/* to your .gitignore file to prevent it from being checked into version control.

After running 'pulumi preview' or 'pulumi up', pulumi will add an encryptionSalt to the Pulumi.dev.yaml file. Be sure to remove it, as it considered sensitive information. The encryptionSalt is used to encode and decode any secrets added to the Pulumi.dev.yaml configuration file.

For production environments, it’s recommended to use an AWS S3 bucket for state management and AWS KMS for secret management.

Validate Dev Stack in LocalStack​

Once you run pulumi up, it should create a bucket in localstack. Validate it using command below:

# adding --endpoint-url makes it point to LocalStack
aws --endpoint-url=http://localhost:4566 s3 ls
# o/p: 
# 2024-11-07 15:24:22 my-bucket-4dbacbf
info

Note that when Pulumi creates resources in AWS, it appends a random string to the end of the resource name. For example, in the case of the S3 bucket in the example above, pulumi appends '4dbacbf' to ensure a unique name.

Setup GitHub Repo​

Once you have validated dev stack, you can create a github repo. Run below commands to push changes to github:

echo "# pulumi-examples" >> README.md
git init
echo ".pulumi/" >> .gitignore
git add .
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/vchoudhari45/pulumi-examples.git
git push -u origin main

Setup Prod Stack in AWS​

For prod stack we will be using aws s3 for storing state and aws kms for secret management. Also we won't allow dev team to manually run prod stack using AWS credentials, instead we will be using github Actions.

Let's start by creating Prod stack in local

# init prod stack
pulumi stack init prod

Create following AWS resources manually using AWS console or CLI:

  • Create AWS multi-region S3 bucket for storing pulumi state.
  • Create AWS multi-region KMS key for secret management.
  • Create AWS IAM service account with least privileges and generate access keys.

Once above resource are created export them to github repository as secret. Below are sample madeup values:

AWS_ACCESS_KEY_ID: OJKRQIBAKAELSIRIBEIN
AWS_KMS_KEY_ID: 52525111-6352-7477-1331-167893214411
AWS_REGION: us-east-1
AWS_S3_BUCKET: s3://pulumi-state-bucket/
AWS_SECRET_ACCESS_KEY: Kkl1rqornqnrqqrpq/59nwkriqqQ52rwknsfnwnf
info

You can create either environment secrets or repository secrets for a specific repository. When you create environment secrets, they are scoped to that particular environment. You can also create secrets at the organization level if you have a paid github account.

After exporting the secrets, we'll set up the GitHub Actions in a branch other than main.

git branch adding-s3
git checkout adding-s3

Let's begin by adding .github/workflows/pulumi_prod_preview_deploy.yaml. This workflow will trigger whenever a pull request is opened against the main branch, allowing us to preview the deployment. The Pulumi preview information will be valuable for peers to review and provide feedback on the changes.

name: pulumi_prod_preview_deploy

# runs whenever a pull request is created against the main branch
on:
  pull_request:
    branches:
      - main

# permission needed for github token to comment on pull request 
permissions:
  issues: write
  pull-requests: write

jobs:
  pulumi_preview:
    name: Pulumi Preview Deployment
    runs-on: ubuntu-latest
    # all the github secrets are referenced using secrets keyword and exported to github runner env
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_REGION: ${{ secrets.AWS_REGION }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install Dependencies
        run: pip install -r requirements.txt

      # configuring pulumi backend is used to store the state
      - name: Configure Pulumi Backend
        run: |
          pulumi login ${{ secrets.AWS_S3_BUCKET }}

      # select the prod stack if it exists else create a prod stack
      - name: Select Or Init Prod Stack
        run: |
          pulumi stack select prod || pulumi stack init --stack prod --secrets-provider="awskms://${{ secrets.AWS_KMS_KEY_ID }}?region=${{ secrets.AWS_REGION }}"

      # this step is necessary as secretes-provider is not versioned controlled for prod stack
      # running this step explicitly assigns secretes-provider incase stack already existed in previous step
      - name: Change Secrets Provider
        run: |
          pulumi stack change-secrets-provider "awskms://${{ secrets.AWS_KMS_KEY_ID }}?region=${{ secrets.AWS_REGION }}" --stack prod

      - name: Running Pulumi Preview
        uses: pulumi/actions@v4
        with:
          command: preview
          stack-name: prod
          cloud-url: ${{ secrets.AWS_S3_BUCKET }}
          secrets-provider: "awskms://${{ secrets.AWS_KMS_KEY_ID }}?region=${{ secrets.AWS_REGION }}"
          comment-on-pr: true
          diff: true

Next, we will add a github action to deploy infrastructure to AWS when changes are merged into the main branch. We'll name this action .github/workflows/pulumi_prod_update_deploy.yaml.

name: pulumi_prod_update_deploy

# runs whenever a changes are marged to main branch
on:
  push:
    branches:
      - main

# permission needed for github token to comment on pull request 
permissions:
  issues: write
  pull-requests: write

jobs:
  pulumi_update:
    name: Pulumi Update Deployment
    runs-on: ubuntu-latest
    # all the github secrets are referenced using secrets keyword and exported to github runner env
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_REGION: ${{ secrets.AWS_REGION }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install Dependencies
        run: pip install -r requirements.txt

      # configuring pulumi backend is used to store the state
      - name: Configure Pulumi Backend
        run: |
          pulumi login ${{ secrets.AWS_S3_BUCKET }}

      # select the prod stack if it exists else create a prod stack
      - name: Select Or Init Prod Stack
        run: |
          pulumi stack select prod || pulumi stack init --stack prod --secrets-provider="awskms://${{ secrets.AWS_KMS_KEY_ID }}?region=${{ secrets.AWS_REGION }}"

      # this step is necessary as secretes-provider is not versioned controlled for prod stack
      # running this step explicitly assigns secretes-provider incase stack already existed in previous step
      - name: Change Secrets Provider
        run: |
          pulumi stack change-secrets-provider "awskms://${{ secrets.AWS_KMS_KEY_ID }}?region=${{ secrets.AWS_REGION }}" --stack prod

      - name: Running Pulumi Update
        uses: pulumi/actions@v4
        with:
          command: up
          stack-name: prod
          cloud-url: ${{ secrets.AWS_S3_BUCKET }}
          secrets-provider: "awskms://${{ secrets.AWS_KMS_KEY_ID }}?region=${{ secrets.AWS_REGION }}"
          comment-on-pr: true
          diff: true

Create a pull request with above changes and pulumi_prod_preview_deploy github action should comment on the PR with pulumi preview output, below is snapshot.

pulumi-1.svg

And once pull request is merged pulumi_prod_update_deploy github action should create infrastructure in AWS.