I believe I am missing some underlying concept with esc and IaC. I created my project named

marketforge

, and created my stack using a config file named

pulumi.scouten.yaml

. I converted my stack using

pulumi config env init

. As I expected the environment was created with all the configuration, and the config file was updated to just have:

environment:  
  - marketforge/scouten

When I run pulumi up locally it works without any errors and deploys. When I try to do it through github actions it fails saying that the configurations are missing. Sure enough, when I go to pulumi the configuration is missing and it shows this error. There is one configuration that is in there and it is coming from my

pulumi.yaml

Type                 Name                 Status         Info
     pulumi:pulumi:Stack  marketforge-scouten  **failed**     1 error
 
Diagnostics:
  pulumi:pulumi:Stack (marketforge-scouten):
    error: Missing required configuration variable 'marketforge:location'
    	please set a value using the command `pulumi config set marketforge:location <value>`

I have tried many different iterations of this github workflow, this was my last attempt:

name: Deploy 
on:
  push:
    branches:
      - master
jobs:
  pulumi-up:
    environment: pulumi_environments
    runs-on: ubuntu-latest
    strategy:
      matrix:
        stack: ["esc"]
    permissions:
      contents: read
      id-token: write
    steps:
      - name: "Checkout code"
        uses: actions/checkout@v4

      - name: "Set up Pulumi"
        uses: pulumi/actions@v6
        with: 
          pulumi-version: 3.158.0
          stack-name: ${{ vars[matrix.stack] }}
          work-dir: ./azure-deploy
          github-token: ${{ secrets.GITHUB_TOKEN }}

      - name: Install Pulumi ESC CLI
        uses: pulumi/esc-action@v1
        with:
          environment: ${{ vars[matrix.stack] }} 
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

      - name: Run Pulumi using ESC
        working-directory: ./azure-deploy
        run: |
          FULL_STACK="${{ vars[matrix.stack] }}"
          TRIMMED_STACK="${FULL_STACK#*/}"
          pulumi login
          
          esc run "ajwtech/${FULL_STACK}" -- env PULUMI_STACK="$TRIMMED_STACK" pulumi install
          esc run "ajwtech/${FULL_STACK}" -- env PULUMI_STACK="$TRIMMED_STACK" pulumi up --yes --stack "$TRIMMED_STACK"

I can see in the run that it has the values and I

Run FULL_STACK="marketforge/scouten"
  FULL_STACK="marketforge/scouten"
  TRIMMED_STACK="${FULL_STACK#*/}"
  pulumi login
  
  esc run "ajwtech/${FULL_STACK}" -- env PULUMI_STACK="$TRIMMED_STACK" pulumi install
  esc run "ajwtech/${FULL_STACK}" -- env PULUMI_STACK="$TRIMMED_STACK" pulumi up --yes --stack "$TRIMMED_STACK"
  shell: /usr/bin/bash -e {0}
Logged in to <http://pulumi.com|pulumi.com> as **************
Installing dependencies...


added 383 packages, and audited 384 packages in 50s

47 packages are looking for funding
  run `npm fund` for details

1 high severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
Finished installing dependencies

[resource plugin random-4.17.0] installing
[resource plugin cloudflare-5.49.0] installing
[resource plugin azure-native-2.84.0] installing
[resource plugin azure-6.18.0] installing
Previewing update (scouten)


@ Previewing update................................
    pulumi:pulumi:Stack marketforge-scouten running error: Missing required configuration variable 'marketforge:location'
@ Previewing update....
    pulumi:pulumi:Stack marketforge-scouten  1 error
Diagnostics:
  pulumi:pulumi:Stack (marketforge-scouten):
    error: Missing required configuration variable 'marketforge:location'
    	please set a value using the command `pulumi config set marketforge:location <value>`

Resources:
    1 unchanged

Error: exit status 255

I also noticed that in pulumi the stack and the environment do not appear to be added to each other. Next I'll try adding

pulumi config env add

into the mix but I'm not sure that gets to the root of this issue.

When I try to do it through github actions it fails saying that the configurations are missing

Is the pulumi.scouten.yaml file checked in for the github action to see it? Config files are just local files, they need to be pushed to the repo.

It is actually, esc, that's exactly what I am trying to get to. I was trying to avoid pushing those config files. I tried something like this in the workflow too:

run: |
          FULL_STACK="${{ vars[matrix.stack] }}"
          TRIMMED_STACK="${FULL_STACK#*/}"
          echo "TRIMMED_STACK=${TRIMMED_STACK}" >> $GITHUB_ENV
          echo "FULL_STACK=${FULL_STACK}" >> $GITHUB_ENV

      - name: Create Pulumi configuration file
        run: |
          mkdir -p ./azure-deploy/config
          cat <<EOF > ./azure-deploy/config/Pulumi.${TRIMMED_STACK}.yaml
          environment:
            - ${TRIMMED_STACK}
          EOF

to just create it on the fly but that didn't work. I just didn't want to push them to a public repository. With ESC I don't see a reason to keep the stack configs around. That feature sounds like a good call. In the meantime is there a viable way to work around this or does it have to be in the repo?

Creating the config file on the fly should work. But I don't think you can use TRIMEMD_STACK in both the file name and env name. The filename will be Pulumi.scouten.yaml but the env name is marketforge/scouten

@echoing-dinner-19531 You are correct I actually had several places I misused that variable. I still have not fixed it though. check this out.

PS D:\Source\MarketForge\azure-deploy> esc env ls        
marketing/dev
marketforge/scouten
marketing/test
PS D:\Source\MarketForge\azure-deploy> pulumi config --stack ajwtech/marketforge/scouten
error: [0] 
Diags: invalid environment name "marketforge/scouten"

PS D:\Source\MarketForge\azure-deploy>

I'm not sure where this goes off the rails. I Can't figure out a way to create the environment so that they will provide vars to Pulumi.

I think esc lists environments without the organisation name, but in pulumi you need to add the organisation name to environment name when referencing it

That is kind of the problem here, I can't figure out where pulumi is getting the environment name "marketforge/scouten"

Isn't that just what's written to the environment block in the stack config?

This is the stack config:

environment:
  - ajwtech/marketforge/scouten

Looking at what you're trying to do,

I was trying to avoid pushing those config files

, you should be able to use

esc run

to grab the values from the environment no problem, should be no need to create config files on the fly. Just to be make sure we're on the same page, ESC projects and stack projects are not in any way related to each other, they are just parallel concepts.

Diags: invalid environment name "marketforge/scouten"

this line makes me think the default organization is different from

ajwtech

? This is confusing, it looks like it's there, but then can't be found. What shows up if you run

pulumi org get-default

?

Maybe you are on to something.

PS D:\Source\MarketForge\azure-deploy> pulumi org get-default                 
No Default Org Specified
PS D:\Source\MarketForge\azure-deploy>

Another thing to try is you might have different credentials or settings between

esc

and

pulumi

CLI? What happens if you run

pulumi env ls

instead of

esc env ls

, do you still get the same list?

same lists PS D:\Source\MarketForge\azure-deploy> pulumi env ls marketing/dev marketforge/scouten tmp/scoutentmp marketing/test PS D:\Source\MarketForge\azure-deploy> esc env ls marketing/dev marketforge/scouten tmp/scoutentmp marketing/test PS D:\Source\MarketForge\azure-deploy>

Ah, I think I know what the issue is Replace

environment:
  - ajwtech/marketforge/scouten

With

environment:
  - marketforge/scouten

that's how it was before. I will put that back now.

I see, always happens to me 😅 Is it all working now? Or is GHA still not working as expected? The specific error above is an issue with parsing, taking in the org name as project and thinking

project/env

is the intended environment name (which is rightfully invalid). I'll make an issue to improve the confusing error message to something more like "the environment name has too many slashes, specify in the format project/environment"

yeah, that was an issue with variables.

You are welcome, feel free to reach out anytime our docs are unclear or errors are confusing. I filed and picked up an issue to improve that error message, you can follow it here, it's public.