Hi folks... we seem to be encountering very strang...
# esc
m
Hi folks... we seem to be encountering very strange behavior with Pulumi ESC. We've checked encodings, a bunch of other stuff too. Somehow there is config drift between our GitHub CICD and local CLI deployment. More detail in-thread.
This is our RAW config for a specific environment:
Copy code
imports:
  - pmb-main-base
values:
  pulumiConfig:
    mainAppArgs:
      FEATURE_OVERRIDE_ONLY_EMAIL_SIGNUPS: true
      FEATURE_OVERRIDE_ROOT_REDIRECT: "browse-our-brains"
      TEXT_OVERRIDE_WEBSITE_TITLE: "IWF"
      TAG_OVERRIDE_I_AM_AVAILABLE_FOR_CSV_LIST: "Networking,Mentoring"
      TAG_OVERRIDE_AREAS_OF_EXPERTISE_CSV_LIST: "Leadership Coaching"
      TEXT_OVERRIDE_MARKETPLACE_BANNER_TITLE: "International Women's Forum Canada Executive Mentor Marketplace"
      TEXT_OVERRIDE_MARKETPLACE_BANNER_SUBTEXT: "Members of the International Women's Forum, Canada who are open and available to support leadership development through mentoring."
      DEVOPS_OVERRIDE_SUPPRESS_GOLIVE_NOTIFICATIONS: true
      TEXT_OVERRIDE_TOPNAV_BECOME_BRAIN: "Become a Mentor"
      TEXT_OVERRIDE_TOPNAV_MEET_BRAINS: "Meet the Mentors"
      TEXT_OVERRIDE_MARKETPLACE_BANNER_PEOPLE_BUTTON: "Browse Mentors"
      TEXT_OVERRIDE_BROWSE_BRAINS_SEARCH_HEADER: "Recommended Mentors"
      HIDE_BROWSE_BRAIN_FILTERS_CSV_LIST: "communities,bookings,length,language,secondary"
      HIDE_BROWSE_OFFER_FILTERS_CSV_LIST: "communities,bookings,length,language,secondary"
      HIDE_ADMIN_MENUS_CSV_LIST: "volition,commission,fin_reports,currency,subscription,emails"
Now, for some reason, something in this config (we've tried the apostrophies and a few other characters) is causing this to not get stitched together properly with a different ESC-sourced JSON during the build ...on github. But a local preview picks it up correctly.
Question for the folks here: what characters (or perhaps something else) are known to cause strange issues when being transposed from the ESC UI YAML into the CLI JSON format? Or any other ideas?
This is a TS snippet that we're using for config composition in our pulumi stack, may be part of the issue - we're normally pretty ok about diagnosing these things but this specific one is a bit of a mystery.
Copy code
const sanitizedEscAppConfig = Object.entries(mainAppConfig).map(([key, value]) => {
    return { [key]: typeof value === 'string' ? value : String(value) };
}).reduce((acc, curr) => ({ ...acc, ...curr }), {}) as SanitizedEscAppConfig;

const sanitizedInfraAppConfig = Object.entries(infraAppConfig).map(([key, value]) => {
    return { [key]:  value }; // Direct, non-typed mapping
}).reduce((acc, curr) => ({ ...acc, ...curr }), {}) as SanitizedEscInfraConfig;

export const getSanitizedEscAppConfig = () => sanitizedEscAppConfig;

export const getFargateEnvironment = (runtimeAppConfig: PulumiManagedAppConfig) => {
    const sanitizedEscAppConfig = getSanitizedEscAppConfig();
    return Object.keys({
        ...sanitizedEscAppConfig,
        ...runtimeAppConfig
    }).map(configKey => {
        return { name: configKey, value: ({ ...sanitizedEscAppConfig, ...runtimeAppConfig } as AppRuntimeConfig)[configKey] as string }
    }).reduce((accumulator, current) => {
        accumulator.push(current);
        return accumulator;
    }, [] as FargateEnvironment);
}
r
is causing this to not get stitched together properly with a different ESC-sourced JSON during the build
Can you say more about how this manifests? What doesn't get stitched properly? What si your result?
m
Hey @red-match-15116! Appreciate the follow up. Trouble is, we're not sure. On CI (standard github XL runners), the ESC config gets interpreted/read differently from local for some reason (locals: WSL + MacOS Linux). Our last resort would be to spin up a cloned AMI and SSH in to see what's going on. The behavior seems really odd, it may have to do with OS encoding is our best guess? Wondering if there have been any similar cases users have brought to you. Generally speaking, Pulumi has been flawless in terms of ENV parity for local deploy vs CICD deploy.
We're using ACT for CI parity locally and are seeing this
r
Yeah we haven't heard any other similar reports. Definitely sounds super odd... How do you know it's different? Are you getting errors? Or the environment is not resolving?
m
When we go to a URL like this: https://app.pulumi.com/REDACTED/Project/Stack/updates/109 The config values being ingested for those (they're two merged ESC environments - one inheriting from a common base, and then tenant specific) are different even on the CI output. Does this help?
r
And the difference is that if you run it locally you get the value from one environment and if you run it in github it's from a different environment? Sorry, just trying to understand what "different" really means.
m
Yup, fair callout. The args listed on the Pulumi CI screen shown above for "mainAppArgs" appear to differ when ran on CI vs Local.
r
I think best thing to do would be to open an issue https://github.com/pulumi/esc/issues/new/choose It's super helpful if you can be extremely explicit about what the exact difference is between local and CI. For example, in CI I see the values from env1, locally I see the values from env2. I'm still not being able to understand exactly what the difference is that you're describing, and maybe I'm just missing it, but any details you can provide would really help in diagnosing.
I'm also wondering if there's some sort of permissions issue at play here - if different users are involved locally vs on CI