Hello All, Hoping someone may have some insight o...
# general
Hello All, Hoping someone may have some insight on strange behavior surrounding authentication. It consistently fails, even when environment variables are populated with valid values. • Using Azure Native to create CDNs in a CI pipeline • Service principal authentication works fine via
az login
Per the docs, we should be able to use the following environment variables for authentication:
Copy code
• After setting them with the same values as
az login
, this error arises:
Copy code
azure.identity._exceptions.CredentialUnavailableError: Please run 'az login' to set up an account
• Setting
, reveals this to be the culprit, but I'm unsure how to ensure that the environment variables are used to authenticate prior to execution of the command.
Copy code
subprocess.CalledProcessError: Command '['/bin/sh', '-c', 'az account get-access-token --output json --resource <https://management.azure.com>']' returned non-zero exit status 1.
• Ironically, another error specifying that service principal credentials should be supplied via configurations/environment variables is thrown when
az login
is used. • Just tried setting values in the the config file and the same behavior is exhibited.... What in the world am I overlooking!??!
Seems strange! What CI system are you using? Are you also able to share how you're setting those env vars? The linked doc also mentions that using the
CLI in a CI scenario is not recommended; have you tried using one of the alternatives?
In the image above, a container with all the dependencies installed is being used. I removed the CI from the equation, which is Gitlab, just for troubleshooting shenanigans....because it's weird at times, too. The issue has persisted unfortunately. AFAIK, I think
itself is used by the Azure Native library. I could be mistaken about that. The following bash snippet is used to set the environment variables by reading them from a JSON document procured from HCP Vault:
Copy code
vault kv get -mount="$VAULT_SECRET_MOUNT" \
  -format=json "$VAULT_SECRET_NAME" | jq ".data.data" > $SECRETS
for KEY in $(jq -r "keys[]" $SECRETS); do
    echo "Setting Environment Variable: $KEY";
    VALUE=$(jq -r ".${KEY}" $SECRETS);
    eval "export ${KEY}=${VALUE@Q}";
I can confirm this approach works and that the values are valid as has been consistently effective for other providers like AWS, Fastly, and DigitalOcean.
Hmmm.....I haven't tried configuring the provider manually in the script. I suppose that'd ensure credentials are being passed accordingly. I'll try that real quick.
My only guess on this one would be that
is creating those variables in a child process, which are not getting propagated to the parent and thus not propagated to subsequent CI steps. Perhaps you could try removing
? Maybe something as simple as
export ${KEY}=${VALUE@Q}
? Or, if
must be present, you could try
export ${KEY}=$(eval ${VALUE@Q})
These are just best guesses, so apologies for any errors/misdirections; I'm much more of a Powershell person 😅
No problem! I appreciate the extra eyes! We can use this syntax to ensure the variables are exported in the current process:
. set_vars.sh
I actually have guard rails in the Pulumi program to ensure that the environment variables are present, otherwise it throws an error. They're definitely there 😄
🤦 I think I found the error! It's my fault! I froze the Azure Native library version in a helper library:
I'll bet that's the issue. Going to bump that and see if it does the thing 🤞
Oh goodness! Hopefully that solves it
Dyslexia be damned -- I misread the operator, which is actually
is currently installed. Upgrading to
now. Maybe that'll help.
To remove the shell as a possible culprit, perhaps you could try setting the values within the pulumi config during the CI run as demonstrated in the doc page: https://www.pulumi.com/registry/packages/azure-native/installation-configuration/#set-configuration-using-pulumi-config
If you do that in the CI environment, so long as you don't commit any changes made to the repo during the run, you don't have to worry about values being committed to source control
And by
do that
I mean using the
pulumi config set azure-native:<key>
Yep, the outcome was the same. Tried that just a second ago, along with expressly instantiating a provider. I actually missed this log event while using the environment variables until just now. It seems to claim that it's using "Client Secret for Authentication" as configured. Three things are interesting: 1. It confirms use of "Service Principal / Client Secret for Authentication" 2. It's getting an OAuth config, but it isn't expressly configured to use OAuth 3. We know that the account has permissions to retrieve the token because the
az account access
command runs fine after using
az login
Starting to wonder if there's a misconfiguration with the service principal/app.
Fascinating... could a service principal client secret be lurking somewhere in your Vault?
I notice that your environment variable creation loop creates vars indiscriminately, so if there are extra key-value pairs in that vault, they'll be created too
Good observation! You're def right (shiv code haha). Luckily, HCP Vault's interface makes it easy to inspect by sorting the keys. No duplicates in sight.
This is beginning to feel like it could be a legit bug. I may open up a Github issue.
I found the problem! I'm so sorry for taking up so much of your time and brainpower. This was totally my fault. tl;dr • Azure is the worst • Enforces maximums/limits on the number of CDNs that can be created for a given subscription • Our use case requires quite a few distributions • I developed a system that programmatically creates distributions across multiple profiles • Root Cause: I had totally forgotten that I have a manually configured
object from when I was using CLI authentication The system had been working for years, so it slipped my mind. Thanks so much for your help!
I'm glad you found the issue! 😄