https://pulumi.com logo
Join Slack
Powered by
I've been running into a problem when creating azu...
# azure
b
I've been running into a problem when creating azure container apps with custom domains. Provisioning seems to work well. The custom domain needs to be defined on the container app, and a managed certificate can be added if there's a CNAME pointing to the container app (for domain verification). When it comes time to destroy, I run into a problem where the managed certificate can't be deleted because it's needed by the custom domain, but the container app can't be deleted because the certificate still exists. I can resolve this by removing the custom domain from the container app using the `az`cli, after which the delete works. Any ideas on how I should resolve this such that pulumi destroy can execute without intervention? I looked into hooks but couldn't figure out how to call an AzureCliScript from within the hook.
s
Are you using azurerm or azure-native? Can you share a code sample? It seems like this should be possible without resorting to az cli but there may need to be a specific order or some dependsOn options set to ensure the right order of ops is run on destroy
b
using azure-native. I've tried a few different dependsOn options but probably not every permutation.... I can pull out a code sample.
s
That would be great, thank you!
b
This references a vanilla ManagedEnvironment and a ResourceGroup defined outside of the class, but otherwise is pretty well contained.
component with container app.cs
s
@brainy-guitar-22719 You can also use the Command Provider to execute any arbitrary command (
az
in this case) if you can't find a more elegant solution using other Pulumi resources. https://www.pulumi.com/registry/packages/command/api-docs/local/command/
b
It's worth noting that when i try to destroy the stack it tries to delete the managed cert before it tries to delete anything else.
Will the Command Provider work in Pulumi cloud. I've been trying to transition the management of my project to Pulumi deployments (OIDC to Azure) and when I try to do AD/Entra stuff in my stack I get the following error: "unable to obtain access token: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account."
s
Meaning you've got an ESC env performing the login for your Pulumi code's context?
Is this being executed by Pulumi Deployments or by a traditional CI/CD runner?
b
I'm getting the 
az login
error when my pulumi program is executed by Pulumi Deployments . Previously I'd been doing everything from local with a valid 
az login
session already established. When running 
pulumi up
that way I did not get any errors. I don't have my head fully wrapped around what an ESC env is, but yes, I converted my config to ESC. Not sure what you mean by 'ESC env performing the login for your Pulumi code's context'. The connection to Azure is managed through OIDC to a azure service principal with roles appropriate for deployment. I'm not confident that the roles are setup correctly for the Entra group role assignments that I tried to do, but the error message indicates to me that the failure is happening at authentication (no token), not with the authorization.
s
Let me see get up to speed on exactly what environmental context is possible within a Pulumi Deployments runner and get back to you.
s
@brainy-guitar-22719 An ESC environment is the "unit of work", IOW, it's the thing you compose in a YAML doc and open when you run Pulumi. You can output ESC values directly as Pulumi config e.g. 
azure-native:*
or as env vars. If you're only outputting Pulumi config, then you also need to emit the values as env vars that can be consumed by the 
az
CLI.
s
Right I am guessing that's how you have OIDC working between Pulumi Cloud and Azure. https://www.pulumi.com/docs/esc/integrations/dynamic-login-credentials/azure-login/ So if you just have something like the above then Pulumi programs know how to auth, but local commands will not. You can expose the same runtime args the az cli does upon login by adding the environmentVariables block, like so:
Copy code
values:
  azure:
    login:
      fn::open::azure-login:
        clientId: your-client-id
        tenantId: your-tenant-id
        subscriptionId: your-sub-id
        oidc: true
  environmentVariables:
    ARM_USE_OIDC: 'true'
    ARM_CLIENT_ID: ${azure.login.clientId}
    ARM_TENANT_ID: ${azure.login.tenantId}
    ARM_OIDC_REQUEST_TOKEN: ${azure.login.oidc.token}
    ARM_OIDC_TOKEN: ${azure.login.oidc.token}
    ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
    ARM_OIDC_REQUEST_URL: <https://api.pulumi.com/oidc>
  pulumiConfig:
    azure-native:useOidc: 'true'
    azure-native:clientId: ${azure.login.clientId}
    azure-native:tenantId: ${azure.login.tenantId}
    azure-native:subscriptionId: ${azure.login.subscriptionId}
    azure-native:oidcRequestToken: ${azure.login.oidc.token}
    azure-native:oidcToken: ${azure.login.oidc.token}
    azure-native:oidcRequestUrl: <https://api.pulumi.com/oidc>
b
I'm using OIDC for connecting to azure for deployments (I followed the instructions here: https://www.pulumi.com/docs/deployments/deployments/oidc/azure/ ). I am not using Azure at the ESC secrets Provider. Presumably that means that Pulumi is serving as the ESC secrets provider.
s
It's an either/or I'm fairly certain, based on this doc: https://www.pulumi.com/docs/deployments/deployments/cloud-credentials/ I think your case here will inform another reason Pulumi will recommend ESC over Deployments auth which is 'you find a reason that you need to execute cloud provider CLI commands directly'.
Container apps are an evolution in Azure from 'web apps that can use containers' and there are some weird things that stem from them offering two managed solutions that do the same thing, but not exactly the same way. It doesn't look like they make it easy to handle custom domains in managed arm or bicep templates that has also considered the need to perform regular teardowns when a managed cert/domain is also in the picture. So I concur with Josh, that the answer to this for now is finding a way to run az cli commands directly where you have a more surgical precision over order-of-operations. Adding Pulumi Deployments into the mix offers a lot of automation advantages you can't get with ordinary CI/CD, with some trade-offs around environment variable control you cannot get with Deployments directly. I think using ESC for auth + exposing runtime environment variables will bring this all together for you though.
b
good stuff. I'll look into ESC vs Deployments - I didn't understand them as alternatives to each other, so clearly I'm missing something. As for Azure and this being tricky - that's fine. I've found with all cloud providers and IaC options that I end up doing some non-ideal stuff around the edges to make it work.
Switched from Deployments to ESC Oidc auth. All the stack defined through azure.native works great, but the one part defined with AzureAd fails. Here's the config
Copy code
values:
  azure:
    login:
      fn::open::azure-login:
        clientId: <redacted>
        tenantId: <redacted>
        subscriptionId: <redacted>
        oidc: true
  environmentVariables:
    ARM_USE_OIDC: 'true'
    ARM_CLIENT_ID: ${azure.login.clientId}
    ARM_TENANT_ID: ${azure.login.tenantId}
    ARM_OIDC_TOKEN: ${azure.login.oidc.token}
    ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
    ARM_OIDC_REQUEST_TOKEN: ${azure.login.oidc.token}
    ARM_OIDC_REQUEST_URL: <https://api.pulumi.com/oidc>
  pulumiConfig:
    azure-native:location: eastus
    azure-native:useOidc: 'true'
    azure-native:clientId: ${azure.login.clientId}
    azure-native:tenantId: ${azure.login.tenantId}
    azure-native:subscriptionId: ${azure.login.subscriptionId}
    azure-native:oidcRequestToken: ${azure.login.oidc.token}
    azure-native:oidcToken: ${azure.login.oidc.token}
    azure-native:oidcRequestUrl: <https://api.pulumi.com/oidc>
And here's the error:
Copy code
error: Running program '/deployment/PlatformEnvironment/bin/Debug/net9.0/PlatformEnvironment.dll' failed with an unhandled exception:
    Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="invocation of azuread:index/getGroup:getGroup returned an error: 1 error occurred:
    	* building client: unable to obtain access token: running Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
My guess is that azuread is using a different auth mechanism. Advice on how I can give it what it needs?
s
Since it's technically a different provider it'll need it's own config I think
Confirmed - https://www.pulumi.com/registry/packages/azuread/installation-configuration/#set-configuration-using-pulumi-config So you'll need to duplicate the config again, essentially.
Copy code
values:
  azure:
    login:
      fn::open::azure-login:
        clientId: <redacted>
        tenantId: <redacted>
        subscriptionId: <redacted>
        oidc: true
    region: eastus
  environmentVariables:
    ARM_USE_OIDC: 'true'
    ARM_CLIENT_ID: ${azure.login.clientId}
    ARM_TENANT_ID: ${azure.login.tenantId}
    ARM_OIDC_TOKEN: ${azure.login.oidc.token}
    ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
    ARM_OIDC_REQUEST_TOKEN: ${azure.login.oidc.token}
    ARM_OIDC_REQUEST_URL: <https://api.pulumi.com/oidc>
  pulumiConfig:
    azure-native:location: ${azure.region}
    azure-native:useOidc: 'true'
    azure-native:clientId: ${azure.login.clientId}
    azure-native:tenantId: ${azure.login.tenantId}
    azure-native:subscriptionId: ${azure.login.subscriptionId}
    azure-native:oidcRequestToken: ${azure.login.oidc.token}
    azure-native:oidcToken: ${azure.login.oidc.token}
    azure-native:oidcRequestUrl: <https://api.pulumi.com/oidc>
    azurerm:location: ${azure.region}
    azurerm:clientId: ${azure.login.clientId}
    azurerm:tenantId: ${azure.login.tenantId}
    azurerm:subscriptionId: ${azure.login.subscriptionId}
    azurerm:oidcToken: ${azure.login.oidc.token}
I note there slightly different guidance in the docs between the providers: https://www.pulumi.com/registry/packages/azuread/installation-configuration/#set-configuration-using-pulumi-config https://www.pulumi.com/registry/packages/azure-native/installation-configuration/#set-configuration-using-pulumi-config So I removed some of the settings around oidc for the tf module. Both providers look to support reading from environmentVariables, so you might be able to reduce this config to just environment variables provided you can get something that works for native, azurerm, and az cli. Worth playing around with a bit to see if you can simplify and still have everything auth the way you need. This is worth a try:
Copy code
values:
  azure:
    login:
      fn::open::azure-login:
        clientId: <redacted>
        tenantId: <redacted>
        subscriptionId: <redacted>
        oidc: true
    region: eastus
  environmentVariables:
    ARM_USE_OIDC: 'true'
    ARM_CLIENT_ID: ${azure.login.clientId}
    ARM_TENANT_ID: ${azure.login.tenantId}
    ARM_OIDC_TOKEN: ${azure.login.oidc.token}
    ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
  pulumiConfig:
    azure-native:location: ${azure.region}
    azure-native:clientId: ${azure.login.clientId}
    azure-native:tenantId: ${azure.login.tenantId}
    azure-native:subscriptionId: ${azure.login.subscriptionId}
    azure-native:oidcToken: ${azure.login.oidc.token}
    azurerm:location: ${azure.region}
    azurerm:clientId: ${azure.login.clientId}
    azurerm:tenantId: ${azure.login.tenantId}
    azurerm:oidcToken: ${azure.login.oidc.token}
b
This worked with a small change: The config prefix was 
azuread
not 
azurerm
. In case someone searches this channel: Make sure your dependencies are current. Somehow I was using an ancient version of the azuread provider.
🙌 1
13 Views
Open in Slack
PreviousNext
Powered by