https://pulumi.com logo
Join Slack
Powered by
Hello all. Running into a very annoying problem. ...
# automation-api
c
Hello all. Running into a very annoying problem. Context: I am hosting a centralized pulumi server on AWS EC2 with S3 backend, managing stacks across multiple AWS accounts. My centralized pulumi server is assuming roles in other aws accounts using diff provider for each. This setup has been working fine for 6+ months. Suddenly running into a bug where pulumi preview or pulumi up just hits an unhandled exception, complaining about AccessDenied permission for a specific resource.
Copy code
Exception: invoke of aws:msk/getBootstrapBrokers:getBootstrapBrokers failed: invocation of aws:msk/getBootstrapBrokers:getBootstrapBrokers returned an error: invoking aws:msk/getBootstrapBrokers:getBootstrapBrokers: 1 error occurred:
 reading MSK Cluster (arn:aws:kafka:us-east-1:11111111111:cluster/qa-msk-cluster/7971b812-1c13-40a0-95a7-77dc6c5deaf1-22) bootstrap brokers: operation error Kafka: GetBootstrapBrokers, https response error StatusCode: 403, RequestID: 2c958b31-8fc2-4c65-8e9a-933a0caf7fee, api error AccessDeniedException: User: arn:aws:sts::222222222222:assumed-role/pulumi-controller-role/i-123123123123 is not authorized to perform: kafka:GetBootstrapBrokers on resource: arn:aws:kafka:us-east-1:1111111111:cluster/qa-msk-cluster/7971b812-1c13-40a0-95a7-77dc6c5deaf1-22
It almost looks like pulumi is not using that cross acount role when its calling the GetBootstrapBrokers API call, its using the local IAM profile permissions hence it doesn’t have the access. It should be using the cross account role in that other account. Funny thing is, I created a brand new project/stack with the same pulumi code with only the MSK resource for test, and everything works as expected on the same server. Has anybody else seen a similar issue? What do you guys suggest?
after struggling a couple days, found the root cause. My mistake. I had a piece of code that were doing the get bootstrap brokers api call and looks like we missed specifying the InvokeOptions to read from the selected provider, so it was using default provider.
Copy code
bootstrap_brokers = aws.msk.get_bootstrap_brokers_output(cluster_arn=arn)
should have been:
Copy code
bootstrap_brokers = aws.msk.get_bootstrap_brokers_output(cluster_arn=arn, opts=pulumi.InvokeOptions(provider=selected_provider))
is there a way to specify/change the default provider at the beginning of the program once, so that I don’t have to specify ResourceOptions or InvokeOptions for hundreds of resources?
l
Hi Cihan, If you are just using a single provider, you can make it a default provider by not specifying it in code and using AWS configuration in 
Pulumi.yaml
(or in the case of automation, by specifying that configuration before you run the stack in code). If you want to continue using explicit providers, there is nothing that quite does what you want yet, though it is being looked at/worked on very shortly I believe.
s
We wrote a simple get provider library that is the only way we create AWS and K8s providers, based on some params. We bypass the whole Pulumi config system as we want some params to be global across all projects and stacks, with others depending on env/stack
12 Views
Open in Slack
PreviousNext
Powered by