https://pulumi.com logo
Powered by
Title
g

gorgeous-country-43026

11/21/2022, 2:13 PM
Hi, I would like someone to confirm if such a huge bug can have slipped past Pulumi team before I create an issue to Github. Somehow I find it almost impossible this could have occured but everything I've done is pointing towards that. In Google Cloud when defining IAM bindings those do not get applied even if Pulumi says they do. I've been debugging this for several days now and all IAM bindings I try to apply just don't appear into Google Cloud Console and likewise all operations requiring those fail within our system. I can define those via Google Cloud Console or via 
gcloud
as expected and they do appear and everything works. But with Pulumi this simply doesn't happen but in Pulumi state those are set thus leading to Pulumi state not being in sync with the actual environment. The code I'm running has worked without issues before and as such this must be a regression. This happens at least with the latest version of 
@pulumi/gcp
(version 6.43.0). I tried downgrading 
@pulumi/gcp
to around when our lower environment was created (and when IAM bindings were successfully created) but that didn't help. So it might be something related to newer Pulumi CLI versions or something. I do not know at this point. This however is fatal for us. Not being able to set IAM bindings reliably obviously is more or less catastrophic from IaC point of view. And because of this I'm thinking this kind of bug just cannot have slipped past and I'm reaching out if anyone else has faced this? This worked for us ~2 months ago but not anymore.
b

billowy-army-68599

11/21/2022, 2:36 PM
please file an issue, sounds like a P1. we haven’t seen any other reports of this, however
g

gorgeous-country-43026

11/21/2022, 3:17 PM
Sure thing, will do asap
Trying to create a minimal repro case. My first attempt didn't surface this so may take a bit longer
f

flaky-arm-38472

11/21/2022, 3:53 PM
did you try running 
pulumi refresh
before applying @gorgeous-country-43026. I have some vague thoughts of this happening to me and it was resolved with a refresh. Not completely sure if it was the same tho.
g

gorgeous-country-43026

11/21/2022, 4:01 PM
I'll try that but I think I did
Anything to get this issue GONE would be nice 😅
nope, no change
I'm also destroying the pulumi resources by commenting out those from code, doing pulumi up to ensure those are not in state, then uncommenting and executing stuff again
Just tried that once again but to no avail
Pulumi says resources were created
But those are not there. The one which is there has been created way earlier
b

billowy-army-68599

11/21/2022, 4:09 PM
you’re in the right project/region right? if you export the ids do they return? if you refresh does it delete the resources?
pulumi is just calling the gcp api, there’s nothing special going on
g

gorgeous-country-43026

11/21/2022, 4:09 PM
Yeah, that's what I'm expecting it to do
And yes, right project/region etc
Refresh does not delete resources
Pulumi thinks they are there
I was also considering that I might be in a wrong project before I wrote my original message and I've triple checked that
Went through all the projects we have in this
Tried creating even extra resources not related to IAM just to see those appear, they do
b

billowy-army-68599

11/21/2022, 4:11 PM
Pulumi thinks they are there
if the refresh doesn’t remove them, they’re somewhere
because refresh will examine the upstream API to determine if they exist
if you do a debug run, you’ll see the API call
g

gorgeous-country-43026

11/21/2022, 4:12 PM
Nice, I'll try that next
b

billowy-army-68599

11/21/2022, 4:12 PM
https://www.pulumi.com/docs/support/troubleshooting/#verbose-logging
g

gorgeous-country-43026

11/21/2022, 5:07 PM
Been going through the vast log created by 
-v=9
but don't see anything that special. But also I do not see any GCP API calls in it, guess I need some other flags to surface those into logs
But can't continue on this today anymore. My kids are going nuts since I'm not giving them any attention
f

flaky-arm-38472

11/22/2022, 2:35 AM
If the resource appears on the stack then it could be the resource on GC is in a hanging state internally or waiting for something. What if you try deleting the resource from the stack first with 
pulumi stack delete 'resource urns'
and then run 
pulumi up
again
g

gorgeous-country-43026

11/22/2022, 8:36 AM
Let me try it. I highly doubt it will help though
Nope
I'll keep on debugging
Very mildly obfuscated code which I just wrote to index.ts: 
export const testBinding = new gcp.pubsub.TopicIAMBinding("testing", {
  topic: pubsubTopics.myTopic.name,
  role: "roles/pubsub.publisher",
  members: [
    backend.theServiceAccount.email.apply(email => `serviceAccount:${email}`)
  ]
});
and no pubsub.publisher role appears for the given service account
For the given topic
Previously this worked without issues
No, wait! It did apply
Progress!
OK. I can confirm this was indeed my own fault. I'm still not 100% certain what caused it but I think it was a whole bunch of mixups. So no bug in Pulumi or GCP just in my brains!
But in general I think this had to do with pulumi resources being created in namespaces BUT those do not get referenced by other resources well enough so that the chain leads to final export at index level and because of this stuff didn't appear or something like that. Sounds fishy and unlikely but that's the best explanation I can give right now
To me it does sound a bit weird reason since to my understanding final exports are pulumi outputs that can be referenced from other stacks if need be. But those should not be related to if resources actually do get applied.
#generalJoin Slack
Powered by