Any idea how am I supposed to debug this? ``` pan...
# typescript
Any idea how am I supposed to debug this?
Copy code
panic: interface conversion: interface {} is nil, not map[string]interface {}
    goroutine 44 [running]:
    <{{{0x10ea25960|{{{0x10ea25960>?, 0x1400a4c5210?}}, {0x0?, 0x0?}}, {0x10a4719ce, 0x8})
        /home/runner/go/pkg/mod/ +0x258
    <{0x1400138b200|{0x1400138b200>?, 0x10e8c6c40?, 0x10e8c6c38?}, {0x10ea25488, 0x1400e55ced0}, {0x10ea36290, 0x1400a1b5980}, {0x10e866300?, 0x1400220cb60?}, 0x1, ...)
        /home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:250 +0xea4
    <*wrappedResource).Update.interceptedHandler[...].func10(0x0|*wrappedResource).Update.interceptedHandler[...].func10(0x0>?, {0x10e866300?, 0x1400220cb60?})
        /home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:100 +0x11c
I had a bunch of that last night; i was using
Copy code
TF_LOG=TRACE pulumi up --logtostderr --logflow -v=10 2> out.txt
did you get that when you were passing large nested configuration?
we were having this 'fun' with medialive
no I do not have anything large
I do not know what to look for in the log file though
I was actually trying DEBUG not TRACE
so what I did was go to bottom and work up; it's quite noisy but at somme point there start to be error trace looking things, you keep working up and you'll run into this panic
somewhere before or after, if lucky, you'll see the key it got up to or the data you specified that it doesn't like
it's shit tier debugging but tbh I'm no go guy to be able to make sense of it otherwise
certainly the gRPC stuff makes it hard as heck
the thing is that I've used the same code 2 days ago without issues so now I do not know if it's something on my Mac or something else
I will try to deploy from CI to see if it can help with anything
I suspect maybe a gitlab provider causing this
damn... i had "fun" like that today
IAC is full of leaky abstractions honestly
thanks @gentle-application-59272 I managed to spot the problem, it has something to do with
connector for IAM, I have a
module to get the cert thumbprints but somehow it wanted to change the thumbprint and failed at that.
Folks I am sorry you are affected by this issue. This seems to be the same as - we generally treat panics as P1s and are trying to make progress here but it has been difficult to get a reliable repro. @bright-wall-54528 has been trying to narrow down a repro but has not succeeded yet.
If you happen to have a program that consistently hits this that would be a huge help for us to fix the issue.
I will try to recreate it. I am quite positive I should be able to. Since Gitlab provides 2 thumbprints and OIDC provider needs only one. The panic likely came from aws provider trying to replace the thumbprint.
@great-sunset-355 I happen to be working on GL OIDC for a KubeCon demo. Where did you see 2 certs? (I'm working against the GL SaaS, not self-hosted.)
@stocky-restaurant-98004 This is a piece of my OIDC component where I use the
to get the cert. Certificates are always an array and 90% of internet examples take the first item of the array. IMO you should get the same result for
as well as
(haven't tested this) Then during the provider / package update (not sure where) pulumi wanted to replace 2nd item in the array but to my surprise, the values were the same, perhaps the order was swapped or who knows what went wrong. In the end that triggered a resource update which led to the panic. But I haven't verified that yet.
Copy code
this.gitlabUrl = "<>"
// maybe use tls url "<tls://>" -> `tls://${}:443`
    const cert = tls.getCertificateOutput({
      url: new URL(`/oauth/discovery/keys`, this.gitlabUrl).toString(),
    const audiences: string[] = [this.gitlabUrl.origin, ...(args.audiences ?? [])];

    // this is always like this in the for IAM condition
    const cond = "";

    const gitlabOidcProvider = new aws.iam.OpenIdConnectProvider(
        url: this.gitlabUrl.origin,
        clientIdLists: audiences,
        thumbprintLists: cert.certificates.apply((x) => => o.sha1Fingerprint)),
      { deleteBeforeReplace: true, parent }
Copy code
@enough-garden-22763 I tried to replicate the panic but sadly, no luck. However, I am happy to provide the log and stack checkpoints via private DM if you are interested
@great-sunset-355 The backend in AWS will lowercase the thumbprints if you did not enter them that way, which can cause a spurious diff.
Oh, interesting observation, I think the original thumbprints were lowercase