https://pulumi.com logo
Join Slack
Powered by
Hi guys. I have a huge problem. Somehow, my state ...
# typescript
f
Hi guys. I have a huge problem. Somehow, my state (in an azure blob) got corrupted. It kreeps failing with go exceptions saying that it's missing an id (had several problems and the cli was interrupted mid-way). It seems to me that the problem is connected to a Random password resource in the state, which has "id": "none". The problem is that I need to make sure that this password doesn't change, because it's the administrator password of a mssql instance. Changing it would trigger a msssql instance recreation. I have the actual password which was generated. But everything fails, no refresh or preview works. Is it expected that a RandomPassword resource has 
"id": "none"
? When trying to look up 
pulumi stack history
, nothing seems really useful. All steps are marked as 
version: 0
. (I was hoping to re-import the state to the last working version. When trying to refresh:
Copy code
Previewing refresh (dev):
     Type                                                  Name                               Plan        
     pulumi:pulumi:Stack                                   photon-infrastructure-dev          running     panic: fatal: An assertion has failed: Read ID was empty
     ├─ azure-native:resources:ResourceGroup               resourceGroup                                     
     ├─ azuread:index:Application                          servicePrincipalApp                               
goroutine 186 [running]:rvicePrincipal                     servicePrincipal                                  
     ├─ azure-native:autgithub.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)                
     ├─ random:index:RandomPassword                        serverAdminPassword               	     ├─ azure-native:resources:ResourceGroup               resourceGroup                                  github.com/pulumi/pulumi/sdk/v3@v3.189.0/go/common/util/contract/failfast.go:23
     ├─ azure-native:managedigithub.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf(0xf7?, {0x2d6094b?, 0x5000003502300?}, {0x0?, 0x7f1649741a78?, 0x30?})
     ├─ azure-native:managedidentity:UserAssignedIdentity  userAssignedAppIdentity                                                                                 	github.com/pulumi/pulumi/sdk/v3@v3.189.0/go/common/util/contract/assert.go:35 +0xe8
     ├─ azure-native:privatedns:PrivateZone                dnsPrivateZone                                                                                                                                                                                  github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin.(*provider).Read(0xc001c23720, {0x3528ec0?, 0xc000943da0?}, {{0xc002e8abd0, 0x64}, {0xc002e8ac21, 0x13}, {0xc002e8abf7, 0x28}, {0x0, ...}, ...})
     ├─ azure-native:authorization:RoleAssignment          photonSPResourceGroupContributor                                                                                                              	github.com/pulumi/pulumi/sdk/v3@v3.189.0/go/common/resource/plugin/provider_plugin.go:1397 +0x296      photonSubnet                                      
 ~   ├─ azure-native:privatedns:VirtualNetworkLink   github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*RefreshStep).Apply(0xc001f1c2c0)rtualNetworkLinkName]
     ├─ azure-native:sql:Server                            mssqlServer                                                                 	github.com/pulumi/pulumi/pkg/v3/resource/deploy/step.go:1287 +0x332
     ├─ azure-native:sql:Database                          database                                                                                                                                        github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).continueExecuteStep(0xc000386620, {0x0, 0x0}, 0x15, {0x35399b8, 0xc001f1c2c0})                
                                                                                             	github.com/pulumi/pulumi/pkg/v3/resource/deploy/step_executor.go:468 +0x191
Resources:                                                                                                                                                                 github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).executeStep(0xc000386620?, 0x15?, {0x35399b8?, 0xc001f1c2c0?})
                                             	github.com/pulumi/pulumi/pkg/v3/resource/deploy/step_executor.go:461 +0x225
                                                                                                                           github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).executeChain(0xc000386620, 0x15, {0xc00005ed00?, 0x23?, 0x0?})
                                                                                                                                                                                                                                                         	ithub.com/pulumi/pulumi/pkg/v3/resource/deploy/step_executor.go:379 +0xe5
                                                                         github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).worker(0xc000386620, 0x15, 0x0)
                                                                                                                                                                        	github.com/pulumi/pulumi/pkg/v3/resource/deploy/step_executor.go:658 +0x31d
                                                                                                                                                                                                                                                           created by github.com/pulumi/pulumi/pkg/v3/resource/deploy.newStepExecutor in goroutine 124
                                                                                          	github.com/pulumi/pulumi/pkg/v3/resource/deploy/step_executor.go:708 +0x1ca
Sorry for the terrible formatting. Looking at the state, the only weird thing I can find is this 
"id": "none"
on the Random password:
Copy code
{
                "urn": "urn:pulumi:dev::photon-infrastructure::random:index/randomPassword:RandomPassword::serverAdminPassword",
                "custom": true,
                "id": "none",
                "type": "random:index/randomPassword:RandomPassword",
                "inputs": {
                    "length": 16,
                    "lower": true,
                    "minLower": 1,
                    "minNumeric": 1,
                    "minSpecial": 1,
                    "minUpper": 1,
                    "numeric": true,
                    "special": true,
                    "upper": true
                },
                "outputs": {
                    // ***
                },
                "parent": "urn:pulumi:dev::photon-infrastructure::pulumi:pulumi:Stack::photon-infrastructure-dev",
                "provider": "urn:pulumi:dev::photon-infrastructure::pulumi:providers:random::default_4_18_2::4901e8e5-d58c-48ca-afab-3e95776f42f1",
                "propertyDependencies": {
                    "length": [],
                    "lower": [],
                    "minLower": [],
                    "minNumeric": [],
                    "minSpecial": [],
                    "minUpper": [],
                    "numeric": [],
                    "special": [],
                    "upper": []
                },
                "additionalSecretOutputs": [
                    "bcryptHash",
                    "result"
                ],
                "created": "2025-07-11T15:33:28.881318649Z",
                "modified": "2025-07-11T15:33:28.881318649Z",
                "sourcePosition": "project:///resources/Database.ts#27,33"
            }
Any idea?
l
Hey Rémi. Can you 
pulumi stack export
and get a valid JSON blob that way? That should tell us whether your state is corrupted fundamentally or whether it's "just" this ID attribute that's gone missing, for instance
Ah cool; too slow 😅
f
yeah I pressed enter too fast 😅
you want the whole state?
l
No I think that's enough
If you preview a targeted update on that 
RandomPassword
, does Pulumi give a diff?
E.g. 
pulumi preview -t 'urn:pulumi:dev::photon-infrastructure::random:index/randomPassword:RandomPassword::serverAdminPassword'
f
let me try
l
Job one is getting an ID for that resource
After that you can hopefully import the password you have to hand over it as per https://www.pulumi.com/registry/packages/random/api-docs/randompassword/#import
f
no, the preview fails: 
panic: fatal: A precondition has failed for old: must have an ID if it is custom
so I suppose that it's really that resource which is the problem
l
OK.
f
can't we just assign it an ID 😅? because a random password resource is purely virtual, there is no actual resource in a cloud provider
l
I suspect that IDs don't matter in random resources
Yes, indeed
f
you'd suggest just to put a uuid?
l
I don't think 
random
uses UUIDs, but could be wrong
Regardless, before doing anything -- I'd take a full 
stack export > backup.json
So you have the state we have now
f
ok done
l
Hmmm
Interesting
I just created a 
RandomPassword
in a TS program
And I get 
"id": "none"
f
yeah that was my next attempt
HA
l
Is this a bug?
f
I did think that it could be logical that on a random resource it could be none
and also, none seems very intentional, not like if smth crashes mid-way while writing or smth
l
The panic you're getting though is that a provider 
Read
is potentially returning an empty ID
So when Pulumi asks the random provider hey do you have a thing with this ID
It's returning an empty ID back, even though in theory the ID doesn't matter since the provider is just assigning 
none
f
yeah it's a circle
l
Let's take a look at the gRPC communications between Pulumi and the provider
If you do 
PULUMI_DEBUG_GRPC=grpc.json pulumi refresh
👍 1
f
I suppose that between runs, it just checks hash of all parameters and a seed or smth, so that wouls make sense with no id'
l
And in theory reproduce the panic
You should have a file 
grpc.json
containing the communications between Pulumi and its various bits
And then in theory you can do 
cat grpc.json | jq | less
and look for 
Read
on the 
RandomPassword
resource
And see what request/response is happening
For me I get something like this:
f
the stack is currently locked by 1 lock(s). Either wait for the other process(es) to end or delete the lock file with 
pulumi cancel
. I suppose I need to cancel first
l
Copy code
{
  "method": "/pulumirpc.ResourceProvider/Read",
  "request": {
    "id": "none",
    "urn": "urn:pulumi:dev::refresh-sames::random:index/randomPassword:RandomPassword::x",
    "properties": {
      "__meta": "{\"schema_version\":\"3\"}",
      "__pulumi_raw_state_delta": {
        "obj": {
          "ps": {
            "keepers": {
              "replace": {
                "raw": null
              }
            },
            "overrideSpecial": {
              "replace": {
                "raw": null
              }
            }
          }
        }
      },
      "bcryptHash": "$2a$10$JohdOOZCWEMk7.U.MAQlTO24fDWvO1juQQ3tg9IBzP/r9ktMEntfC",
      "id": "none",
      "length": 16,
      "lower": true,
      "minLower": 0,
      "minNumeric": 0,
      "minSpecial": 0,
      "minUpper": 0,
      "number": true,
      "numeric": true,
      "result": "VfX28w:<fIy!RkWk",
      "special": true,
      "upper": true
    },
    "inputs": {
      "length": 16
    },
    "name": "x",
    "type": "random:index/randomPassword:RandomPassword",
    "resourceStatusAddress": "127.0.0.1:45041",
    "resourceStatusToken": "f073bde2-acb2-4879-a7bb-36f565a4849b"
  },
  "response": {
    "id": "none",
    "properties": {
      "__meta": "{\"schema_version\":\"3\"}",
      "__pulumi_raw_state_delta": {
        "obj": {
          "ps": {
            "keepers": {
              "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
              "value": {
                "replace": {
                  "raw": null
                }
              }
            },
            "overrideSpecial": {
              "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
              "value": {
                "replace": {
                  "raw": null
                }
              }
            }
          }
        }
      },
      "bcryptHash": {
        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
        "value": "$2a$10$JohdOOZCWEMk7.U.MAQlTO24fDWvO1juQQ3tg9IBzP/r9ktMEntfC"
      },
      "id": "none",
      "length": 16,
      "lower": true,
      "minLower": 0,
      "minNumeric": 0,
      "minSpecial": 0,
      "minUpper": 0,
      "number": true,
      "numeric": true,
      "result": {
        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
        "value": "VfX28w:<fIy!RkWk"
      },
      "special": true,
      "upper": true
    },
    "inputs": {
      "length": 16
    }
  },
  "metadata": {
    "kind": "resource",
    "mode": "client",
    "name": "random"
  }
}
Yea your panic probably meant Pulumi didn't release its lock
f
looking for the read...
l
/Read
if you are using 
less
As in, hit slash, then 
Read
f
yeah yeah I know
l
If you are using 
--target
it might limit the scope of the gRPC calls
f
yeah fouend it
l
Also that env var doesn't overwrite, it appends, so if you use it multiple times in a row, you probably want to 
rm grpc.json
each time
Otherwise it grows and grows and it's really hard to find stuff IME 😂
f
shit, what's the cmd in vim for no line number
:set nonumber. I never remember
on the 
"method": "/pulumirpc.ResourceProvider/Diff"
, it found DIFF_NONE
I don't get it, the debug info looks good
l
OK you can delete that/the properties
Just in case any of that stuff is sensitive
But yes, the ID looks OK
What 
pulumi version
are you on?
f
wrt sensitive, it's just hashes I suppose. but anyway. pulumi version v3.190.0
l
Hm. Not sure if @echoing-dinner-19531 knows if Pulumi reads/refresh (
--run-program
) might have changed in the latest versions?
(Apologies for beyond-the-grave ping Fraser)
f
it fails without run program.
in github actions I run on federated credentials / trusted ... / token exchange against azure Entra, while locally I'm on az cli
if I don't 
--run-program
it tries to use OIDC and fails, even if I put useOidc: false (locally)
l
If you don't 
--run-program
, Pulumi only has what is saved in your state to go on, so yea
f
interesting:
Copy code
{
  "method": "/pulumirpc.ResourceProvider/Handshake",
  "request": {
    "engineAddress": "127.0.0.1:33369",
    "rootDirectory": "/home/remi/.pulumi/plugins/resource-random-v4.18.2",
    "programDirectory": "/home/remi/.pulumi/plugins/resource-random-v4.18.2",
    "configureWithUrn": true,
    "supportsViews": true,
    "supportsRefreshBeforeUpdate": true
  },
  "errors": [
    "rpc error: code = Unimplemented desc = Handshake is not yet implemented"
  ],
  "metadata": {
    "kind": "resource",
    "mode": "client",
    "name": "random"
  }
}
l
That's fine
f
ok
l
Pulumi will still work with non-
Handshake
-ing providers
e
reads/refresh (
--run-program
) might have changed in the latest versions?
I moved some ID code, but not in a way that should have affected anything. Just lifting it from the grpc layer to the engine layer.
l
Worth trying to repro with N-1 e.g. 3.188.0?
e
Interesting that it's the literal string 
"none"
👍 1
l
Or whenever prior
I assume that's just the provider
Though I swear I've seen the random provider use IDs before
That weren't UUIDs, just short strings
But perhaps just making that up 😅
f
gemini cli suggested to put the actual plain text password as id 🤣
l
Is 
RandomPassword
a new resource?
As in, recently added to the Pulumi/bridged TF provider?
Because I'm not hallucinating when it comes to 
RandomString
Copy code
{
  "urn": "urn:pulumi:dev::refresh-sames::random:index/randomString:RandomString::y",
  "custom": true,
  "id": "=*4bKSi$Jm{zZGWS",
  "type": "random:index/randomString:RandomString",
  "inputs": {
    "length": 16
  },
  "outputs": {
    "__meta": "{\"schema_version\":\"2\"}",
    "__pulumi_raw_state_delta": {
      "obj": {
        "ps": {
          "keepers": {
            "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
            "ciphertext": "v1:w32vVKjDi3L6BGGV:dBnNkVP3hiv7b4mNV2hgSVcGk3lLZdsGft8juxkMg9qTHO/CuMv4hA=="
          },
          "overrideSpecial": {
            "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
            "ciphertext": "v1:eXwcagA6xxlWWdPs:C8B9x5Mm5p2QFAu4gmAISoKJBxmiXAnuuLiS1ew+gwU4MxX9o3iWZA=="
          }
        }
      }
    },
    "id": "=*4bKSi$Jm{zZGWS",
    "length": 16,
    "lower": true,
    "minLower": 0,
    "minNumeric": 0,
    "minSpecial": 0,
    "minUpper": 0,
    "number": true,
    "numeric": true,
    "result": "=*4bKSi$Jm{zZGWS",
    "special": true,
    "upper": true
  },
  "parent": "urn:pulumi:dev::refresh-sames::pulumi:pulumi:Stack::refresh-sames-dev",
  "provider": "urn:pulumi:dev::refresh-sames::pulumi:providers:random::default_4_18_2::8ee25848-9399-44b0-b22e-1f58f81f1009",
  "propertyDependencies": {
    "length": []
  },
  "created": "2025-08-20T11:35:05.81612676Z",
  "modified": "2025-08-20T11:35:05.81612676Z",
  "sourcePosition": "project:///node_modules/.pnpm/@pulumi+random@4.18.2/node_modules/@pulumi/randomString.ts#160,9"
}
`RandomString`s do get a unique ID
f
yep
hum, it's not pulumi native, it's a tf bridged resource?
l
The random provider bridges the TF random one under the hood, yes
f
the question is, the important info on that resource is its properties. the idea makes little sense when it's smth virtual, so is the id purely random or dependent on the actual input config (like a hash)
l
Well, generally I think most providers/the Pulumi engine still wants IDs to be unique per provider/resource type combination etc.
So while it doesn't matter what it is (could be any string)
f
because if it's pure random, I can just assign whatever I want
l
You probably don't want >1 with "none"
e
yeh I think RandomPassword just has "none" as an ID
I think that's a red herring for this
l
The question though is if you had two `RandomPassword`s with 
id: "none"
, that'd be OK?
I guess they'd have different URNs, so yea
OK
f
that's really strange
l
Then potential red herring
(False alarm; apologies for idiom)
f
you think I try adding smth like the one you got for your random string? =*4bKSi$Jm{zZGWS
l
I suspect it won't make a difference
But you could try
I don't know why I suspect that 😂
f
ok, I try to modify it and stack import?
l
Yes
e
Do any of the resources have the empty string for their ID
or ID missing from the JSON completly
f
let me see
no 
"id": null
or 
"id": ""
e
and every resource has an "id" field?
because if it was "" the engine would have just not written it to the JSON at all I think
f
hum
jq -s 'map(select(has("id") | not))[]' infra-dev-state.json
doesn't seem correct
e
Think you need to index into the resources field first 
jq '.resources | map(select(has("id") | not))' infra-dev-state.json
f
yeah I was writing exactly that
🙂
cat infra-dev-state.json | jq '.deployment.resources | map(select(has("id") | not))'
found a few things'
the stack itself has no id
AND
Copy code
{
  "urn": "urn:pulumi:dev::photon-infrastructure::azure-native:network:PrivateDnsZoneGroup::dnsPrivateZoneGroup",
  "custom": true,
  "type": "azure-native:network:PrivateDnsZoneGroup",
  "inputs": {
    "name": "default",
    "privateDnsZoneConfigs": [
      {
        "name": "<http://privatelink.database.windows.net|privatelink.database.windows.net>",
        "privateDnsZoneId": "/subscriptions/a96ab92e-6d33-4fe6-b216-3bbd86c7fdda/resourceGroups/photon-rg/providers/Microsoft.Network/privateDnsZones/privatelink.database.windows.net"
      }
    ],
    "privateEndpointName": "photonPrivateEndpoint",
    "resourceGroupName": "photon-rg"
  },
  "parent": "urn:pulumi:dev::photon-infrastructure::pulumi:pulumi:Stack::photon-infrastructure-dev",
  "dependencies": [
    "urn:pulumi:dev::photon-infrastructure::azure-native:privatedns:PrivateZone::dnsPrivateZone",
    "urn:pulumi:dev::photon-infrastructure::azure-native:privatedns:VirtualNetworkLink::vnetLink",
    "urn:pulumi:dev::photon-infrastructure::azure-native:network:PrivateEndpoint::privateEndpoint",
    "urn:pulumi:dev::photon-infrastructure::azure-native:resources:ResourceGroup::resourceGroup"
  ],
  "provider": "urn:pulumi:dev::photon-infrastructure::pulumi:providers:azure-native::default_3_5_1::ccef02bb-0f8a-4320-817a-e3d1c354da44",
  "propertyDependencies": {
    "name": [],
    "privateDnsZoneConfigs": [
      "urn:pulumi:dev::photon-infrastructure::azure-native:privatedns:PrivateZone::dnsPrivateZone"
    ],
    "privateEndpointName": [
      "urn:pulumi:dev::photon-infrastructure::azure-native:network:PrivateEndpoint::privateEndpoint"
    ],
    "resourceGroupName": [
      "urn:pulumi:dev::photon-infrastructure::azure-native:resources:ResourceGroup::resourceGroup"
    ]
  },
  "created": "2025-08-08T12:03:44.70702762Z",
  "modified": "2025-08-08T12:03:44.70702762Z",
  "sourcePosition": "project:///resources/Network.ts#98,26"
}
was focused on the password all this time
tyhe good news is that I can mess up with this one, easy to recreate
but changing the db server password would force a recreate
AND this should be easy to find the id
e
Might be in stack history you can find what the ID was, or just find it on azure. Not sure the format but might just be the azure uuid
Very odd that it's missing though 🤔 any idea what operation happened to lead up to this?
f
yeah I think that at some point there was a problem with that
but it's easy to find out about the id
it's directly the cloud provider id
= /subscriptions/xxx/ResourceGroups/xxx/providers/ ...
OK, I need to run to fetch my kids, but thx very much guys
I think I should be able to fix that one. Find out the id, replace it and stack import, right?
(in fact, the id is more or less the url path in Azure portal)
e
yup
f
it worked, by the way
what's funny is that the resource declaration did not have an id, but that resource was part of my outputs, and I realised that the same resource in the outputs (from before I suppose) had the id
l
Nice!
f
Love those comments on my PRs, btw
Open in Slack
Previous
Powered by