https://pulumi.com logo
Join Slack
Powered by
Hi, While migration from Pulumi-cloud to S3+dynamo...
# aws
w
Hi, While migration from Pulumi-cloud to S3+dynamoDB for state backend, while importing I am getting this error:
s
DIY backends do not support the concepts of organizations like Pulumi cloud does. The organization must be called (literally) 
organization
.
I don't know all the spots where that might need to be changed, so I would recommend a string search/replace.
w
so if I have backend in Pulumi-cloud in this format: org/project/stack and in S3-backend as stack-name, how to switch? So I am using a Gitlab-pipeine for this migration. Can you specify the exact steps??
Consider this code:
After import, I am getting that "Organizations" error
m
Your stack references have to be 
organization/$PROJECT/$STACK
(and they only work for states stored in the same 
.pulumi
path in the same S3 bucket)
w
can you please elaborate more? I am sorry I didn't got it.
m
Right now, your stack references go to 
my-org/my-project/my-stack
, where 
my-org
is your organization on Pulumi Cloud. With a self-hosted backend, they now go to 
organization/my-project/my-stack
. So all you should have to do is replace "my-org" with "organization"
I don't think there are other places that require an organization to be specified
w
But all this data is being exported into a json file and then Imported into S3 backend using the same file. How can I change this within the file? Or like is there any other way to migrate without without exporting into a json file and then importing using the same file?
m
You will have to make that change both in your program and in your state file
e
Do you know if it's actually the 
import
command failing there?
or is it `preview`/`up` after the import?
m
@wooden-portugal-65883 In your state, you'll have resources with 
"urn": "urn:pulumi:my-stack::my-project::pulumi:pulumi:StackReference::my-org/my-other-project/my-stack"
and I believe you'll have to rewrite those for it to work
But when it's not the import that's failing, it should be sufficient to just change that within your program and have the next "up" resolve the problem
w
It is the Import that is failing, I have "pulumi up" commented for now :
m
Do you have stack references in your state?
e
Can you try passing 
--stack=dev
to the import command?
w
@echoing-dinner-19531 I did, but still getting that.
so Like what process will you suggest if I want to migrate my state from Pulumi-cloud to S3+dynamoDB as backend?
e
Might be worth running with 
-v10 --logtostderr
to see if theres any more logs about what is actually returning that error
w
so I exported the json and downloaded it as an artifact, urn are in this format:
"urn": "urnpulumidev:pulumi migration testpulumipulumiStack:pulumi-migration-test-dev"
urn": "urnpulumidev:pulumi migration testpulumiprovidersaws:aws-provider"
m
Are these your only URNs?
w
No, I have total 4-5 resources like s3, sqs...
urnpulumidev:pulumi migration testawss3/bucket:appBucket
m
OK, but do you have a stack reference anywhere?
What does 
cat my_state.json | grep my-org
give you?
Also, do you have any mention of your org in one of the 
Pulumi.yaml
or 
Pulumi.<stack>.yaml
files?
Finally, have you confirmed that you can init a new stack using your backend? E.g.,
Copy code
cd my/source/dir
pulumi login s3://...
pulumi stack init smoke-test
w
I have reference of my org in secrets provider: secrets_providers": { "type": "service", "state": { "url": "https://api.pulumi.com", "owner": "ABC", "project": "pulumi-migration-test", "stack": "dev" } },
where ABC is my org's name
m
OK, that might be it. You cannot have a Pulumi-hosted secrets provider paired with a custom backend, I think (https://www.pulumi.com/docs/iac/concepts/secrets/#configuring-secrets-encryption)
Do you already have a KMS key to use with your new stack?
w
So I need to delete this block From the json file?
m
No, don't do that, that will most likely lose you all your secrets 🙂
Instead, you want to change your provider as described here: https://www.pulumi.com/docs/iac/concepts/secrets/#changing-the-secrets-provider-for-a-stack
w
Yes I have a KMS key with S3+dynamoDB
m
I think you need to do the secrets provider change before exporting your state
So that the state is encrypted with the new key
w
but then whether it will be able to export the state if I changed the secrets provider?
e
yeh you can change secret provider then export
w
ok let me try
e
We should probably just hard block imports to the diy backend if we see the service secret provider, that's never gonna work
w
Okay
ok so this time, the secrets-provider block completely changed, it has rul and encryptedkey in the block.
but what about the urn?? Whether these also need to be changed before migrated?
e
org names aren't in URNs so I don't think they need changing
w
Still getting the same error:
m
The error message originates here: https://github.com/pulumi/pulumi/blob/e60c951303bd2dd51cbb8573804feb72bdc1c21b/pkg/backend/diy/store.go#L171 This function is parsing a stack reference in the format 
<org>/<project>/<stack>
. So somewhere there's a stack reference. Perhaps Fraser knows where such references are created/used in the import process. Are you able to create and up a new stack without importing anything? So just init a new stack but then immediately run 
pulumi up
with an empty program, rather than importing? This would help identify whether the problem is within the state you're importing or within your project/backend configuration.
The only place that GitHub's code search turns up for me where the 
ParseReference
function I linked to above is used is when parsing a stack reference: https://github.com/pulumi/pulumi/blob/e60c951303bd2dd51cbb8573804feb72bdc1c21b/pkg/backend/diy/backend.go#L679 This is used quite a bit in the CLI (https://github.com/search?q=repo%3Apulumi%2Fpulumi%20parseStackReference&amp;type=code). I'm fairly certain that you're already failing on the 
RequireStack
call that 
pulumi import
does early on. It reads the project, gets the backend, and then parses a stack reference: https://github.com/pulumi/pulumi/blob/e60c951303bd2dd51cbb8573804feb72bdc1c21b/pkg/cmd/pulumi/stack/io.go#L137 So this points to the problem being with your environment/project/backend setup. Can you take the exported stack state to a different, clean environment (a new container or a fresh VM) where you have not logged into Pulumi previously? In the screenshot above it looks like you're logging in multiple times and remain in the same project directory, which might contain configuration files that are still pointing to the Pulumi Cloud backend.
OK, another, simpler question: What does 
pulumi stack --show-name -Q
give you right before the import? This will print the full stack reference, including the organization and project
Open in Slack
PreviousNext
Powered by