Hi all, I need something similar to a dynamic provider in Go. As this isnt supported, how have other people been doing this? My initial use case is creating a MySQL schema after creating an rds instance.

Not a replacement for dynamic providers, but automation api can help you manage related workflows like database migraitons: https://github.com/pulumi/automation-api-examples/tree/main/go/database_migration

thanks, from the looks of it this is basically creating a "dynamic stack" and then using up/destroy

thanks, from the looks of it this is basically creating a "dynamic stack" and then using up/destroy

Not sure what you mean by this. This program invokes the pulumi engine with a pulumi program and generates a state file.

yes, thats what I meant 🙂

Have you had the chance to read about the automation api? https://pkg.go.dev/github.com/pulumi/pulumi/sdk/v2/go/x/auto https://www.pulumi.com/blog/automation-api/ There is some conceptual information that might be helpful.

I guess basically by ensuring an idempotent creation/update?

Calling

stack.up()

will be idempotent (running the pulumi program). It would be your job to make sure that your database migrations are invoked in the program in an idempotent fashion.

We also really want to use Go for Pulumi if possible (90% of our codebase is already Go), but the lack of dynamic providers is challenging - we need to hit our own application APIs at various points in the deploy process for each customer. In the GitHub issue and this thread the most common workaround suggested is the automation API - this allows a single Go program to chain multiple different stacks together and pass outputs between them, for example. It's a great feature, but then we can't get the optimal parallelism for resource graph resolution - I have to add sequential barriers to break the deploy process at one or more points, which is what we already do with CloudFormation nested stacks and what I was hoping to avoid. I got excited by lambda.GetInvocation and resource transformations as places to insert arbitrary code, but these run during resource graph construction (which is already arbitrary code) and not during the update phase. Is there any way to inject arbitrary Go code during the resource creation process, particularly for AWS? I don't even need a full CRUD model, if I could just invoke a Lambda function as resources are updated, we'd be set.

@rhythmic-nail-73192 rather than implementing a dynamic resource, you could implement a full custom resource or component resource in a provider (make your own provider). https://github.com/justinvp/vpc/blob/main/provider/cmd/pulumi-resource-vpc/vpc.go I have done similar things to model local processes as a part of pulumi workflows (run arbitrary commands, shell scripts, binaries, etc, as a part of workflow).

Neat, thank you @lemon-agent-27707! And then how do I register the custom provider? Or would I re-compile pulumi from source? Is this a valid (albeit hacky) workaround?

output.Apply

blocks until the input is ready, so I could trigger actions as a side-effect of the apply e.g.

bucket = s3.Bucket(...)

def side_effect(x: str) -> str:
    lambda._get_invocation(...)
    return x

# Force this resource to wait until bucket is finished, and our custom code has run
other_bucket = s3.Bucket(
    logging_prefix=bucket.id.apply(
        side_effect
    )
)

If implemented in go, providers are binaries that live on your path. Not sure what platform you are on, but on mac you can see this in ~/.pulumi/plugins

Awesome! Good to know, thanks