Hi everyone, First time writing here, want to say that I’m very exciting about the project! I’m reading docs on Pulumi Packages and wondering if I got it right that native providers can only be implemented with Go? That’s the only example I see in docs It might be obvious by the fact that pulumi itself is written on Go and “native” refers to that 🙂 But then I Iook at the “write once, use in all languages” pic here and it gives me hope that it’s not the case Also, does it support Java generation or maybe that’s at least on the horizon?

Technically you could write a provider in any language that supports gRCP, but getting that raw interface fully correct is hard and we only have an SDK to help with that for Go. There are component providers for Node and Python, and that's what awsx uses. The plan is to support all types of providers in all languages eventually. I've been chipping away at this problem for a couple of months now.

Java generation

Probably, but what exactly do you mean by Java generation? If you mean generating the Java SDK from a provider then yes, we're already doing that for our internal providers.

hey Fraser, thanks for the quick response! yeah, since it uses grpc it should theoretically work with any language. Any success with other languages? How complicated is that?

If you mean generating the Java SDK from a provider then yes, we’re already doing that for our internal providers.

yes, that. I was looking at the provider boilerplate repo and there’s no mention of Java, so I thought maybe it’s not fully supported yet.

// Language is the SDK language.
type Language string

const (
	DotNet Language = "dotnet"
	Go     Language = "go"
	NodeJS Language = "nodejs"
	Python Language = "python"
)

How complicated is that?

Kinda. It's pretty easy to just manually write the interface code, it's the ongoing maintenance cost of keeping it in sync that's really hard. I've been trying to use our code gen work to code gen the interfaces so they can easily be kept up to date. I might just give up with that and get the first version done manually in the new year given how slowly those code gen experiments have gone.

yes, that. I was looking at the provider boilerplate repo and there’s no mention of Java

Ah yeh it's done a slightly different way for Java. See for example the make file for pulumi-aws that generates the Java SDK at https://github.com/pulumi/pulumi-aws/blob/master/Makefile#L46 It's probably just that the boilerplate hasn't been updated for this. Again we've been thinking about how to better do the boilerplate repos given how hard it is to keep templates up to date in all downstream repos, we're probably going to move to something like nodejs packages (and then just like "node run build" rather than "make"), or just shipping support for building providers in the pulumi cli itself, not sure yet but we'll do something.

It’s pretty easy to just manually write the interface code, it’s the ongoing maintenance cost of keeping it in sync that’s really hard

Are you talking about keeping it in sync with API of the system that the provider is for? Is it solved with codegen in Go or is it still a manual process? Also really curious how a plugin like this would work, I assume it won’t work right away. Say I write Java provider. It’s straightforward to use its Resources in code. But on pulumi CLI side it should somehow recognise the plugin, then download it (e.g. jar file) and then be able to run it (e.g. java -jar …). All of it should be built in pulumi CLI, right?

it’s not possible to support a new system with component providers however, right?

Not really, components don't expose the correct Create/Update/Delete lifecycle.

Are you talking about keeping it in sync with API of the system that the provider is for? Is it solved with codegen in Go or is it still a manual process?

Yes, so if we add a new provider method, or add extra information to a parameter, or change the documentation, how do we make sure every SDK gets updated. The Go sdk is currently a manual process, but it's not so bad just keeping one SDK up to date.

Also really curious how a plugin like this would work, I assume it won’t work right away....

Correct, you either have to generate a native binary (No idea if java can do this?) or use a feature I added a features last year (called shimless) which allows us to use the java language host that starts pulumi java programs to also start pulumi java providers. Again shimless is currently only implemented for Go, but it's another bit of the system I hope to bring all languages in sync with.

Yes, so if we add a new provider method, or add extra information to a parameter, or change the documentation, how do we make sure every SDK gets updated.

from what I understand there’s a manually written provider and code-generated sdk, so the only thing that should be manually kept up-to-date is provider or are you talking about kind of a meta sdk, that is sdk you use to create providers?

Correct, you either have to generate a native binary (No idea if java can do this?) or use a feature I added a features last year (called shimless) which allows us to use the java language host that starts pulumi java programs to also start pulumi java providers.

sounds interesting, any docs/examples on shimless?

sounds interesting, any docs/examples on shimless?

Not really, it needs some added for sure. But currently the only thing using it is some tests in pulumi/pulumi. So the go plugin supports "RunPlugin": https://github.com/pulumi/pulumi/blob/master/sdk/go/pulumi-language-go/main.go#L658 And then we have a test provider that tells the engine to use the go runtime: https://github.com/pulumi/pulumi/blob/master/tests/testprovider/PulumiPlugin.yaml

from what I understand there’s a manually written provider and code-generated sdk, so the only thing that should be manually kept up-to-date is provider

or are you talking about kind of a meta sdk, that is sdk you use to create providers?

So yes each provider needs writing manually in one language but then it can generate it's SDKs for each language automatically using our code generators. But the SDK that lets you write providers (So that your not dealing with raw grpc and protobuf.structs) also currently needs manually writing for each language. So far we've only written that fully for Go, and enough for component providers in Python and Node. It's that provider interface that I'd rather not have to write fully manually for every language, because everytime we add a new provider option or method it means manually updating N sdks and that's really easy to end up out of sync.

got it will take a look at shimless as well appreciate your help, Fraser!