Hey, we're fine tuning our codegen for pulumi-scala and that yesterday's announcement from @enough-garden-22763 about automatic plugin installation makes us a bit curious about how this mechanism works. Our codegen generates, the same way as pulumi-java does it,

plugin.json

file per each provider package and puts it into the jar (onto the classpath). These files are then searched for using classpath search using an auxiliary jar (similar to what pulumi-java does) with that functionality to expose all the packages that user has in his program. We currently generate a json of this shape:

{
  "resource": true,
  "name": "<PLUGIN_NAME>",
  "version": "<PLUGIN_VERSION>"
}

Should we pass information about

pluginDownloadURL

here also?

Yes you probably should. Automatic plugin downloads are based on what ever the language runtime returns from GetRequiredPlugins. Most of the language plugins lookup a plugin.json file to get that data, and it does need to include downloadURL.

ok, so that applies to community-based packages, right? because we've checked schemas and official packages do not have the

pluginDownloadURL

(and they also have the capability to get installed without providing

--server

hence the hunch official packages get magical treatment in CLI). so the general rule would be - if schema does contain

pluginDownloadURL

codegen should publish that information in

plugin.json

for

GetRequiredPlugins

official Pulumi providers don't need a pluginDownloadUrl because the default is to assume it comes from github.com/pulumi. So yes you only need to put pluginDownloadURL if the schema specifies it.

Cool! How does your building/publishing infrastructure track what needs to be rebuilt due to change in schema? We are currently just scraping this dataset

curl -s <https://api.github.com/repos/pulumi/registry/contents/themes/default/data/registry/packages> | jq '.[].name'

and so we pull each yaml file and process it to obtain latest version, then install the plugin in that version, pull the schema using

pulumi package get-schema x

and use that schema to codegen the package. This has led us to the

--server

problem and @limited-rainbow-51650 explained to me yesterday that for community packages we should check schema itself (probably by fetching it from schema url available in yaml file?) to see

pulumiDownloadURL

field. Is there anything that is out of order in this algorithm? I guess there's probably a lot given the heuristics we do to try and guess how things work on your side.

I'm not sure, paging providers team to answer that (@enough-garden-22763)

Great to hear from you Łukasz Biały! This honestly sounds like a decent algorithm right now for your needs. My team works with the registry but does not own it, so my understanding of how community packages land there is a little incomplete and I'd have to ask around about this. For our packages the plugin and the schema are built jointly and they land in registry with a push-based mechanism on GHA triggers, so we do not have this problem you all have of polling for providers upgrades. Given that your solution sounds resaonable. You can certainly rely on pulumi package get-schema x once you've installed a plugin to extract a schema from a plugin and you can rely on pulumi plugin install resource pkg ver to get it in the first place. I'm guessing that the curl bit is brittle and you'd prefer to have a RSS feed or some such of new packages in the registry? Does something else break down with community packages? Can I help capture some feature requests on this?

Let me test the new flow to see if we now can cover all 146 of the packages in your registry and I will get back with any possible feature requests. One thing that would definitely be nice to have is to have a way of getting information on schema updates. RSS feed sounds nice but I guess whatever we do, state management (so what was already built in in what compat matrix configuration, eg against which version of schema and language sdk) will be a concern on our side. I think it's fine, we can live with that.

Currently schema only changes upon new provider releases, so I'm used to thinking in these terms, but you're right there might be reasons to update schema more often in the future, like in case of docs corrections.

Wait, so my understanding was that on your side schemas are generated once the APIs on the other side change for native libs and when TF providers get a new version released for classics. Once that happens your CI pipeline picks it up and codegens all the language-specific packages and also prepares the resource plugin (no idea how that last thing works yet). That would mean that you do have an opportunity to publish an event or RSS entry once you generate a new schema, right?

We don't check if there are schema changes. A new TF provider version could be a bugfix release, which we then take in in a new release of our Pulumi bridged provider. This doesn't have any schema changes, but still you should create a release of your Pulumi Scala sdk to match the new version of the plugin binary.

Sure, but the same logic applies - there's an event that triggers the generation of schemas - either something changes in, let's say, AWS API for aws-native, or a new version of AWS TF provider is released and that leads to your pipeline handling that. Is that correct? By handling I mean - generating schema file, running codegen on the schema file, etc.

That's true! There's some detection automation that looks at upstream to find out there's a new version. Then it opens tickets for us in pulumi-aws and such and the pipeline continues from there.

Cool beans, we'll mirror that logic (but scrape your registry instead)! 🙂