Is there a good example of a Terraform bridge repo...
# general
b
Is there a good example of a Terraform bridge repo to start from? All the official Pulumi ones (e.g.
tls
,
random
, etc) depend on quite a bit of Pulumi specific things (e.g.
pulumictl
) and it’s not clear if it is the right place to start. Ideally the repo should also take care of things like building the SDK binaries, etc
s
Hello, and thanks for your interest! I'm Josh and I work on the Providers team. First place to start with a bridged provider is here: https://github.com/pulumi/pulumi-tf-provider-boilerplate And for a finished repo based off the most recent boilerplate, I think our most recent provider is: https://github.com/pulumi/pulumi-artifactory/ We're very interested in your feedback as you go through the process. Please don't hesitate to reach out if anything is unclear or you have suggestions for improvements. Also, the #contribute channel may have some helpful members and historical content, so check it out if you haven't already. Hope this helps!
Oh, I'd also be interested to know what provider you're bridging.
b
Hey Josh - thanks. I did find the boilerplate repo and have been using it since yesterday, and opened a couple of bug reports on pulumi-teraform-bridge and the boilerplate repo (just some small things).
I’ll be honest, it’s been a pretty painful experience - not so much the core bridging experience, but everything on the edges - getting all the packages to compile, dealing with issues where things are hardcoded to the
pulumi
github org or a Pulumi S3 bucket, etc. This is exacerbated by the fact that a Pulumi plugin is a very involved thing (language SDKs, binaries, etc), so it’s not as simple as just getting the repo itself going. I’m currently trying to get the CI scripts to run - probably been doing this for the past 5-6 hours and still not done 😄
I’m trying to bridge the Frontegg provider. Here is the bridged repo: https://github.com/okera/pulumi-frontegg
So one example I’ve hit right now - I don’t really care about the language SDKs beyond Go, at least for now. But the
publish_sdk
step in the CI scripts publish all 4 of them, without really any way to limit it. And so now I need to figure out the PyPI package name and credentials and so on.
I’m happy to chat more in depth if it is helpful.
s
Part of my onboarding was creating a new bridged provider and I had a similar experience. I also agree the CI stuff is not where it needs to be to make non-Pulumi ownership easy enough. I was about to say "publishing an SDK in all supported languages shouldn't be any extra work" and then I read more closely and also remembered that PyPi etc was all set up for me.
We do have a standard naming convention for our SDKs. It might need to be more explicit, but it's at least implied in the README: https://github.com/pulumi/pulumi-aws#installing
In the boilerplate, the README that will eventually become the provider's README at the end of the instructions is here: https://github.com/okera/pulumi-frontegg/blob/main/README-PROVIDER.md This is not necessarily most obvious thing, but I wanted to try to give a clean separation between the boilerplate instructions and the README content for the new provider itself.
b
Yep, I’ve read all of it. It’s mostly small stuff, but it’s small stuff that adds up.
As an example, it would be really convenient if rather than relying on every machien you want to dev this on to have dotnet/python/nodejs installed that you instead have a docker that can do this
(which is kind of what you do in the CI world, but not for the original package authoring)
A few other thoughts: 1. It’s not actually clear how to publish something that isn’t an official Pulumi package. Where should the packaged binaries go? How do I consume them for my Pulumi program (when CI is building them on GitHub)? 2. There’s not a ton of clarity about what needs to be checked into the repo and what doesn’t. As far as I can gather, the output of
make build_sdks
should be checked in, but it’s not called out. 3. Versioning is only tangentially referred to, and it’s not really 100% clear how to get versioning like what the other packages have and how to maintain it.
b
hey! I can probably help here https://github.com/jaxxstorm/pulumi-scaleway
1. packaged binaries go in GitHub releases now, and you just need to update your `pluginDownloadUrl`: https://github.com/jaxxstorm/pulumi-scaleway/blob/master/provider/resources.go#L85 2. yes we should fix that, the SDKs do need to be checked in, then published to providers 3. would love some elaboration there, are you talking versioning across the sdks? or bumping providers?
This is exacerbated by the fact that a Pulumi plugin is a very involved
thing (language SDKs, binaries, etc), so it’s not as simple as just getting the repo itself going.
I’m currently trying to get the CI scripts to run - probably been doing this for the past 5-6 hours and still not done
I totally agree here, it's hard 😞 part of the issue is the multiple languages as you say, and honestly we could do a better job of documenting this
b
I think I mostly got it now, and amusingly I was using your scaleway repo as a sanity check on what I was doing.
I think the last missing piece is the GitHub releases part - my repo doesn't seem to have a release yet even though I ran the CI
@billowy-army-68599 is there an explanation of the variety of secrets needed for the CI actions? For example, the
tag_sdk
step is failing, and I have a feeling it needs
GITHUB_TOKEN
to be defined, but not sure what that needs to be set to, and I can’t find any docs to this effect. If there’s an explanation somewhere of what these all should be, it would be very helpful.
b
you need: `GITHUB_TOKEN`: to publish to github releases `NPM_TOKEN`: to publish to npm `PYPI_PASSWORD`: to publish to pypi `NUGET_PUBLISH_KEY`: to publish nuget packages
@bored-table-20691 I'm happy to jump on a call and get you over the line if that helps later this week?
b
I think I’m pretty close - managed to get the release thing going, I just disabled pypi/nuget for now. For
GITHUB_TOKEN
, what kind of token should this be and what permissions? Does it have to be a personal access token (which I’d prefer to avoid as we want this tied to the org rather than to any individual)?
s
We do it at Pulumi with PulumiBot, a special GH user in our org.
The token prrrrrrobably just needs repo read/write, unless you're using it to comment on issues or things like that.
b
OK, so it sounds like it probably needs to be a PAT, and can’t be a GitHub app token (which would probably more likely be a clientid/secret.
b
github actions should provision a token inside the repo that has perms already
b
Yeah this is why I’m confused that it needs me to specify it manually.
b
do you have it in your CI workflow file?
Specifically, it’s this task that fails:
Copy code
tag_sdk:
    name: tag_sdk
    needs: publish_sdk
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repo
      uses: actions/checkout@v2
    - name: Install pulumictl
      uses: jaxxstorm/action-install-gh-release@v1.2.0
      with:
        repo: pulumi/pulumictl
    - name: Add SDK version tag
      run: git tag sdk/$(pulumictl get version --language generic) && git push origin
        sdk/$(pulumictl get version --language generic)
b
what's the error?
b
https://github.com/okera/pulumi-frontegg/runs/5350443151?check_suite_focus=true
Copy code
! [remote rejected] sdk/0.0.2 -> sdk/0.0.2 (refusing to allow a GitHub App to create or update workflow `.github/workflows/release.yml` without `workflows` permission)
From reading the GitHub forums, seems like it is a permission issue with the token
b
yep, you need this permission on the token: https://github.com/settings/tokens/
b
Right, but this would require me to set up a personal access token (tied to my account)
i.e. no way to connect it to the org
b
yeah we tie ours to a bot user, I'm not sure how to solve that from a github perspective
b
Yeah OK - that's helpful. It would be good if that piece could be documented. Same for the s3 credentials that seem to be needed by a bunch of the CI scripts (or at least make them optional).
How required is the tag_sdk task?
b
the s3 credentials are no longer needed as of last week, we pull binaries from github releases now
b
Hmm - following the steps in CI-mgmt to generate the scripts still generated the need for them last week (Thursday or Friday). Was the change after that?
b
it hasn't been rolled out yet, but if you're using the latest pulumi sdk and terraform bridge, you no longer need to upload to s3
b
Yep, I have that too (need to fill it in): https://github.com/okera/pulumi-frontegg/blob/main/provider/resources.go#L72 But the Actions workflows generates a need for it (and fails if it is not present): https://github.com/okera/pulumi-frontegg/blob/main/.github/workflows/release.yml#L208 I commented it out to make it succeed
This is another one that I ended up disabling as I wasn’t sure if it was needed or what token was needed for it): https://github.com/okera/pulumi-frontegg/blob/main/.github/workflows/release.yml#L104
b
yep you don't need that one
b
@billowy-army-68599 @stocky-restaurant-98004 FYI, this is the commit I ended up with to disable all the things I didn’t care about/didn’t work to get the CI to run: https://github.com/okera/pulumi-frontegg/commit/a0b9da443a26d7f0f46f121b9bdb4c3c85e57182 This is after copying it from ci-mgmt per the instructions there.
s
You have no intention of publishing to the registry, correct?
b
I mean, eventually that could be nice but not a priority right now