Hey guys - I am having trouble using Pulumi Packag...
# general
p
Hey guys - I am having trouble using Pulumi Packages with a private repository.. Can anyone please help? I am building the package successfully and I have the binary + package in Github (using Github releases), and I am able to do npm install locally and fetch the package as well. The problem is when I run ‘pulumi up’ - I get the following error:
Copy code
warning: resource plugin xyz is expected to have version >=0.0.23, but has 0.0.1; the wrong version may be on your path, or this may be a bug in the plugin
My suspect is that it is unable to install the pulumi plugin, and that my
pluginDownloadURL
inside my schema.yaml is wrong - the issue is that I can’t find any examples online on how to do it properly... my schema.yaml file is as follows:
Copy code
# yaml-language-server: $schema=<https://raw.githubusercontent.com/pulumi/pulumi/master/pkg/codegen/schema/pulumi.json>
---
name: xyz
pluginDownloadURL: <github://api.github.com/amitOrganization/pulumi-amit-packages>
resources:
  xyz:index:StaticPage:
    isComponent: true
    inputProperties:
      indexContent:
        type: string
        description: The HTML content for index.html.
    requiredInputs:
      - indexContent
    properties:
      bucket:
        "$ref": "/aws/v4.0.0/schema.json#/resources/aws:s3%2Fbucket:Bucket"
        description: The bucket resource.
      websiteUrl:
        type: string
        description: The website URL.
    required:
      - bucket
      - websiteUrl
language:
  go:
    generateResourceContainerTypes: true
    importBasePath: <http://github.com/pulumi/pulumi-xyz/sdk/go/xyz|github.com/pulumi/pulumi-xyz/sdk/go/xyz>
  nodejs:
    packageName: "@amitOrganization/xyz"
    dependencies:
      "@pulumi/aws": "^4.0.0"
    devDependencies:
      typescript: "^3.7.0"
e
You probably have a copy of the provider binary on PATH (likely in go/bin), the engine always prefers to use the binary on PATH vs what's in the plugin cache. You can turn this off with the envvar PULUMI_IGNORE_AMBIENT_PLUGINS.
p
@echoing-dinner-19531 Thanks for the quick answer! How can I set this env variable? Can’t find in the documentations anything about it..
PULUMI_IGNORE_AMBIENT_PLUGINS=true
?
e
Yeh it's not documented anywhere, but "true" or "1" will enable it.
p
@echoing-dinner-19531 So looks like I get a different error message which is good, but still it fails:
Copy code
error: Could not automatically download and install resource plugin 'pulumi-resource-xyz' at version v0.0.23, install the plugin using `pulumi plugin install resource xyz v0.0.23`.
Underlying error: error downloading plugin xyz to file: failed to download plugin: xyz-0.0.23: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-xyz-v0.0.23-darwin-arm64.tar.gz>
Maybe my
pluginDownloadURL
is still wrong? its value is:
Copy code
<github://api.github.com/amitOrganization/pulumi-rise-packages>
e
Have you got an envvar GITHUB_TOKEN set with a github pac token set? Plugin downloads always fall back to get.pulumi.com (I should probably fix that), but it probably did the fallback because the github download failed. If you turn on verbose logs (-v11 --logtostderr) I think it should print the github requests
p
@echoing-dinner-19531 Yup I got
GITHUB_TOKEN=ghp_...
set (I set it in my terminal before running pulumi up) Looking at the logs I found kind of the same error message:
Copy code
I0206 20:37:41.095459   48664 plugins.go:788] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23>
Underlying error: error downloading plugin xyz to file: failed to download plugin: xyz-0.0.23: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-xyz-v0.0.23-darwin-arm64.tar.gz>\u003c{%reset%}\u003e\n","color":"raw","severity":"error"}},{"sequence":4,"timestamp":1675708662,"summaryEvent":{"maybeCorrupt":false,"durationSeconds":2,"resourceChanges":{"create":1},"PolicyPacks":{}}}]}
The part
pulumi-resource-xyz-v0.0.23-darwin-arm64.tar.gz
looks correct, even though entering: https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23 In my browser or doing cURL - I get not found error
e
Have you got a github release for that version? e.g. if you look at one of our providers we have a GH release with all the .tar.gzs for the different platforms in a release notice: https://github.com/pulumi/pulumi-aws/releases/tag/v5.29.1 That's what the downloader is looking for
p
@echoing-dinner-19531 Yes - thats the release, maybe I did it incorrectly?
e
No that looks ok. I think the download url is wrong, because it hit:
Copy code
<https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23>
That's our org, it should of hit your org...
Was there another
full plugin download url:
first?
p
@echoing-dinner-19531 Nope there is only one log containing full plugin download url:
Copy code
I0206 20:53:03.655370   49041 plugins.go:788] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23>
So in that case, where do I configure it? in the schema.yaml file inside my package? which field should I use?
e
The schema looks right, if you look at the SDKs generated they should have some utility files with the url in it as well, can you check those are pointing at your org?
p
@echoing-dinner-19531 I see a
utilities.ts
file with this part:
Copy code
/** @internal */
export function resourceOptsDefaults(): any {
    return { version: getVersion(), pluginDownloadURL: "<github://api.github.com/@amitOrganization/pulumi-rise-packages>" };
}
e
@amitOrganization
looks odd? where did the
@
come from?
p
@echoing-dinner-19531 I’ve got it in the shcema.yaml file:
Copy code
packageName: "@amitOrganization/xyz"
and in scripts/install-pulumi-plugin.js:
Copy code
var res = childProcess.spawnSync("pulumi", ["plugin", "install", "--server", "<github://api.github.com/@amitOrganization[/pulumi-rise-packages]>"].concat(args), {
    stdio: ["ignore", "inherit", "inherit"]
});
e
Is your github org actually named
@amitOrganization
I've never seen a github org with an at symbol in the name?
Oh wait that's package name, that probably should have "@" because npm
This might be a codegen bug...
p
@echoing-dinner-19531 It has a different name but don’t know if I can share it, doesn’t start with ‘@’ though
btw I took a look inside my GitHub actions logs for the release workflow (I used almost the same one as in: https://github.com/pulumi/pulumi-component-provider-go-boilerplate/blob/main/deployment-templates/release.yml ) and saw this part:
Copy code
[4/4] Building fresh packages...
success Saved lockfile.
$ node scripts/install-pulumi-plugin.js resource xyz ${VERSION}
[resource plugin xyz-0.0.24] installing
error: [resource plugin xyz-0.0.24] downloading from <github://api.github.com/amitOrganizaton/pulumi-rise-packages>: failed to download plugin: xyz-0.0.24: 404 HTTP error fetching plugin from <https://api.github.com/repos/amitOrganizaton/pulumi-rise-packages/releases/tags/v0.0.24>. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: <https://github.com/settings/tokens>

There was an error installing the resource provider plugin. You may try to manually installing the plugin by running `pulumi plugin install resource xyz v0.0.24`
Done in 13.16s.
e
Ah there's your error!
Copy code
404 HTTP error fetching plugin from <https://api.github.com/repos/amitOrganizaton/pulumi-rise-packages/releases/tags/v0.0.24>
The cli can't access that repo
Probably because GITHUB_TOKEN isn't set, or is set to a token with the wrong permissions
p
@echoing-dinner-19531 I’ve added it in multiple places inside my action \ makefile, still the same error... I see that it is run from the yarn install command
Copy code
"install": "node scripts/install-pulumi-plugin.js resource xyz "
and inside install-pulumi-plugin there is this part:
Copy code
var res = childProcess.spawnSync("pulumi", ["plugin", "install", "--server", "<github://api.github.com/amitOrganization[/pulumi-rise-packages]>"].concat(args), {
    stdio: ["ignore", "inherit", "inherit"]
});
.
.
} else if (res.error || res.status !== 0) {
    console.error("\nThere was an error installing the resource provider plugin. " +
            "You may try to manually installing the plugin by running " +
            "`pulumi plugin install " + args.join(" ") + "`");
}
which the error comes from maybe the url is still incorrect? should it be with the square brackets? (e.g
[/pulumi-rise-packages]
)
e
It shouldn't have the square brackets...
What's your pluginDownloadUrl in schema.yaml say exactly?
Cause I'm struggling to see where square brakets could get added by the generator code 😕
p
@echoing-dinner-19531 I see - its just that the documentation is kinda confusing regarding the square brackets😅 my
pluginDownloadUrl
is:
Copy code
pluginDownloadURL: <github://api.github.com/amitOrganization/pulumi-rise-packages>
So I removed the square brackets from the nstall-pulumi-plugin and it still didn’t help, same error from the CICD ..
e
Ah that's meant to be that the repository name is optional!
p
Interesting, I will try without!
e
If you just have "github://api.github.com/amitOrganization" the engine will look at your package name for the repo name and if your package is "rise-packages" then it will result in the same URL of "github://api.github.com/amitOrganization/pulumi-rise-packages"
p
@echoing-dinner-19531 Sorry for the long response didn’t have time to keep working on it just re-ran everything, and stumbled upon a new error on the Build sdk step for nodejs 😞 So I started the repo again from scratch, changed the bare minimum and it looks like the error is happening again - which makes me think this has something to do with the pulumi converter \ codegen? two lines for copy-paste purposes:
Copy code
Error: node_modules/@types/node/ts4.8/util.d.ts(1485,42): error TS1005: ',' expected.
Error: node_modules/@types/node/ts4.8/util.d.ts(1485,44): error TS1068: Unexpected token. A constructor, method, accessor, or property was expected.
Full error:
e
Ah
Yes @types/node had a release like yesterday, for TS4 and pulumi is still tied to TS3 and the syntax has changed.
I had to go round and fix all our references of that from "latest" to "ts3.7" see https://github.com/pulumi/pulumi/pull/12091/files
p
@echoing-dinner-19531 Thank you so much!! Fixed it
going back to the same old errors now 😅
e
Still around the pluginDownloadURL?
p
@echoing-dinner-19531 yeah basically.. even though I am trying different things, still trying to handle this error from the build sdk stage:
Copy code
error: [resource plugin xyz-0.0.6] downloading from : failed to download plugin: xyz-0.0.6: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-xyz-v0.0.6-linux-amd64.tar.gz>
There was an error installing the resource provider plugin. You may try to manually installing the plugin by running `pulumi plugin install resource xyz v0.0.6`
e
Right, same as before but to summerize things to check: pluginDownloadURL in the schema that GITHUB_TOKEN is set to access the repo that there's a github release in the repo for 0.0.6 with the expected .tar.gz files in it
p
So I added these two fields:
Copy code
publisher: amitOrganization
repository: my-repo
Which now gave me this error which is better, it’s looking at the right spot now:
Copy code
error: [resource plugin xyz-0.0.7] downloading from <github://api.github.com/amitOrganization>: failed to download plugin: xyz-0.0.7: 404 HTTP error fetching plugin from <https://api.github.com/repos/amitOrganization/pulumi-xyz/releases/tags/v0.0.7>. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: <https://github.com/settings/tokens>
Which makes me believe I need to pass GITHUB_TOKEN some place else (I am confident it has the right permissions) p.s -regarding the things you mentioned:
Copy code
pluginDownloadURL: <github://api.github.com/amitOrganization>
plus the release does exists.
ways I have tried setting up the GITHUB_TOKEN env: 1.
Copy code
env:
  GITHUB_TOKEN: ${{ secrets.access_token_github }}
run: |
  make build_${{ matrix.language }}_sdk
2.
Copy code
env:
  GITHUB_TOKEN: ${{ secrets.access_token_github }}
run: |
  export GITHUB_TOKEN=${{ env.GITHUB_TOKEN }}
  make build_${{ matrix.language }}_sdk
and also inside the makefile:
Copy code
GITHUB_TOKEN         := ${GITHUB_TOKEN}
e
p
@echoing-dinner-19531 from my understanding it’s the name of your github secret , can be anything, for example I have an organization secret named
USERS_S3_FILE_PATH
which is totally custom and works great
but let me try setting up one named GITHUB_TOKEN - worth a shot
e
yeh I'm not a github actions expert, just the docs style is what I'm used to seeing so something to double check