https://pulumi.com logo
#general
Title
# general
p

prehistoric-sandwich-7272

02/06/2023, 3:58 PM
Hey guys - I am having trouble using Pulumi Packages with a private repository.. Can anyone please help? I am building the package successfully and I have the binary + package in Github (using Github releases), and I am able to do npm install locally and fetch the package as well. The problem is when I run ‘pulumi up’ - I get the following error:
Copy code
warning: resource plugin xyz is expected to have version >=0.0.23, but has 0.0.1; the wrong version may be on your path, or this may be a bug in the plugin
My suspect is that it is unable to install the pulumi plugin, and that my
pluginDownloadURL
inside my schema.yaml is wrong - the issue is that I can’t find any examples online on how to do it properly... my schema.yaml file is as follows:
Copy code
# yaml-language-server: $schema=<https://raw.githubusercontent.com/pulumi/pulumi/master/pkg/codegen/schema/pulumi.json>
---
name: xyz
pluginDownloadURL: <github://api.github.com/amitOrganization/pulumi-amit-packages>
resources:
  xyz:index:StaticPage:
    isComponent: true
    inputProperties:
      indexContent:
        type: string
        description: The HTML content for index.html.
    requiredInputs:
      - indexContent
    properties:
      bucket:
        "$ref": "/aws/v4.0.0/schema.json#/resources/aws:s3%2Fbucket:Bucket"
        description: The bucket resource.
      websiteUrl:
        type: string
        description: The website URL.
    required:
      - bucket
      - websiteUrl
language:
  go:
    generateResourceContainerTypes: true
    importBasePath: <http://github.com/pulumi/pulumi-xyz/sdk/go/xyz|github.com/pulumi/pulumi-xyz/sdk/go/xyz>
  nodejs:
    packageName: "@amitOrganization/xyz"
    dependencies:
      "@pulumi/aws": "^4.0.0"
    devDependencies:
      typescript: "^3.7.0"
e

echoing-dinner-19531

02/06/2023, 4:28 PM
You probably have a copy of the provider binary on PATH (likely in go/bin), the engine always prefers to use the binary on PATH vs what's in the plugin cache. You can turn this off with the envvar PULUMI_IGNORE_AMBIENT_PLUGINS.
p

prehistoric-sandwich-7272

02/06/2023, 4:38 PM
@echoing-dinner-19531 Thanks for the quick answer! How can I set this env variable? Can’t find in the documentations anything about it..
PULUMI_IGNORE_AMBIENT_PLUGINS=true
?
e

echoing-dinner-19531

02/06/2023, 4:55 PM
Yeh it's not documented anywhere, but "true" or "1" will enable it.
p

prehistoric-sandwich-7272

02/06/2023, 6:30 PM
@echoing-dinner-19531 So looks like I get a different error message which is good, but still it fails:
Copy code
error: Could not automatically download and install resource plugin 'pulumi-resource-xyz' at version v0.0.23, install the plugin using `pulumi plugin install resource xyz v0.0.23`.
Underlying error: error downloading plugin xyz to file: failed to download plugin: xyz-0.0.23: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-xyz-v0.0.23-darwin-arm64.tar.gz>
Maybe my
pluginDownloadURL
is still wrong? its value is:
Copy code
<github://api.github.com/amitOrganization/pulumi-rise-packages>
e

echoing-dinner-19531

02/06/2023, 6:32 PM
Have you got an envvar GITHUB_TOKEN set with a github pac token set? Plugin downloads always fall back to get.pulumi.com (I should probably fix that), but it probably did the fallback because the github download failed. If you turn on verbose logs (-v11 --logtostderr) I think it should print the github requests
p

prehistoric-sandwich-7272

02/06/2023, 6:43 PM
@echoing-dinner-19531 Yup I got
GITHUB_TOKEN=ghp_...
set (I set it in my terminal before running pulumi up) Looking at the logs I found kind of the same error message:
Copy code
I0206 20:37:41.095459   48664 plugins.go:788] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23>
Underlying error: error downloading plugin xyz to file: failed to download plugin: xyz-0.0.23: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-xyz-v0.0.23-darwin-arm64.tar.gz>\u003c{%reset%}\u003e\n","color":"raw","severity":"error"}},{"sequence":4,"timestamp":1675708662,"summaryEvent":{"maybeCorrupt":false,"durationSeconds":2,"resourceChanges":{"create":1},"PolicyPacks":{}}}]}
The part
pulumi-resource-xyz-v0.0.23-darwin-arm64.tar.gz
looks correct, even though entering: https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23 In my browser or doing cURL - I get not found error
e

echoing-dinner-19531

02/06/2023, 6:45 PM
Have you got a github release for that version? e.g. if you look at one of our providers we have a GH release with all the .tar.gzs for the different platforms in a release notice: https://github.com/pulumi/pulumi-aws/releases/tag/v5.29.1 That's what the downloader is looking for
p

prehistoric-sandwich-7272

02/06/2023, 6:47 PM
@echoing-dinner-19531 Yes - thats the release, maybe I did it incorrectly?
e

echoing-dinner-19531

02/06/2023, 6:51 PM
No that looks ok. I think the download url is wrong, because it hit:
Copy code
<https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23>
That's our org, it should of hit your org...
Was there another
full plugin download url:
first?
p

prehistoric-sandwich-7272

02/06/2023, 6:54 PM
@echoing-dinner-19531 Nope there is only one log containing full plugin download url:
Copy code
I0206 20:53:03.655370   49041 plugins.go:788] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-xyz/releases/tags/v0.0.23>
So in that case, where do I configure it? in the schema.yaml file inside my package? which field should I use?
e

echoing-dinner-19531

02/06/2023, 6:55 PM
The schema looks right, if you look at the SDKs generated they should have some utility files with the url in it as well, can you check those are pointing at your org?
p

prehistoric-sandwich-7272

02/06/2023, 6:57 PM
@echoing-dinner-19531 I see a
utilities.ts
file with this part:
Copy code
/** @internal */
export function resourceOptsDefaults(): any {
    return { version: getVersion(), pluginDownloadURL: "<github://api.github.com/@amitOrganization/pulumi-rise-packages>" };
}
e

echoing-dinner-19531

02/06/2023, 6:57 PM
@amitOrganization
looks odd? where did the
@
come from?
p

prehistoric-sandwich-7272

02/06/2023, 7:00 PM
@echoing-dinner-19531 I’ve got it in the shcema.yaml file:
Copy code
packageName: "@amitOrganization/xyz"
and in scripts/install-pulumi-plugin.js:
Copy code
var res = childProcess.spawnSync("pulumi", ["plugin", "install", "--server", "<github://api.github.com/@amitOrganization[/pulumi-rise-packages]>"].concat(args), {
    stdio: ["ignore", "inherit", "inherit"]
});
e

echoing-dinner-19531

02/06/2023, 7:01 PM
Is your github org actually named
@amitOrganization
I've never seen a github org with an at symbol in the name?
Oh wait that's package name, that probably should have "@" because npm
This might be a codegen bug...
p

prehistoric-sandwich-7272

02/06/2023, 7:02 PM
@echoing-dinner-19531 It has a different name but don’t know if I can share it, doesn’t start with ‘@’ though
btw I took a look inside my GitHub actions logs for the release workflow (I used almost the same one as in: https://github.com/pulumi/pulumi-component-provider-go-boilerplate/blob/main/deployment-templates/release.yml ) and saw this part:
Copy code
[4/4] Building fresh packages...
success Saved lockfile.
$ node scripts/install-pulumi-plugin.js resource xyz ${VERSION}
[resource plugin xyz-0.0.24] installing
error: [resource plugin xyz-0.0.24] downloading from <github://api.github.com/amitOrganizaton/pulumi-rise-packages>: failed to download plugin: xyz-0.0.24: 404 HTTP error fetching plugin from <https://api.github.com/repos/amitOrganizaton/pulumi-rise-packages/releases/tags/v0.0.24>. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: <https://github.com/settings/tokens>

There was an error installing the resource provider plugin. You may try to manually installing the plugin by running `pulumi plugin install resource xyz v0.0.24`
Done in 13.16s.
e

echoing-dinner-19531

02/06/2023, 7:05 PM
Ah there's your error!
Copy code
404 HTTP error fetching plugin from <https://api.github.com/repos/amitOrganizaton/pulumi-rise-packages/releases/tags/v0.0.24>
The cli can't access that repo
Probably because GITHUB_TOKEN isn't set, or is set to a token with the wrong permissions
p

prehistoric-sandwich-7272

02/06/2023, 7:29 PM
@echoing-dinner-19531 I’ve added it in multiple places inside my action \ makefile, still the same error... I see that it is run from the yarn install command
Copy code
"install": "node scripts/install-pulumi-plugin.js resource xyz "
and inside install-pulumi-plugin there is this part:
Copy code
var res = childProcess.spawnSync("pulumi", ["plugin", "install", "--server", "<github://api.github.com/amitOrganization[/pulumi-rise-packages]>"].concat(args), {
    stdio: ["ignore", "inherit", "inherit"]
});
.
.
} else if (res.error || res.status !== 0) {
    console.error("\nThere was an error installing the resource provider plugin. " +
            "You may try to manually installing the plugin by running " +
            "`pulumi plugin install " + args.join(" ") + "`");
}
which the error comes from maybe the url is still incorrect? should it be with the square brackets? (e.g
[/pulumi-rise-packages]
)
e

echoing-dinner-19531

02/06/2023, 7:38 PM
It shouldn't have the square brackets...
What's your pluginDownloadUrl in schema.yaml say exactly?
Cause I'm struggling to see where square brakets could get added by the generator code 😕
p

prehistoric-sandwich-7272

02/06/2023, 9:07 PM
@echoing-dinner-19531 I see - its just that the documentation is kinda confusing regarding the square brackets😅 my
pluginDownloadUrl
is:
Copy code
pluginDownloadURL: <github://api.github.com/amitOrganization/pulumi-rise-packages>
So I removed the square brackets from the nstall-pulumi-plugin and it still didn’t help, same error from the CICD ..
e

echoing-dinner-19531

02/06/2023, 9:07 PM
Ah that's meant to be that the repository name is optional!
p

prehistoric-sandwich-7272

02/06/2023, 9:08 PM
Interesting, I will try without!
e

echoing-dinner-19531

02/06/2023, 9:09 PM
If you just have "github://api.github.com/amitOrganization" the engine will look at your package name for the repo name and if your package is "rise-packages" then it will result in the same URL of "github://api.github.com/amitOrganization/pulumi-rise-packages"
p

prehistoric-sandwich-7272

02/08/2023, 9:03 PM
@echoing-dinner-19531 Sorry for the long response didn’t have time to keep working on it just re-ran everything, and stumbled upon a new error on the Build sdk step for nodejs 😞 So I started the repo again from scratch, changed the bare minimum and it looks like the error is happening again - which makes me think this has something to do with the pulumi converter \ codegen? two lines for copy-paste purposes:
Copy code
Error: node_modules/@types/node/ts4.8/util.d.ts(1485,42): error TS1005: ',' expected.
Error: node_modules/@types/node/ts4.8/util.d.ts(1485,44): error TS1068: Unexpected token. A constructor, method, accessor, or property was expected.
Full error:
e

echoing-dinner-19531

02/08/2023, 9:18 PM
Ah
Yes @types/node had a release like yesterday, for TS4 and pulumi is still tied to TS3 and the syntax has changed.
I had to go round and fix all our references of that from "latest" to "ts3.7" see https://github.com/pulumi/pulumi/pull/12091/files
p

prehistoric-sandwich-7272

02/08/2023, 9:43 PM
@echoing-dinner-19531 Thank you so much!! Fixed it
going back to the same old errors now 😅
e

echoing-dinner-19531

02/08/2023, 9:44 PM
Still around the pluginDownloadURL?
p

prehistoric-sandwich-7272

02/08/2023, 9:47 PM
@echoing-dinner-19531 yeah basically.. even though I am trying different things, still trying to handle this error from the build sdk stage:
Copy code
error: [resource plugin xyz-0.0.6] downloading from : failed to download plugin: xyz-0.0.6: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-xyz-v0.0.6-linux-amd64.tar.gz>
There was an error installing the resource provider plugin. You may try to manually installing the plugin by running `pulumi plugin install resource xyz v0.0.6`
e

echoing-dinner-19531

02/08/2023, 9:49 PM
Right, same as before but to summerize things to check: pluginDownloadURL in the schema that GITHUB_TOKEN is set to access the repo that there's a github release in the repo for 0.0.6 with the expected .tar.gz files in it
p

prehistoric-sandwich-7272

02/08/2023, 9:55 PM
So I added these two fields:
Copy code
publisher: amitOrganization
repository: my-repo
Which now gave me this error which is better, it’s looking at the right spot now:
Copy code
error: [resource plugin xyz-0.0.7] downloading from <github://api.github.com/amitOrganization>: failed to download plugin: xyz-0.0.7: 404 HTTP error fetching plugin from <https://api.github.com/repos/amitOrganization/pulumi-xyz/releases/tags/v0.0.7>. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: <https://github.com/settings/tokens>
Which makes me believe I need to pass GITHUB_TOKEN some place else (I am confident it has the right permissions) p.s -regarding the things you mentioned:
Copy code
pluginDownloadURL: <github://api.github.com/amitOrganization>
plus the release does exists.
ways I have tried setting up the GITHUB_TOKEN env: 1.
Copy code
env:
  GITHUB_TOKEN: ${{ secrets.access_token_github }}
run: |
  make build_${{ matrix.language }}_sdk
2.
Copy code
env:
  GITHUB_TOKEN: ${{ secrets.access_token_github }}
run: |
  export GITHUB_TOKEN=${{ env.GITHUB_TOKEN }}
  make build_${{ matrix.language }}_sdk
and also inside the makefile:
Copy code
GITHUB_TOKEN         := ${GITHUB_TOKEN}
e

echoing-dinner-19531

02/08/2023, 9:57 PM
p

prehistoric-sandwich-7272

02/08/2023, 9:58 PM
@echoing-dinner-19531 from my understanding it’s the name of your github secret , can be anything, for example I have an organization secret named
USERS_S3_FILE_PATH
which is totally custom and works great
but let me try setting up one named GITHUB_TOKEN - worth a shot
e

echoing-dinner-19531

02/08/2023, 9:59 PM
yeh I'm not a github actions expert, just the docs style is what I'm used to seeing so something to double check