https://pulumi.com logo
Powered by
p

prehistoric-account-60014

12/31/2019, 6:09 PM
Has anybody gotten an issue similar to 
could not get cloud url: unmarshalling credentials file: unexpected end of JSON input
? Specifically after calling 
gcloud auth activate-service-account --key-file "$KEY_FILE"
successfully?
I’m also getting this issue attempting to create k8s resources: 
error: could not deserialize deployment: constructing secrets manager of type "service": getting access token: unmarshalling credentials file: unexpected end of JSON input
w

white-balloon-205

12/31/2019, 9:44 PM
It sounds like one of the files in your 
~/.pulumi
folder got corrupted and is no longer valid JSON. Likely a 
credentials.json
?
Did you hand edit anything there?
p

prehistoric-account-60014

12/31/2019, 9:45 PM
I thought it was that at first so I ran 
pulumi login
before each command and then the second issue came up
I did not hand edit, it’s all just running in CI
Let me post the relevant code
Copy code
curl -fsSL <https://get.pulumi.com/> | bash
  export PATH=$PATH:$HOME/.pulumi/bin
  pulumi login
That is ran once in CI
Copy code
#!/bin/sh

set -e

if [ -z ${PULUMI_STACK+x} ]; then
  echo PULUMI_STACK is not set. Exiting...
  exit 1
fi

pulumi login
pulumi stack init "$PULUMI_STACK" || true

if [[ "$PULUMI_STACK" != "production" ]] || [[ "$PULUMI_STACK" != "staging" ]]; then
  pulumi config refresh --stack development || true
  json="$(pulumi config --stack development --show-secrets --json)"
  keys="$(echo $json | jq -r 'keys[]')"

  for key in $(echo $keys | tr '\n' ' ')
  do
    value="$(echo $json | jq -r '.["'"$key"'"].value')"
    is_secret="$(echo $json | jq -r '.["'"$key"'"].secret')"
    pulumi config --stack "$PULUMI_STACK" set "$key" "$value" $(if [ $is_secret = 'true' ]; then echo '--secret'; fi)
  done
fi
And this is run for each Pulumi project
The line 10 which has 
pulumi login
I added after seeing the first error but then I subsequently got the other one.
And if it helps, I’m running this right before I install Pulumi:
Copy code
curl <https://sdk.cloud.google.com> | bash > /dev/null
  export PATH=$PATH:/root/google-cloud-sdk/bin
  KEY_FILE="$(mktemp)"
  echo "$GCP_SERVICE_ACCOUNT" > "$KEY_FILE"
  gcloud auth activate-service-account --key-file "$KEY_FILE"
Also, some of the Pulumi projects are being updated without issue. So if the credentials are indeed corrupted, they don’t seem to be at first.
Any ideas on what could be happening here?
m

mammoth-elephant-55302

02/03/2020, 7:30 PM
It's because echo is only printing {
Maybe you need to do echo (cat )
Is this on a docker machine? How certain are you that gcloud auth is run successful if it isn't a completely new container or VM?
Are you running gcloud auth from within Google Cloud? (Like on a CE instance?)
p

prehistoric-account-60014

02/04/2020, 12:58 AM
It doesn’t seem to just be taking the first line of the JSON document. This is being run on a GitLab runner, so yes, it’s on docker
m

mammoth-elephant-55302

02/04/2020, 2:42 AM
I wonder if this https://stackoverflow.com/questions/26276348/echo-json-over-command-line-into-file Maybe gcloud auth was built with a more flexible parser
p

prehistoric-account-60014

02/04/2020, 2:43 AM
I just re-ran the CI job, it was about a month old at this point, and it seemed to pass. I’m rebasing some 100+ commits that have happened since then and making sure everything is good to go but it seems a fix in Pulumi might’ve removed this issue.
That stack overflow question seems extremely relevant, though. So I’ll keep working on this and report back.
I added 
cat "$KEY_FILE"
to see if the JSON was malformed and it seems fine. This time everything worked, though, so it’s possible it wasn’t before.
👍 1
Ok, so the issue is back. I thought that it could be because I was using 
mktemp
so instead I saved the keyfile in 
"$HOME/key.json"
but that didn’t help.
I’m re-adding the 
cat
in there to make sure the key file is not malformed…
w

white-balloon-205

02/04/2020, 8:53 PM
Which error exactly are you seeing? I believe there were two different errors mentioned in this thread.
p

prehistoric-account-60014

02/04/2020, 8:53 PM
Copy code
error: could not get cloud url: unmarshalling credentials file: unexpected end of JSON input
The issue seems to happens some times and not happen other times without changing anything between runs.
w

white-balloon-205

02/05/2020, 3:15 AM
Interesting. That error is definitely from Pulumi itself, and should be related to having a malformed 
~/.pulumi/credentials.json
. What OS are you on? Are you doing anything you think could possibly lead to corrupting the contents of that file?
p

prehistoric-account-60014

02/05/2020, 3:31 PM
I’m running on GitLab CI with the 
node:10-alpine
image. As for what I’m doing, I don’t think there’s anything that would result in a malformed credentials file. I basically do the two things I linked towards the beginning of the thread, install Pulumi once and then deploy a stack for each service.
I’m going to 
cat "$HOME/.pulumi/credentials.json
before running 
pulumi up
and see if the JSON is somehow malformed.
It seems that the JSON file is missing a closing bracket.
Copy code
{
    "current": "<https://api.pulumi.com>",
    "accessTokens": {
        "<https://api.pulumi.com>": "[MASKED]"
    },
    "accounts": {
        "<https://api.pulumi.com>": {
            "accessToken": "[MASKED]",
            "username": "miguel",
            "lastValidatedAt": "2020-02-05T17:15:45.14108518Z"
        }
    }
An interesting observation is that when I output the file contents, it seems that all lines except that last bracket is printed immediately, then the program continues and sometimes that final line is printed later. For the stacks were this happen, Pulumi shows the preview. For the ones were it doesn’t, the error happens.
I’m going to change the command to do 
cat "$HOME/.pulumi/credentials.json" | jq -c
to see if it’s all there or if there’s something weird with a file descriptor staying open causing the error
To be more specific, this is the output of a failure case:
Copy code
{
    "current": "<https://api.pulumi.com>",
    "accessTokens": {
        "<https://api.pulumi.com>": "[MASKED]"
    },
    "accounts": {
        "<https://api.pulumi.com>": {
            "accessToken": "[MASKED]",
            "username": "miguel",
            "lastValidatedAt": "2020-02-05T20:01:08.004514991Z"
        }
    }
error: stack 'review-mo-pulumi-pzf6a4' already exists
error: no previous deployment
error: could not get cloud url: unmarshalling credentials file: unexpected end of JSON input
}
That being stdout
When there’s a success case, the bracket will be in the proper place, albeit delayed, and there will be no error message.
What’s being printed is 
cat "$HOME/.pulumi/credentials.json"
just before calling 
pulumi up
Copy code
cat "$HOME/.pulumi/credentials.json"
pulumi stack init "$PULUMI_STACK" || true
pulumi config refresh --stack makeswift/development || true
json="$(pulumi config --stack makeswift/development --show-secrets --json)"
That’s the actual script running that’s causing the above result. The first two errors are ignored on purpose by the 
|| true
but the 
pulumi config
needs the credentials to show the secrets and that’s where it fails.
w

white-balloon-205

02/05/2020, 9:54 PM
Huh - that is very interesting and does suggest where the issue could be here. Any chance you could open a GitHub issue with these details? If not - I’ll pull one together myself a bit later today.
p

prehistoric-account-60014

02/05/2020, 11:12 PM
Yes, I’ll probably get to it tomorrow. If you do open one before that, please let me know. I’ll gladly add some comments with details when I get the chance.
Some more information. I traced the script to see what was running and it seems that the command that resulted in the error this time was a 
pulumi config
. Another observation is that there are multiple scripts running in parallel configuring different stacks and reading the credentials file. I don’t know if Pulumi modifies the file while this is happening, but if it does it might mean that running Pulumi commands concurrently isn’t safe as one command might attempt to read the file while another one is modifying it. Does this ring any bells?
w

white-balloon-205

02/06/2020, 5:14 AM
That is very plausible explanation. We haven’t seen this before - but it is possible that it is quite rare to run pulumi CLI commands concurrently so that this is very uncommon in practice. We will look into it.
p

prehistoric-account-60014

02/06/2020, 2:29 PM
Gotcha. Well, I’ve got some time on my hands now so I’ll go ahead and open the issue.
I think the issue is concurrent Pulumi commands. I’ve changed the program so that it deploys each project in series instead of in parallel and the issue has gone away.
🍧 1
👍 1
9 Views
Powered by