Hey folks, hitting a weird one with a stack that was working fine and seemingly randomly broke (I was messing around with

pulumi logs -f -s

simultaneously maybe?)

FEATURE=staging PROJECT_NAME=qa npm run create-db-seed destroy

> qa@ create-db-seed /Users/daniel/code/RedJade/infra/scripts
> tsc && node ./bin/scripts/create-db-seed.js "destroy"

project name qa
feature name staging
(node:74675) ExperimentalWarning: Conditional exports is an experimental feature. This feature could change at any time
CommandError: code: 255
 stdout: 
 stderr: error: invalid character 'y' after top-level value

 err?: 

    at Object.createCommandError (/Users/daniel/code/RedJade/infra/scripts/node_modules/@pulumi/pulumi/automation/errors.js:71:17)
    at ChildProcess.<anonymous> (/Users/daniel/code/RedJade/infra/scripts/node_modules/@pulumi/pulumi/automation/cmd.js:63:40)
    at ChildProcess.emit (events.js:321:20)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12) {
  commandResult: CommandResult {
    stdout: '',
    stderr: "error: invalid character 'y' after top-level value\n",
    code: 255,
    err: undefined
  },
  name: 'CommandError'
}

Same thing happened to me yesterday but I was able to get around it by moving my automation code out of a folder shared with another stack, so was able to hand wave it away -- but then it popped up again today. It was working, working, then not working. Has anyone seen this before?

Looks like an error message, probably beginning with "y", is being returned at some point when a valid response (maybe json-like) is expected. No way to tell where the error is from this.

yeah I went down that route yesterday and like I said, just moving my code resolved the issue. I am fairly certain i didn't change anything to cause an error

@tall-scientist-89115 the error appears to come from the golang JSON encoder: https://go.googlesource.com/go/+/master/src/encoding/json/decode_test.go#460 what's in your stack outputs?

interestingly enough, changing the working directory to any directory other than the two directories which have encountered the problem before and it goes away edit: got a 409 "another update in progress" in that case

Hey what versions of pulumi SDK and CLI are you using?

cli is v3.1.0

A couple of things. I’m curious what your automation api script is doing. Are there many stacks? Are you selecting them concurrently? I think you mentioned that you only want to get the stack outputs, is that right? The next release will expose a

LocalWorkspace.stackOutputs(stackName: string)

method which will enable you to get the stack outputs without selecting each stack. I still would like to get down to the bottom of whats going on here though.

I am selecting them concurrently but once the problem begins it will also happen for just 1. That method looks perfect!

I am selecting them concurrently

I think that might be the issue.

selectStack

modifies global state and might be causing some unexpected behavior.

for sure that makes sense to me

copy/paste describing would be great! I’m about to head off to bed so being able to take a closer look later would be really helpful

So here's the folder structure, the scripts folder having its own dependency chain separate from the infra folder's is largely unnecessary and was done in response to the problem yesterday

Yeah just from a quick glance I’m guessing things are going wrong here:

return await Promise.all([appSvcsStack, clusterStack, containerStack]).then(
    ([appSvcsStack, clusterStack, containerStack]) => {
      elapsed_time("fetching stacks");
      return Promise.all([appSvcsStack.outputs(), clusterStack.outputs(), containerStack.outputs()]).then(
        ([appSvcsOuts, clusterOuts, containerOuts]) => {

I think if you did the

selectStack

step serially:

const appSvcsStack = await LocalWorkspace.selectStack({
    stackName: `RedJade/qa/app-services`,
    workDir: "..",
});
const clusterStack = await LocalWorkspace.selectStack({
    stackName: `RedJade/${projectName}/cluster-configuration`,
    workDir: "..",
});
const containerStack = await LocalWorkspace.selectStack({
    stackName: `RedJade/global/container-registry`,
    workDir: "..",
});

return Promise.all([appSvcsStack.outputs(), clusterStack.outputs(), containerStack.outputs()]).then(
    ([appSvcsOuts, clusterOuts, containerOuts]) => {
        elapsed_time("fetching stack outputs");
        return { ... }

and then the rest of your workflow I think you would be okay.

haha yeah, the problem is the die is cast

haha yeah, the problem is the die is cast

Ah interesting, once it gets borked theres no recovering it huh?

yeah isolating down to one and doing as you said it is done

y’know… this sounds very similar to https://pulumi-community.slack.com/archives/C019YSXN04B/p1619045148151000

nice find

In any case, I think we need to take a look at the

selectStack

functionality, especially for folks who might want to be doing tasks like getting outputs or destroying multiple stacks who don’t need the global state modifying behavior of

selectStack

.

seems good

pulumi stack ls
NAME                           LAST UPDATE     RESOURCE COUNT  URL
RedJade/app-services           7 hours ago     67              <https://app.pulumi.com/RedJade/qa/app-services>
RedJade/cluster-configuration  1 week ago      29              <https://app.pulumi.com/RedJade/qa/cluster-configuration>
RedJade/cluster-services       6 days ago      58              <https://app.pulumi.com/RedJade/qa/cluster-services>
RedJade/identity               1 week ago      16              <https://app.pulumi.com/RedJade/qa/identity>
RedJade/managed-infra          1 week ago      7               <https://app.pulumi.com/RedJade/qa/managed-infra>
RedJade/staging-apps           1 day ago       0               <https://app.pulumi.com/RedJade/qa/staging-apps>
RedJade/staging-database       52 minutes ago  0               <https://app.pulumi.com/RedJade/qa/staging-database>
staging-database               5 hours ago     0               <https://app.pulumi.com/mypersonalaccount/qa/staging-database>

Okay yeah let’s check in tomorrow 🙏🏽

hey np, thanks for the great product/all your help up to this point