This keeps happening even after i deleted and re-created the stack

runnig with --logtostder gives "Operation not permitted" message:

I0226 09:29:22.671328   19436 log.go:56] Error closing 'nodejs' language plugin during shutdown; ignoring: 1 error occurred:

* operation not permitted
 
Diagnostics:
  pulumi:pulumi:Stack (devops-eks-identity-test1):
    The Pulumi runtime detected that 16 promises were still active
    at the time that the process exited. There are a few ways that this can occur:
      * Not using `await` or `.then` on a Promise returned from a Pulumi API
      * Introducing a cyclic dependency between two Pulumi Resources
      * A bug in the Pulumi Runtime
    Leaving promises active is probably not what you want. If you are unsure about
    why you are seeing this message, re-run your program with the `PULUMI_DEBUG_PROMISE_LEAKS`
    environment variable. The Pulumi runtime will then print out additional
    debug information about the leaked promises.
 
    error: an unhandled error occurred: Program exited with non-zero exit code: 1
 
error: an error occurred while advancing the preview
<http://github.com/pulumi/pulumi/pkg/engine.printPlan|github.com/pulumi/pulumi/pkg/engine.printPlan>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/engine/plan.go:251
<http://github.com/pulumi/pulumi/pkg/engine.update|github.com/pulumi/pulumi/pkg/engine.update>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/engine/update.go:155
<http://github.com/pulumi/pulumi/pkg/engine.Update|github.com/pulumi/pulumi/pkg/engine.Update>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/engine/update.go:88
<http://github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).runEngineAction|github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).runEngineAction>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/httpstate/backend.go:826
<http://github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).apply|github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).apply>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/httpstate/backend.go:766
<http://github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).apply-fm|github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).apply-fm>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/httpstate/backend.go:662
<http://github.com/pulumi/pulumi/pkg/backend.PreviewThenPrompt|github.com/pulumi/pulumi/pkg/backend.PreviewThenPrompt>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/apply.go:112
<http://github.com/pulumi/pulumi/pkg/backend.PreviewThenPromptThenExecute|github.com/pulumi/pulumi/pkg/backend.PreviewThenPromptThenExecute>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/apply.go:194
<http://github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).Update|github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudBackend).Update>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/httpstate/backend.go:662
<http://github.com/pulumi/pulumi/pkg/backend.UpdateStack|github.com/pulumi/pulumi/pkg/backend.UpdateStack>
        /Users/travis/gopath/src/github.com/pulumi/pulumi/pkg/backend/stack.go:73
<http://github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudStack).Update|github.com/pulumi/pulumi/pkg/backend/httpstate.(*cloudStack).Update>

The program is based on "kubernets the prod way" example:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as util from "./util";
import * as iam from "./lib";

const baseline = new iam.BaselineIam("baselineIam", {
    groups: {
        // Create default EKS admins group.
        defineEksAdminsGroup: true,
        defineRoute53AdminsGroup: true
    },
});

//
// EKS management user. Deploys EKS, passes AWS IAM Role ARNs to EKS, so that workloads can be
// correlated to AWS IAM.
//

const eksAdminCiUser = new util.BotUser("eksAdminCiUser", {
    groupMembership: {
        groups: [
            baseline.groups.eksAdmins!.name,
            baseline.groups.useExistingIamRoles!.name, // To use pass role ARNs to k8s RoleBindings.
        ],
    },
});
const eksAdminCiUserKey = eksAdminCiUser.createAccessKey("eksAdminCiUser");

export const eksUserCiUserAccessKey = {
    id: eksAdminCiUserKey.id,
    secret: eksAdminCiUserKey.secret,
};

const route53User = new util.BotUser("router53User", {
    groupMembership: {
        groups: [
            baseline.groups.route53Admins!.name
        ]
    },
});

const route53UserKey = route53User.createAccessKey("route53User");

export const route53UserAccessKey = {
    id: route53UserKey.id,
    secret: route53UserKey.secret,
};

const kubeAppRole = util.newRoleWithPolicies(
    "kubeAppRole",
    {
        description: "Infrastructure management role for CI users",
        assumeRolePolicy: eksAdminCiUser.user.arn.apply(util.assumeRolePolicy),
    },
    {
        ecrPowerUser: aws.iam.AmazonEC2ContainerRegistryPowerUser,
        passRole: baseline.policies.useExistingIamRoles!.arn,
    },
);


export const kubeAppRoleArn = kubeAppRole.arn;

Verbose log output: https://gist.github.com/adaniline-traderev/cd6269c62898180e803343b644c87135 Not clear exactly what is failing.

I just wish Pulumi gave more descriptive error messages

How do I upgrade it? I was running

curl -fsSL <https://get.pulumi.com/> | sh

, but still have 0.16.14

curl --fail --silent -L "<https://pulumi.io/latest-version>

gives 0.16.14

Re-ran it with 0.16.16, and updated log output in the gist above - I've got the same result

By commenting everything, and then uncommenting it line by line I've pin pointed the error to the creation of one out of two

aws.iam.RolePolicyAttachment

resources. Not sure why one is created, and the other one gives an error - does not look like this is an IAM permission

Also, why is the error in preview stage?

Give me a few mins to finish up some calls and I can look at this more closely

Certainly there is a bug though - we never expect a Go backtrace to be exposed to the user.