https://pulumi.com logo
Powered by
Title
w

worried-city-86458

06/04/2021, 6:13 AM
Something went seriously wrong with my aws eks stack, to the point that I couldn't work out how to repair it, so I eventually destroyed then removed it completely. Only problem now is when I try to create the stack again from scratch I still get the same error! 😕 😖
user@fd330487a4e1:~/gemini$ ./gemini deploy alpha

[05:54:34 INF] Deploying resources...
[05:54:34 INF] Deploying pharos/aws-eks/alpha...
[05:54:36 DBG] Installing plugins...
[05:54:43 DBG] Setting config...
[05:54:44 DBG] Previewing stack...
Previewing update (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/previews/cd4e4131-edb6-4431-8986-60d6a2f8ed03>
...

Resources:
    + 29 to create
Update stack? [y/n] (n): y
[05:54:52 DBG] Updating stack...
Updating (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/1>

[05:54:54 INF] Deployed pharos/aws-eks/alpha in 00:00:19
[05:54:54 INF] Deployed resources in 00:00:19
Pulumi.Automation.Commands.Exceptions.CommandException: code: 2
stdout: Updating (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/1>


stderr: panic: fatal: An assertion has failed: cannot complete a resource 'urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha' whose registration isn't pending

goroutine 101 [running]:
<http://github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)|github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)>
        /Users/runner/work/pulumi/pulumi/sdk/go/common/util/contract/failfast.go:23
<http://github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf|github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf>(0xc000934800, 0x1d9144c, 0x40, 0xc000979948, 0x1, 0x1)
        /Users/runner/work/pulumi/pulumi/sdk/go/common/util/contract/assert.go:33 +0x1a5
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).ExecuteRegisterResourceOutputs(0xc000e2ec00|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).ExecuteRegisterResourceOutputs(0xc000e2ec00>, 0x7f50af8f8098, 0xc000946b00)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/step_executor.go:154 +0x151
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).handleSingleEvent(0xc000944378|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).handleSingleEvent(0xc000944378>, 0x208e460, 0xc000946b00, 0x0, 0x2)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment_executor.go:378 +0x318
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute.func3(0xc000e10240|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute.func3(0xc000e10240>, 0xc000944378, 0xc000932940, 0x20bc730, 0xc0015584c0, 0x0, 0x0, 0x20bc7d8, 0xc00092f470, 0x0, ...)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment_executor.go:242 +0x20c
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute(0xc000944378|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute(0xc000944378>, 0x20bc7d8, 0xc00092f470, 0x7f50af8f7dc0, 0xc0005fd080, 0x7fffffff, 0x0, 0x2d04608, 0x0, 0x0, ...)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment_executor.go:258 +0x7e5
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*Deployment).Execute(...)|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*Deployment).Execute(...)>
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment.go:405
<http://github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run.func1|github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run.func1>(0x20c6bf0, 0xc0005fd080, 0xc00093e280, 0x20bc7d8, 0xc00092f470, 0xc00092f400, 0xc0009328d0, 0xc0008ddc20)
        /Users/runner/work/pulumi/pulumi/pkg/engine/deployment.go:252 +0x287
created by <http://github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run|github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run>
        /Users/runner/work/pulumi/pulumi/pkg/engine/deployment.go:237 +0x308


   at Pulumi.Automation.Commands.LocalPulumiCmd.RunAsyncInner(IEnumerable`1 args, String workingDir, IDictionary`2 additionalEnv, Action`1 onStandardOutput, Action`1 onStandardError, EventLogFile eventLogFile, CancellationToken
cancellationToken)
   at Pulumi.Automation.Commands.LocalPulumiCmd.RunAsync(IEnumerable`1 args, String workingDir, IDictionary`2 additionalEnv, Action`1 onStandardOutput, Action`1 onStandardError, Action`1 onEngineEvent, CancellationToken
cancellationToken)
   at Pulumi.Automation.Workspace.RunStackCommandAsync(String stackName, IEnumerable`1 args, Action`1 onStandardOutput, Action`1 onStandardError, Action`1 onEngineEvent, CancellationToken cancellationToken)
   at Pulumi.Automation.WorkspaceStack.RunCommandAsync(IEnumerable`1 args, Action`1 onStandardOutput, Action`1 onStandardError, Action`1 onEngineEvent, CancellationToken cancellationToken)
   at Pulumi.Automation.WorkspaceStack.UpAsync(UpOptions options, CancellationToken cancellationToken)
   at Pulumi.Automation.WorkspaceStack.UpAsync(UpOptions options, CancellationToken cancellationToken)
   at Pharos.Gemini.DeployCommand.OnExecuteAsync(CommandContext context, Settings settings)
   at Pharos.Gemini.AsyncCommandBase`1.ExecuteAsync(CommandContext context, TSettings settings)user@fd330487a4e1:~/gemini$

user@fd330487a4e1:~/gemini$ pulumi cancel --stack pharos/aws-eks/alpha
This will irreversibly cancel the currently running update for 'alpha'!
Please confirm that this is what you'd like to do by typing ("alpha"): alpha
The currently running update for 'alpha' has been canceled!
Totally stuck now. Any help appreciated.
l

lemon-agent-27707

06/04/2021, 2:54 PM
Are you seeing this when you run the same program just via the CLI? Is this just a single stack, or are you orchestrating multiple deployments? Would be helpful to see code, versions of things you are using, etc.
w

worried-city-86458

06/04/2021, 6:46 PM
@lemon-agent-27707 The main thing I changed recently is how I orchestrate all the steps with the dotnet automation api. I used to run either preview or update. Now I default to always run preview (unless cli flag passed to skip preview), followed by update iff approved. I don't suppose this is problematic somehow. On a retry I did notice that the preview first appeared in the pulumi ui only to disappear when the update started. The update immediately fails on the client side as above but the stack seems to hang in the pulumi ui and has to be cancelled.
protected override async Task<int> OnExecuteAsync(CommandContext context, Settings settings)
{
    Logger.LogInformation("Deploying resources...");
    using var totalTimeLogger = new ElapsedTimeLogger(Logger, "Deployed resources");

    var deployedResources = OrderedResources.Where(resource => settings.Resources.HasFlag(resource));
    foreach (var resource in deployedResources)
    {
        var info = ResourceInfo[resource];
        var stackFullName = $"{Config.Pulumi.Organization.Name}/{info.ProjectName}/{settings.Environment.ToLower()}";
        Logger.LogInformation($"Deploying {stackFullName}...");
        using var resourceTimeLogger = new ElapsedTimeLogger(Logger, $"Deployed {stackFullName}");

        var stackName = $"{Config.Pulumi.Organization.Name}/{settings.Environment.ToLower()}";
        var stackArgs = new InlineProgramArgs(info.ProjectName, stackName, PulumiFn.Create(ServiceProvider, info.StackType));
        var stack = await LocalWorkspace.CreateOrSelectStackAsync(stackArgs);

        Logger.LogDebug("Installing plugins...");
        foreach (var plugin in RequiredPlugins)
        {
            var pluginArgs = plugin.Split(" ");
            await stack.Workspace.InstallPluginAsync(pluginArgs[0], pluginArgs[1]);
        }

        Logger.LogDebug("Setting config...");
        var config = Config.Environment.ToTokens("Gemini:Environment")
            .ToDictionary(entry => entry.Key, entry => new ConfigValue(entry.Value.ToValueString()));
        await stack.SetAllConfigAsync(config);

        if (settings.Destroy)
        {
            if (!settings.NonInteractive &&
                AnsiConsole.Confirm("[red]Destroy stack?[/]", false) &&
                AnsiConsole.Prompt(new TextPrompt<string>($@"[red]Confirm destroy stack[/] [blue]""{stackFullName}""[/]").AllowEmpty()) == stackFullName)
            {
                Logger.LogDebug("Destroying stack...");
                var result = await stack.DestroyAsync(new DestroyOptions { OnStandardOutput = AnsiConsole.WriteLine });
                Logger.LogDebug($"Destroyed stack ({result.Summary.Result})");
            }
            else
            {
                Logger.LogWarning($"Destroy stack skipped ({(settings.NonInteractive ? "non-interactive; " : "")}unapproved)");
            }
            continue;
        }

        if (settings.Repair)
        {
            if (!settings.NonInteractive)
            {
                Logger.LogDebug("Repairing stack...");
                var exportState = await stack.ExportStackAsync();
                var json = await RepairStackAsync(exportState.Json.GetRawText());
                var importState = StackDeployment.FromJsonString(json);
                var jsonPath = JsonPath.Parse("$.deployment.pending_operations[*].resource.urn").Evaluate(importState.Json);
                if (importState.Json.IsEquivalentTo(exportState.Json))
                {
                    Logger.LogWarning("Repaired stack ignored (equivalent)");
                }
                else if (jsonPath.Error != null)
                {
                    Logger.LogWarning($"Repaired stack ignored (error): {jsonPath.Error}");
                }
                else if (jsonPath.Matches is { Count: > 0 })
                {
                    Logger.LogWarning("Repaired stack ignored (pending resources):");
                    foreach (var match in jsonPath.Matches)
                    {
                        Logger.LogWarning(match.Value.ToString());
                    }
                }
                else
                {
                    await stack.ImportStackAsync(importState);
                    Logger.LogDebug("Repaired stack");
                }
            }
            else
            {
                Logger.LogWarning("Repair stack skipped (non-interactive)");
            }
            continue;
        }

        if (settings.Refresh)
        {
            Logger.LogDebug("Refreshing stack...");
            await stack.RefreshAsync(new RefreshOptions { OnStandardOutput = AnsiConsole.WriteLine });
        }

        if (!settings.SkipPreview)
        {
            Logger.LogDebug("Previewing stack...");
            var result = await stack.PreviewAsync(
                new PreviewOptions
                {
                    Diff = !settings.NoDiff,
                    ExpectNoChanges = settings.ExpectNoChanges,
                    OnStandardOutput = AnsiConsole.WriteLine
                });
            if (result.ChangeSummary.All(entry => entry.Key == OperationType.Same))
            {
                Logger.LogDebug("Update stack skipped (unchanged)");
                continue;
            }
        }
        else
        {
            Logger.LogDebug("Preview stack skipped");
        }

        if (settings.Approve || !settings.NonInteractive && AnsiConsole.Confirm("[yellow]Update stack?[/]", false))
        {
            Logger.LogDebug("Updating stack...");
            var result = await stack.UpAsync(
                new UpOptions
                {
                    Diff = !settings.NoDiff,
                    ExpectNoChanges = settings.ExpectNoChanges,
                    OnStandardOutput = AnsiConsole.WriteLine
                });
            Logger.LogDebug($"Updated stack ({result.Summary.Result})");
            if (result.Summary.Result != UpdateState.Succeeded)
            {
                return -1;
            }
        }
        else
        {
            Logger.LogDebug($"Update stack skipped ({(settings.NonInteractive ? "non-interactive; " : "")}unapproved)");
        }
    }
    return 0;
}
So I stripped my stack back to nothing but creating an aws provider: 
public sealed class EksStack : StackBase<EksStack>
{
    public EksStack(IOptions<Config> options) : base(options)
    {
        var config = Config.Environment.Aws;
        var provider = Config.Environment.CreateProvider();
        var prefix = $"{Config.Environment.Name}-{Resources.AwsEks.ToName()}";
    }
}

public static class ConfigExtensions
{
    public static Provider CreateProvider(this EnvironmentConfig config) =>
        new($"{config.Name}-aws",
            new ProviderArgs
            {
                AssumeRole = new ProviderAssumeRoleArgs { RoleArn = config.Aws.AssumeRoleArn },
                DefaultTags = new ProviderDefaultTagsArgs { Tags = config.DefaultTags() },
                Region = config.Aws.Region
            });
}
Then if I run update with preview first it fails as above: 
user@d1360ec0d0a9:~/gemini$ ./gemini deploy alpha awseks

[19:22:49 INF] Deploying resources...
[19:22:49 INF] Deploying pharos/aws-eks/alpha...
[19:22:51 DBG] Installing plugins...
[19:22:51 DBG] Setting config...
[19:22:52 DBG] Previewing stack...
Previewing update (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/previews/cad1dc6f-83f4-4c04-b406-d744cdf5cfeb>

+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha]
    + pulumi:providers:aws: (create)
        [urn=urn:pulumi:alpha::aws-eks::pulumi:providers:aws::alpha-aws]
        assumeRole               : "{ \"roleArn\": \"[REDACTED]\" }"
        defaultTags              : "{ \"tags\": { \"Environment\": \"Alpha\", \"EnvironmentType\": \"Development\" } }"
        region                   : "us-west-2"
        skipCredentialsValidation: "true"
        skipGetEc2Platforms      : "true"
        skipMetadataApiCheck     : "true"
        skipRegionValidation     : "true"
        version                  : "4.6.0"
Resources:
    + 2 to create
Update stack? [y/n] (n): y
[19:22:59 DBG] Updating stack...
Updating (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/4>

[19:23:01 INF] Deployed pharos/aws-eks/alpha in 00:00:11
[19:23:01 INF] Deployed resources in 00:00:11
Pulumi.Automation.Commands.Exceptions.CommandException: code: 2
stdout: Updating (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/4>


stderr: panic: fatal: An assertion has failed: cannot complete a resource 'urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha' whose registration isn't pending

goroutine 83 [running]:
<http://github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)|github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)>
        /Users/runner/work/pulumi/pulumi/sdk/go/common/util/contract/failfast.go:23
<http://github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf|github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assertf>(0xc000d72300, 0x1d9144c, 0x40, 0xc001311948, 0x1, 0x1)
        /Users/runner/work/pulumi/pulumi/sdk/go/common/util/contract/assert.go:33 +0x1a5
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).ExecuteRegisterResourceOutputs(0xc000cbe100|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*stepExecutor).ExecuteRegisterResourceOutputs(0xc000cbe100>, 0x7fd8a89e0750, 0xc000594940)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/step_executor.go:154 +0x151
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).handleSingleEvent(0xc000d5e018|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).handleSingleEvent(0xc000d5e018>, 0x208e460, 0xc000594940, 0x0, 0x2)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment_executor.go:378 +0x318
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute.func3(0xc00009f560|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute.func3(0xc00009f560>, 0xc000d5e018, 0xc000d540b0, 0x20bc730, 0xc000d60180, 0x0, 0x0, 0x20bc7d8, 0xc000c29290, 0x0, ...)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment_executor.go:242 +0x20c
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute(0xc000d5e018|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*deploymentExecutor).Execute(0xc000d5e018>, 0x20bc7d8, 0xc000c29290, 0x7fd8a89e0370, 0xc0000dd340, 0x7fffffff, 0x0, 0x2d04608, 0x0, 0x0, ...)
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment_executor.go:258 +0x7e5
<http://github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*Deployment).Execute(...)|github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*Deployment).Execute(...)>
        /Users/runner/work/pulumi/pulumi/pkg/resource/deploy/deployment.go:405
<http://github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run.func1|github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run.func1>(0x20c6bf0, 0xc0000dd340, 0xc000c4c3c0, 0x20bc7d8, 0xc000c29290, 0xc000c29200, 0xc000c2c900, 0xc00160af00)
        /Users/runner/work/pulumi/pulumi/pkg/engine/deployment.go:252 +0x287
created by <http://github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run|github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run>
        /Users/runner/work/pulumi/pulumi/pkg/engine/deployment.go:237 +0x308


   at Pulumi.Automation.Commands.LocalPulumiCmd.RunAsyncInner(IEnumerable`1 args, String workingDir, IDictionary`2 additionalEnv, Action`1 onStandardOutput, Action`1 onStandardError, EventLogFile eventLogFile, CancellationToken
cancellationToken)
   at Pulumi.Automation.Commands.LocalPulumiCmd.RunAsync(IEnumerable`1 args, String workingDir, IDictionary`2 additionalEnv, Action`1 onStandardOutput, Action`1 onStandardError, Action`1 onEngineEvent, CancellationToken
cancellationToken)
   at Pulumi.Automation.Workspace.RunStackCommandAsync(String stackName, IEnumerable`1 args, Action`1 onStandardOutput, Action`1 onStandardError, Action`1 onEngineEvent, CancellationToken cancellationToken)
   at Pulumi.Automation.WorkspaceStack.RunCommandAsync(IEnumerable`1 args, Action`1 onStandardOutput, Action`1 onStandardError, Action`1 onEngineEvent, CancellationToken cancellationToken)
   at Pulumi.Automation.WorkspaceStack.UpAsync(UpOptions options, CancellationToken cancellationToken)
   at Pulumi.Automation.WorkspaceStack.UpAsync(UpOptions options, CancellationToken cancellationToken)
   at Pharos.Gemini.DeployCommand.OnExecuteAsync(CommandContext context, Settings settings)
   at Pharos.Gemini.AsyncCommandBase`1.ExecuteAsync(CommandContext context, TSettings settings)
After cancelling the stack, if I run again but skip preview it works as expected: 
user@d1360ec0d0a9:~/gemini$ ./gemini deploy alpha awseks --skip-preview

[19:25:56 INF] Deploying resources...
[19:25:56 INF] Deploying pharos/aws-eks/alpha...
[19:25:58 DBG] Installing plugins...
[19:25:58 DBG] Setting config...
[19:25:59 DBG] Preview stack skipped
Update stack? [y/n] (n): y
[19:26:01 DBG] Updating stack...
Updating (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/5>

+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha]
    + pulumi:providers:aws: (create)
        [urn=urn:pulumi:alpha::aws-eks::pulumi:providers:aws::alpha-aws]
        assumeRole               : "{ \"roleArn\": \"arn:aws:iam::854999447207:role/deployer\" }"
        defaultTags              : "{ \"tags\": { \"CostCenter\": \"Beacon\", \"Environment\": \"Alpha\", \"EnvironmentType\": \"Development\" } }"
        region                   : "us-west-2"
        skipCredentialsValidation: "true"
        skipGetEc2Platforms      : "true"
        skipMetadataApiCheck     : "true"
        skipRegionValidation     : "true"
        version                  : "4.6.0"
Resources:
    + 2 created

Duration: 2s
[19:26:07 DBG] Updated stack (Succeeded)
[19:26:07 INF] Deployed pharos/aws-eks/alpha in 00:00:10
[19:26:07 INF] Deployed resources in 00:00:10
@bored-oyster-3147 is this a bug with the dotnet automation api or am I missing something? Is it something fundamental like it's not valid to run preview before update?
b

bored-oyster-3147

06/04/2021, 7:34 PM
@lemon-agent-27707 will be more helpful here because 
An assertion has failed: cannot complete a resource 'urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha' whose registration isn't pending
is indicative of an error on the CLI/engine side and not an SDK issue
l

lemon-agent-27707

06/04/2021, 7:47 PM
@worried-city-86458 I am not a dotnet expert, so will need some time to look through the code you posted. One question: Is it possible that there is global state anywhere in your automation script? Are you accidentally saving/caching a reference to a provider or resource during a preview, and then trying to reference that in an update? This is a failure mode we've seen manifest elsewhere in similar ways.
Perhaps this chunk? 
public static class ConfigExtensions
{
    public static Provider CreateProvider
One thing to know, is that resources cannot be cached and shared across program runs. Every resource in a program needs to have an SDK call 
new aws.s3.Bucket(..)
so that it gets added to the dependency graph for the new state. If a resource was somehow cached from a preview run, and then referenced during an update then it would not appear in the dependency graph and other resources that reference the cached/undelcared resource in a new update run would then fail.
b

bored-oyster-3147

06/04/2021, 8:17 PM
That's a good point. The 
CreateProvider
you pointed out is a function so that is actually newing it up when invoked so I wouldn't expect that to be the culprit. I also don't see anything else in the snippets that is indicative of any instances being preserved across runs but idk if I'm not seeing the whole picture. The error that we are seeing appears to be with the root Stack resource. Have you tried maybe completely deleting the stack and re-creating it, as opposed to just destroying everything? I wonder if there isn't something corrupt about that root resource for some reason. Might be helpful to get a look at the state JSON for your small provider-only example and see what that looks like
w

worried-city-86458

06/04/2021, 8:21 PM
@lemon-agent-27707 so 
CreateProvider
is an extension method in c#, declared as static, but it's a factory that always returns a new instance.
Only the "options" injected and used as config are effectively static across preview / up since they have "root lifetime scope" in the DI container; and they're only used as simple data types to feed into the resource creation parameters.
Ah but my stack is registered with the DI container as a singleton... that must be it!
public static IHostBuilder ConfigureServices(this IHostBuilder hostBuilder) =>
    hostBuilder.ConfigureServices((context, services) =>
    {
        services.Configure<Config>(context.Configuration.GetSection(Constants.ConfigKey));

        services.AddSingleton<EksStack>()
            .AddSingleton<K8sStack>();
    });
I'll change that to transient and give it a go...
b

bored-oyster-3147

06/04/2021, 8:29 PM
oh 100% that is the issue
w

worried-city-86458

06/04/2021, 8:34 PM
PS. there's also an issue that I can't destroy from pulumi cli with full stack name: 
user@d1360ec0d0a9:~/gemini$ pulumi destroy --stack pharos/aws-eks/alpha
error: no Pulumi.yaml project file found (searching upwards from /home/user/gemini). If you have not created a project yet, use `pulumi new` to do so
(
pulumi stack rm --stack pharos/aws-eks/alpha
works)
Hence I added destroy to my cli that uses the automation api
[08:35:51 INF] (DeployCommand) Deploying resources...
[08:35:51 INF] (DeployCommand) Deploying pharos/aws-eks/alpha...
[08:35:54 DBG] (DeployCommand) Installing plugins...
[08:35:55 DBG] (DeployCommand) Setting config...
Destroy stack? [y/n] (n): y
Confirm destroy stack "pharos/aws-eks/alpha": pharos/aws-eks/alpha
[08:36:06 DBG] (DeployCommand) Destroying stack...
Destroying (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/6>


 -  pulumi:providers:aws alpha-aws deleting
 -  pulumi:providers:aws alpha-aws deleted
 -  pulumi:pulumi:Stack aws-eks-alpha deleting
 -  pulumi:pulumi:Stack aws-eks-alpha deleted

Resources:
    - 2 deleted

Duration: 1s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still
maintained.
If you want to remove the stack completely, run 'pulumi stack rm pharos/alpha'.
[08:36:11 DBG] (DeployCommand) Destroyed stack (Succeeded)
[08:36:11 INF] (DeployCommand) Deployed pharos/aws-eks/alpha in 00:00:19
[08:36:11 INF] (DeployCommand) Deployed resources in 00:00:19
@lemon-agent-27707 @bored-oyster-3147 changing my stack registration to transient worked: 
user@a993bbcf7d96:~/gemini$ ./gemini deploy alpha awseks
[20:40:51 INF] Deploying resources...
[20:40:51 INF] Deploying pharos/aws-eks/alpha...
[20:40:52 DBG] Installing plugins...
[20:40:58 DBG] Setting config...
[20:40:59 DBG] Previewing stack...
Previewing update (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/previews/cce14912-a05a-435e-bf25-be55af91373e>

+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha]
    + pulumi:providers:aws: (create)
        [urn=urn:pulumi:alpha::aws-eks::pulumi:providers:aws::alpha-aws]
        assumeRole               : "{ \"roleArn\": \"arn:aws:iam::854999447207:role/deployer\" }"
        defaultTags              : "{ \"tags\": { \"CostCenter\": \"Beacon\", \"Environment\": \"Alpha\", \"EnvironmentType\": \"Development\" } }"
        region                   : "us-west-2"
        skipCredentialsValidation: "true"
        skipGetEc2Platforms      : "true"
        skipMetadataApiCheck     : "true"
        skipRegionValidation     : "true"
        version                  : "4.6.0"
Resources:
    + 2 to create
Update stack? [y/n] (n): y
[20:41:06 DBG] Updating stack...
Updating (pharos/alpha)

View Live: <https://app.pulumi.com/pharos/aws-eks/alpha/updates/1>

+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:alpha::aws-eks::pulumi:pulumi:Stack::aws-eks-alpha]
    + pulumi:providers:aws: (create)
        [urn=urn:pulumi:alpha::aws-eks::pulumi:providers:aws::alpha-aws]
        assumeRole               : "{ \"roleArn\": \"arn:aws:iam::854999447207:role/deployer\" }"
        defaultTags              : "{ \"tags\": { \"CostCenter\": \"Beacon\", \"Environment\": \"Alpha\", \"EnvironmentType\": \"Development\" } }"
        region                   : "us-west-2"
        skipCredentialsValidation: "true"
        skipGetEc2Platforms      : "true"
        skipMetadataApiCheck     : "true"
        skipRegionValidation     : "true"
        version                  : "4.6.0"
Resources:
    + 2 created

Duration: 2s
[20:41:12 DBG] Updated stack (Succeeded)
[20:41:12 INF] Deployed pharos/aws-eks/alpha in 00:00:21
[20:41:12 INF] Deployed resources in 00:00:21
Thanks for your help! 😊
🙌 2
Another issue is that, because I'm using a cli to wrap the dotnet automation api, I'd like "interactive" to be configurable rather than always disabled. That way I can get the rich cli output unless I also pass 
--non-interactive
to my cli and pass that to the automation api.
l

lemon-agent-27707

06/04/2021, 9:08 PM
Hm I am not sure that is something that we would support. The reason I say that is that the fact that the automation api uses the CLI is an implementation detail. One day it might be an RPC interface. My suggestion here would be that it would be awesome to see some community supported renderers for engine events in the various languages. Might be something that we contribute to eventually but it isn't on our radar right now
A few folks have done things in go, and nodejs, but nothing in dotnet that I'm aware of yet
b

bored-oyster-3147

06/04/2021, 10:31 PM
@lemon-agent-27707 Out of curiosity what additional functionality do these engine event renderers provide on top of the ability to hook into the engine event listener?
l

lemon-agent-27707

06/05/2021, 3:29 PM
By renderer, I mean a general purpose utility that takes engine events and turns them into some sort of UX. This is the event renderer for the pulumi CLI (or at least the entrypoint): https://github.com/pulumi/pulumi/blob/master/pkg/backend/display/display.go#L36-L64
👍 2
It isn't particularly reusable at this point, but you could imagine that being factored out into a library. That way when others want to build CLIs with automation api, they can just call the renderer with engine events to show the "friendly" CLI output.
#automation-apiJoin Slack
Powered by