can pulumi do automated rollbacks yet?

Look at Calvin's message from yesterday https://pulumi-community.slack.com/archives/C84L4E3N1/p1580317469279500

thanks!

Guys this is what I did to handle rollback, I first create an output object called

export const appTag ='your image tag';

before I run pulumi update, I query the output file like this

export PREV_APP_TAG=$(pulumi stack output -j | jq '.appTag')

And then if my update fails, i set this

pulumi config set app:tag $PREV_APP_TAG

and then run a pulumi update again.

And you put the commit sha/git tag on that?

yes exactly @green-school-95910

Clever! Never though of using the output as a flag for successful completion. Way shorter than my solution.

fam this discussion is good thanks

this works for rolling back when only the image changes. does anyone have suggestions for how to handle k8s objects (configmaps, services, etc) changing?

In this case my solution works, it is more general as it uses just git and the platform, I mentioned in the other thread that I linked above

that makes sense. our setup is a bit different. we have multiple pulumi programs in a single git repo and only deploy the programs that have changed -- either a new image (application change) or pulumi code (k8s change) . right now we are thinking of having a stage at the end of each pipeline that allows you to deploy any of the apps in our repo. you normally wouldn't run anything in this stage as it is only for rollbacks and it would only be available for certain users. if you need to roll back you would just go the previous job and invoke the pulumi up step in the stage mentioned above on any applications you want to roll back. its a manual process right not but it's fairly easy to implement.

More generally - Pulumi does not support rollback, but we advocate for roll forward by reverting to a previous commit/config and then doing a subsequent

up

to achieve rollback. More explanation is at https://www.pulumi.com/docs/troubleshooting/faq/#does-pulumi-support-automatic-rollback-in-the-event-of-an-error-or-failure.

@gentle-diamond-70147 so pulumi's opinion is that rollbacks should be manual?

IMHO a rollback command would do more harm than good. The environment that it is running in might be the cause of the failure, rolling back from it could be impossible and make the problem worse. For example, if the breaking change was a credential change that is missing some permissions (permission to create and update but not to delete for example) a recreated resource would fail both ways.

The re-running the last job kinda makes sense to me; IIRC in gitlab, when you re-run a job, you're re-running it with the variables that were defined when the job first ran. So you avoid the problem you describe of mismatched credentials (provided they haven't expired, of course).

@bitter-dentist-28132 re: your question from yesterday, I would put it as "the decision to rollback should be manual". Generally you will want to review what happened before automatically proceeding with a rollback. The changes to achieve the rollback can be automated by doing the roll forward as I described.

@gentle-diamond-70147 i definitely get the distinction, though if you're relying on manual rollback, aren't you potentially leaving your stack in an inconsistent / unusable state?

Potentially, yes, it could be left in an inconsistent/unusable state. Automatic rollback could could also make an inconsistent/unusable state even worse, hence the recommendation for a human to review what caused a deployment to fail and then make a decision on what to do next.