https://pulumi.com logo
#general
Title
# general
b

bitter-dentist-28132

01/30/2020, 3:18 PM
can pulumi do automated rollbacks yet?
❤️ 1
g

green-school-95910

01/30/2020, 5:34 PM
❤️ 1
b

bitter-dentist-28132

01/30/2020, 5:37 PM
thanks!
❤️ 2
t

thankful-optician-22583

01/30/2020, 6:13 PM
Guys this is what I did to handle rollback, I first create an output object called
Copy code
export const appTag ='your image tag';
before I run pulumi update, I query the output file like this
Copy code
export PREV_APP_TAG=$(pulumi stack output -j | jq '.appTag')
And then if my update fails, i set this
Copy code
pulumi config set app:tag $PREV_APP_TAG
and then run a pulumi update again.
💯 1
g

green-school-95910

01/30/2020, 6:15 PM
And you put the commit sha/git tag on that?
❤️ 1
t

thankful-optician-22583

01/30/2020, 6:16 PM
yes exactly @green-school-95910
g

green-school-95910

01/30/2020, 6:17 PM
Clever! Never though of using the output as a flag for successful completion. Way shorter than my solution.
❤️ 1
t

thankful-optician-22583

01/30/2020, 6:21 PM
fam this discussion is good thanks
b

big-caravan-87850

01/30/2020, 6:32 PM
this works for rolling back when only the image changes. does anyone have suggestions for how to handle k8s objects (configmaps, services, etc) changing?
g

green-school-95910

01/30/2020, 6:33 PM
In this case my solution works, it is more general as it uses just git and the platform, I mentioned in the other thread that I linked above
💯 1
Essentially is a script to be run on errors of the pipeline that finds the last successful before the current and starts a new one at that commit
partypus 8bit 1
b

big-caravan-87850

01/30/2020, 7:35 PM
that makes sense. our setup is a bit different. we have multiple pulumi programs in a single git repo and only deploy the programs that have changed -- either a new image (application change) or pulumi code (k8s change) . right now we are thinking of having a stage at the end of each pipeline that allows you to deploy any of the apps in our repo. you normally wouldn't run anything in this stage as it is only for rollbacks and it would only be available for certain users. if you need to roll back you would just go the previous job and invoke the pulumi up step in the stage mentioned above on any applications you want to roll back. its a manual process right not but it's fairly easy to implement.
g

gentle-diamond-70147

01/30/2020, 8:30 PM
More generally - Pulumi does not support rollback, but we advocate for roll forward by reverting to a previous commit/config and then doing a subsequent
up
to achieve rollback. More explanation is at https://www.pulumi.com/docs/troubleshooting/faq/#does-pulumi-support-automatic-rollback-in-the-event-of-an-error-or-failure.
b

bitter-dentist-28132

01/30/2020, 10:19 PM
@gentle-diamond-70147 so pulumi's opinion is that rollbacks should be manual?
g

green-school-95910

01/30/2020, 10:26 PM
IMHO a rollback command would do more harm than good. The environment that it is running in might be the cause of the failure, rolling back from it could be impossible and make the problem worse. For example, if the breaking change was a credential change that is missing some permissions (permission to create and update but not to delete for example) a recreated resource would fail both ways.
Even with systems/platforms that have it (helm or kubernetes deployment) I prefer to use a script to rerun the last successful job instead of rolling back from current state
b

bitter-dentist-28132

01/31/2020, 7:55 PM
The re-running the last job kinda makes sense to me; IIRC in gitlab, when you re-run a job, you're re-running it with the variables that were defined when the job first ran. So you avoid the problem you describe of mismatched credentials (provided they haven't expired, of course).
g

gentle-diamond-70147

01/31/2020, 8:05 PM
@bitter-dentist-28132 re: your question from yesterday, I would put it as "the decision to rollback should be manual". Generally you will want to review what happened before automatically proceeding with a rollback. The changes to achieve the rollback can be automated by doing the roll forward as I described.
I hope that distinction makes sense. :)
b

bitter-dentist-28132

01/31/2020, 8:10 PM
@gentle-diamond-70147 i definitely get the distinction, though if you're relying on manual rollback, aren't you potentially leaving your stack in an inconsistent / unusable state?
i guess a more broad question would be, how do you ensure you don't break prod?
g

gentle-diamond-70147

01/31/2020, 8:13 PM
Potentially, yes, it could be left in an inconsistent/unusable state. Automatic rollback could could also make an inconsistent/unusable state even worse, hence the recommendation for a human to review what caused a deployment to fail and then make a decision on what to do next.
i guess a more broad question would be, how do you ensure you don't break prod?
One of our engineers did a talk on exactly that recently! 🙂 https://twitter.com/PulumiCorp/status/1221843815872499713
😮 1
11 Views