https://pulumi.com logo
#kubernetes
Title
# kubernetes
g

gifted-balloon-26385

01/30/2024, 9:17 PM
What’s the “right” way to handle code deploys with Pulumi k8s? Right now we pull the latest image from our repo and set that in the Deployment spec, but this means that we have to run pulumi up every time we want to deploy a new version of code. We don’t want to automatically run pulumi up unless we review the preview because something might automatically change underneath us that could be important, but we want automatic deploys to happen. As a solution, we’re using
pulumi up --target <urn of our app deployment>
, which, when it works, works, but it feels like a hack because
---target
causes issues for us (e.g https://pulumi-community.slack.com/archives/C84L4E3N1/p1705511577833539) I feel like there should be a better way to do this?
c

cuddly-computer-18851

01/30/2024, 10:54 PM
something might automatically change underneath us that could be important,
I think this is the problem to try to solve.
g

gifted-balloon-26385

01/30/2024, 11:23 PM
i think it’s pretty challenging to do when you rely on higher level pulumi providers, we’ve had a lot of minor changes happen under our feet with e.g Pulumi EKS that can cause our nodes to recycle and stuff
c

cuddly-computer-18851

01/30/2024, 11:25 PM
That's normal k8s behaviour, none of that should impact your Pulumi state
g

gifted-balloon-26385

01/30/2024, 11:29 PM
I’m talking about e.g Node AMIs being changed (because pulumi EKS pulls the latest node image). That’s something that happened to us and caused all of our workloads to move to new nodes. I fixed that by specifying a node AMI in the resource inputs, but the point is that I’ve found it unrealistic to rely on
pulumi up
being an inert activity. Maybe this is a pulumi EKS specific problem, not sure
c

cuddly-computer-18851

01/30/2024, 11:40 PM
Yeah you definitely don't want to upgrade your EKS AMI w/o realising it!
pulumi up
should (imho) be run regularly enough that changes are small, and not have unpredictable side-effects. If something is altering your infrastructure out from under Pulumi, that's not ideal. If you want less possible side-effects from a given
up
I'd look at a more micro-stack arch.
I 100% would not have my application Deployments in the same stack as my EKS infra.
g

gifted-balloon-26385

01/30/2024, 11:43 PM
it still would make me feel better to preview them before they run. like the fact that i’ve had a bunch of issues just makes me not trust letting it run on its own, even if i chase down the current ones. our app deployments are actually in a separate stack but there are other resources in there as well that I wouldn’t want accidentally changed
c

cuddly-computer-18851

01/30/2024, 11:45 PM
Ultimately you need to get to a place where you trust your CICD and testing. Deploy to lower level environments, automated tested, promote.
g

gifted-balloon-26385

01/30/2024, 11:46 PM
i mean we run preview in CI on PRs before we merge them - that’s how i’m trying to trust it
c

cuddly-computer-18851

01/30/2024, 11:46 PM
Trust, but verify 😃
g

gifted-balloon-26385

01/30/2024, 11:47 PM
that’s what i’m trying to do! but it sounds like your suggestion is that we should be cool closing our eyes and hoping that nothing ever changes underneath that. i get how that’s an ideal but it doesn’t comport with my experience with pulumi at all so far
maybe i just need to rip out all the higher level libraries we use (e.g pulumi eks) to get things to feel more like that
c

cuddly-computer-18851

01/30/2024, 11:48 PM
If you run
pulumi up --refresh
ten times without code changes, does it ever change anything?
g

gifted-balloon-26385

01/30/2024, 11:49 PM
sometimes yes it does. there’s one specific issue i’m trying to chase down with EKS where our security groups keep getting deleted and recreated, was in the middle of filing a ticket about it earlier
c

cuddly-computer-18851

01/30/2024, 11:50 PM
If it does, then it's (a) a bug (b) something is changing things w/o pulumi's knowledge (c) your pulumi code is fetching dynamic values, which should be avoided if you want consistency