The reason I ask is yesterday it showed a new AMI was used w

Question

The reason I ask is yesterday it showed a new AMI was used which is different from one of our other clusters and now the cluster is in a bad state

fierce-dinner-20116 · Answer

Maybe you re also affected by this <https github com pulumi pulumi eks issues 84>

damp-book-35965 · Answer

Holy shit yes

damp-book-35965 · Answer

Im running windows worker nodes

fierce-dinner-20116 · Answer

It s also not immediately clear to me whether or not the ami is intended to be stable between pulumi updates

damp-book-35965 · Answer

There are 2 issues you don t want to update till you actually want to update and you cannot default to windows

damp-book-35965 · Answer

How did that happen

fierce-dinner-20116 · Answer

damp-book-35965 · Answer

Also I was running linux before that so this is a crazy scenario

fierce-dinner-20116 · Answer

It seems that windows support was only added recently

fierce-dinner-20116 · Answer

But I assume it s just because whatever API call they made before didn t have a strict enough filter

damp-book-35965 · Answer

Yes and whatever comes on the top wins

fierce-dinner-20116 · Answer

And it was fine before since Linux was the only platform that was being used

damp-book-35965 · Answer

True

fierce-dinner-20116 · Answer

fierce-dinner-20116 · Answer

there is a fix in the works it seems

fierce-dinner-20116 · Answer

I guess people that use pulumi for CD though got bitten by this

damp-book-35965 · Answer

Luckily I didn t run this in production

breezy-hamburger-69619 · Answer

We have a fix in flight

damp-book-35965 · Answer

Yes < breezy hamburger 69619> following the PR slightly smiling face

white-balloon-205 · Answer

Definitely working to get a fix out here Note that as a workaround and in general to ensure you are always using a specific AMI that you specify you can pass `nodeAmiId` explicitly For production installations you most likely will want to do this so that you are in full control of the timing of new AMI rollouts to nodes

damp-book-35965 · Answer

Absolutely I think it should be made a requirement and not an optional to make sure best practices are followed

fierce-dinner-20116 · Answer

I don t know if changing this behavior would break with how terraform does things but I agree with < damp book 35965> It seems like making the `nodeAmiId` a mandatory field and moving the existing logic to an optional helper function would help prevent people from accidentally shooting themselves in the foot

fierce-dinner-20116 · Answer

Or maybe there could be a `awsx eks` package that has this behavior since the theme of that package is better defaults

white-balloon-205 · Answer

Yes that s something I do think we should consider Feel free to open an issue to track

breezy-hamburger-69619 · Answer

Update <https github com pulumi pulumi eks pull 85> has merged and we ll be cutting a new release soon to include it

fierce-dinner-20116 · Answer

breezy-hamburger-69619 · Answer

Thank you < fierce dinner 20116> for filing the very informative issue It s greatly appreciated

busy-pizza-73563 · Answer

< damp book 35965> Meanwhile you can just pin the AMI to `ami 032ed5525d4df2de3`

damp-book-35965 · Answer

Yes doing exactly that actually exposed a best practice for us

damp-book-35965 · Answer

So will be doing it for all clusters going forward