Hey i've got a bit of an unusual one. We're runnin...
# general
n
Hey i've got a bit of an unusual one. We're running pulumi inside Temporal workers (via automation API) and for temporal to know if the worker is dead/stuck we trigger "heartbeat" on every stdout line of pulumi. Since pulumi outputs periodic
.
as it works, this usually works but we still occasionally see heartbeat timeouts even with relatively long timeouts (3 minutes) Are there any other ways to hook into pulumi to know if it is still "alive"? some kind of healthcheck we can do on it? The coarse solution would be to just check if the pulumi process is still running at all...
m
Interesting question! I don't think there is anything like a healthcheck that exists. I found eventstreams in the go SDK but not sure if that is more granular than the progress dots. Are you in a container?
@narrow-match-14013 did you figure out a solution? @lemon-scooter-94063 has some experience with Temporal and suggested just widening out the timeouts on whether alive or not.
n
Yeah slowly widening the timeouts has been the strategy for now. We're in a container and using a temporal typescript worker to call typescript pulumi (via inline function)
well one option is to just not use heartbeat at all but some of our programs can take >1hour and it would be nice to know if they've failed earlier than that (when a worker is recycle or some other reason)