Has anyone had to deal with updating sagemaker end...
# aws
Has anyone had to deal with updating sagemaker endpoints that have autoscaling enabled? You have to deregister the endpoint before you update it, is there a good way to orchestrate this via pulumi? I’ve tried making the appautoscaling.Target depend on the endpoint resource but no luck there
Current state: I define an endpoint configuration. Endpoint gets created off that configuration, then I register that endpoint’s variant as an autoscaling target. I try to update that endpoint by specifying a new model artifact version in the endpoint config. It fails because the endpoint can’t be updated while registered as an autoscaling target (???). Try to deregister first by making the endpoint depend on the autoscaling target and it fails because the variant the endpoint wants to connect to doesn’t exist yet. Set deleteBeforeReplace and it deletes and replaces before the endpoint even comes about. You can try to wing it with timing them at the same time but gl with that. So I’m going to try and perform an autoscaling deregister on update of the endpoint with that pulumi command provider. As for re-registering it… I’m hoping I can somehow track the status of that endpoint update and re-register after it completes. Haven’t done enough research yet
i ended up figuring it out. used local.command + the aws sdk to get the current variant id and deregister it before running an update on the endpoint. Trigger the local.command by having it run an echo in the beginning and passing the newest variant id to the echo, so every time we change the variant id it executes and deregisters. Then I update the endpoint, then let new aws.appautoscaling.target re-register the endpoint
if anyone runs into this and needs examples lemme know
Thanks @blue-carpenter-4997 would love to see the examples and see if they can be used for other resource types.