I am using pulumi to create a Glue Streaming Job, that needs to run continuously. To trigger the run I created an ON-DEMAND Trigger in pulumi that runs it automatically.
There are several issues I am facing:
1. Any subsequent update to the Job (script location change, parameter changes, etc) would update the Glue Job definition, but does not stop the old run and start a new one. The only way I can think of forcing this is to trigger a job replacement on every change of an input. Is there a more reasonable way?
2. If I update the job script and upload it to S3, nothing in the glue job arguments changes, so the Job won't even be updated. I still want to trigger stopping and rerunning the Job because it won't pull the new version of the script otherwise.
Any ideas how to achieve this properly?