If anyone else runs into this, it's likely caused by accidently committing a similar named stack to your repository. When the automation job runs, if you're expecting to create a stack on s3, it will still pick up the local stack json file and use whatever configuration you left when it was created. In our case, we just needed to stop tracking those local files that were unintentionally committed.