This message was deleted.
# general
s
This message was deleted.
b
can you share your CI config?
also cc @clever-sunset-76585
c
but no output from 
pulumi up
 is added afterwards..
The GitLab integration adds notes to the MR for
pulumi preview
only. I am curious — are you running
pulumi up
in the MR or are you running it once the MR is merged? Also I realize it isn’t the same thing but you can now subscribe to stack update (i.e.
pulumi up
) success/failure from the Pulumi Console and get email notifications. If you go to the Pulumi Console for the stack in question, you should see a “Watch” button to the right and just below the avatar button that’ll let you subscribe to updates.
a
Thanks for the assistance @billowy-army-68599 and @clever-sunset-76585. You can find the CI-related files below
r
Why does the GitLab integration not provide output of
pulumi up
in the MR? Would be highly appreciated! 🙏
PS: Until now we ran
pulumi preview
for
only: merge_request
and
pulumi up
for
only: main, dev
.
c
Why does the GitLab integration not provide output of 
pulumi up
 in the MR?
Would be highly appreciated!
That’s good feedback! I will take that feedback back to our team.
PS: Until now we ran 
pulumi_up
 for 
only: merge_request
 and 
pulumi up
 for 
only: main, dev
 .
Do you mean
pulumi preview
for
only: merge_request
…?
👍 2
r
We actually had a case where GitLab marked the jobs of the pipeline as successful or "passed with warnings" although it was a failure. Hence, super misleading! The fact that it was a failure and the reason for it would only be discoverable via the pulumi UI. So especially for that case, the output of
pulumi up
would be really helpful! 🙂 Getting email notifications for that are also nice, but it would be better to have all the information and links to pulumi UI on the merge request (as is already the case for
pulumi preview
).
c
Ah I see. I would love for you to checkout the stack email notifications in the meantime and let us know what you think!
r
So, we now changed the workflow such, that everything is executed as follows (all on
only: merge_requests
and
when: manual
for
pulumi up
) :
Copy code
assert-formatting:
  stage: format
  before_script:
    - pip install black
  script:
    - black --check __main__.py
  only:
    - merge_requests
    - dev
    - master

pulumi-preview:
  stage: infrastructure-preview
  script:
    - ./scripts/pulumi-run.sh -o ${CI_PROJECT_ROOT_NAMESPACE} -p ${CI_PROJECT_NAME} -s ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME} -c "preview --suppress-outputs" -r "eu-central-1" > pulumi-log.txt
  # Create an artifact archive with just the pulumi log file,
  # which is created using console-redirection in run-pulumi.sh.
  artifacts:
    paths:
      - pulumi-log.txt
    expire_in: 1 week
  only:
    - merge_requests

##################################
## Updating
pulumi-up:
  stage: infrastructure-update
  script:
    - ./scripts/pulumi-run.sh -o ${CI_PROJECT_ROOT_NAMESPACE} -p ${CI_PROJECT_NAME} -s ${CI_COMMIT_REF_NAME} -c "up --yes --suppress-outputs" -r ${AWS_DEFAULT_REGION}
  only:
    - merge_requests
  when: manual
And we like the process of this! However, we still don't get the
pulumi up
output (as you said), instead three times the
pulumi preview
output 🤷
c
However, we still don’t get the 
pulumi up
 output (as you said), instead three times the 
pulumi preview
 output
Yeah each note corresponds to a specific exceution of the
preview
command. That is, each time you push an update to that same MR, a new comment is added related to the execution triggered by the latest commit to that MR.
Oh interesting. Just noticed that that last comment in your screenshot is an actual update. Hmm I guess you ran the actual update in the MR pipeline?
r
Yes, that is actually nice, so one can work really nicely on changes until the preview looks as desired. Also, we just tested the same problem we had before and now we do get notified that the
pulumi up
failed. And the
Update
link brings us to the respective error message. If only that error message could be also output into the MR, we would be already happy! 😊
Oh interesting. Just noticed that that last comment in your screenshot is an actual update. Hmm I guess you ran the actual update in the MR pipeline?
Yes, as mentioned above we run
pulumi up
in
only: merge_request
with
when: manual
. PS: Just corrected the typo in the code shared above (
only: merge_request
applied everywhere).
c
Yes, as mentioned above we run 
pulumi up
  in 
only: merge_request
 with 
when: manual
 .
Ok yeah I was suspecting it was a typo but I guess not 🙂 Do you run a
destroy
or something if it is successful then?
r
Why would we run a destroy if it is successful?
c
If you are asking that then I take it that this is not an ephemeral update and in fact the actual infrastructure. 🙂 It is generally not recommended to run the actual update in an MR because multiple MRs can be opened. Typically, pipeline concurrency is enforced for the default/main branch pipelines which allows you to control the number of simultaneous updates in flight (note that the Pulumi service also provides concurrency control, by default to avoid corruption of resources). By nature of an MR, running only a
preview
in the MR pipeline and merging the approved changes to let the push build be the only pipeline where you apply the update.
Nothing will stop you from using this setup but I suspect that you might hit failures applying the updates in MRs since the Pulumi Service disallows (due to concurrency control) multiple updates for the same stack. So you might end up having to re-run pipelines when you encounter that, which may not be a good experience.
a
Thanks for pointing that out @clever-sunset-76585! For sure that would be the safer way to go, with the one drawback being that we would no longer get the
pulumi up
error notification in the merge request 😕. So since it is just the two of us on our backend team, I think we can be careful about it for the time being. But you're right that this is definitely an issue that we will run into in the future as our team grows. So we'll be keen to hear about any updates from Pulumi 🙂
c
Yeah my suggestion would be to not do what you are doing and instead use the stack update email notification just so that when your team grows someone on your team doesn’t accidentally introduce an issue and thereby break your infrastructure 😬 Note that with MRs GitLab also allows you to control who can merge the MR eventually, which can be a good way to protect mutations to the infrastructure. By applying the update directly in the MR, you would be losing out on those protections, unfortunately.
👍 1
a
That makes a lot of sense! I don't mean to disregard your suggestion 🙂. It's just that, at the moment (and especially since we do a lot of head-to-head trial-and-error), doing the following workflow all within a single MR:
Create issue -> Initialize MR -> Develop -> See pulumi preview -> Manually accept update -> Discover error -> Document discussion -> Develop -> Preview -> Manually accept update -> See that pulumi update was successful -> Merge
Is a more effective strategy than:
Create issue -> Initialize MR -> Develop -> See pulumi preview -> Merge ("successfully") -> Wait for email -> Discover error -> Document discussion (somewhere?) -> Reopen issue -> Initialize new MR (tenuously tied to the former) -> Develop -> See preview -> Merge -> Wait for email -> See that pulumi update was successful -> Party
Hopefully that makes our current concern a bit clearer? But of course in the future, when we are working on more than one infrastructure-related issue at a time and when the changes are more marginal and less likely to fail during the update, then the route you suggest is definitely the better option!
c
No hard feelings at all 🙂 I understand that each team has their own reasoning for doing things a certain way. It’ll be a good idea, though, to re-evalute your setup at a later time. That said, I see your argument for doing it the way you are currently. However, I am surprised that a
pulumi preview
in your MR pipeline isn’t giving you the failure you might see once it is merged. I recommend digging into that and look at ways to surface that in the preview. So that your MR pipeline is reliable and you have an assurance that the changes will pass when merged. On a slightly related note, you could also look into our integration test framework that could help you test your infrastructure effectively. Although depending on what you are doing in your infrastructure app it can cause your pipeline build times to become long. The integration test framework is good for setting up and tearing-down the infrastructure.