https://pulumi.com logo
Join Slack
Powered by
I have a stack on DigitalOcean with three componen...
# general
h
I have a stack on DigitalOcean with three components: a domain, a bucket, and a droplet. I would like to destroy and rebuild the droplet, but leave the domain and bucket alone. What’s the best practices way of doing this? I’m hoping for something more elegant than “just manually delete the droplet then run pulumi up” 🙂
l
If this is a one-time thing, it can be as simple as commenting out the code for the droplet and running up, then commenting it back in. You can also change its Pulumi name, that will force Pulumi to destroy and recreate. If this is likely to happen repeatedly, the best approach would be to split your project in two, based on the deployment cycle. The domain and bucket would be in the base project, and export whatever properties the droplet needs. And the droplet would be in its own project, and get the required properties via StackReferences. Then you can destroy/up the droplet project any time you need.
h
It turns out the address is specific to the droplet unless I do some other shenanigans — but if it were an elastic IP or something, it’d be as simple as making two directories, splitting the Pulumi.yaml across them appropriately to get two projects, and making the droplet project depend on the base project. I will definitely look into that approach.
l
So the domain is dependent on the droplet and needs to be updated when the droplet is? Then keep it in the same project as the droplet.
h
For now, but if I use reserved IPs (I think that’s their name for elastic IPs) I can keep the same IP and just switch between droplets, which makes replacing the droplet much easier as you described above.
l
And DigitalOcean domains can't refer to droplets, they need to refer to their IP addresses? That's not awesome.
h
It is more probable to assume I am not taking full advantage of their provider than to assume defects in the provider, heh.
https://www.pulumi.com/registry/packages/digitalocean/api-docs/reservedipassignment/ shows what looks like a reasonable approach: create the reserved IP, create the droplet, assign the reserved IP to the droplet
The domain and reserved IP could go in the base project with the bucket. The droplet project would perform the assignment as necessary.
l
You can't modify a resource after it's created. If you create the domain in one project, then you can't update it in the other.
h
Right — so the domain <-> reserved IP connection would be made in the base project
foo.example.com responding to 172.16.0.1
droplet-foo would be created, and the reserved IP 172.16.0.1 would be assigned to it
Rebuild the droplet project and create droplet-bar, the foo droplet is removed and destroyed, the bar droplet gets assigned
The assignment is a resource in the droplet project
Unless I am completely misunderstanding this
l
It does look weird. Is there no way to swap between which droplet is exposed via the domain? Maybe a load balancer, or a droplet version?
h
Most of what I do is in Kubernetes where the ingress handles a lot of this
l
Ah there is a load balancer. Use that to make your life might easier.
When you replace your droplet, the LB will look after it. The domain should point at the LB.
h
Okay, so the LB acts like the reserved IP. That works. Wonder which one costs more per month? Heh.
l
Yea, cost is the perpetual scourge to the dream architecture...
h
Oooh, reserved IPs are free when attached to a droplet, $5/mo if not (because IPv4)
Load balancers are $12/mo
And since I’m not near the volume or capacity for a LB, I’m gonna go with the cheap-to-free option
l
Right. Well for now then, you're doing to have to destroy the domain any time you destroy the droplet.
So put the bucket in a different project. Best you can do.
h
Wait, you don’t think I can put the domain and the reserved IP and the bucket in the base project, and then the droplet and assignment in the droplet project?
w
I like to protect the domain and bucket, then run with 
--exclude-protected
h
Yeah, right now I’m seeing idempotency failures — running pulumi up twice on the same code wants to destroy and recreate the bucket
Protecting things sounds wise
l
running pulumi up twice on the same code wants to destroy and recreate the bucket
What does Pulumi say is the reason for this? This happens only because the name of the resource is changing, or some property that triggers replacement is changing.
What type of resource is the bucket? Is it a DigitalOcean volume? I see that description is a triggering property.. that's kinda weird.
h
It was a spaces bucket. Yesterday I deleted the entire stack and then manually cleaned everything up so I could try again.
Pulumi didn’t say why it wanted to destroy and recreate the bucket. I protected the bucket in my code and reran pulumi up. Here’s what I saw:
Copy code
View Live: <https://app.pulumi.com/mathuin/pocket-lightning/prod/previews/bc5e1ae0-849f-495b-930c-cf8d63f46bcc>

     Type                                Name                   Plan     Info
     pulumi:pulumi:Stack                 pocket-lightning-prod           
     └─ digitalocean:index:SpacesBucket  pl-bucket                       1 error


Diagnostics:
  digitalocean:index:SpacesBucket (pl-bucket):
    error: unable to replace resource "urn:pulumi:prod::pocket-lightning::digitalocean:index/spacesBucket:SpacesBucket::pl-bucket"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
This is the code that creates the bucket:
Copy code
bucket = do.SpacesBucket("pl-bucket", region=region, opts=pulumi.ResourceOptions(protect=True))
Copy code
region = config.get("region") or "sfo2"
is where region is defined
w
@hallowed-australia-10473 I believe you are missing the 
--exclude-protected
CLI flag. If I'm right, try:
Copy code
pulumi destroy --exclude-protected
h
I’m not destroying, I’m rerunning up. There’s no exclude protected for up
w
Oh, I gotcha, the issue you're having is that your protected resource now needs to be replaced, I see.
h
Well, Pulumi thinks so, and I can’t figure out why
w
Hmm... something that's odd to me is that the bucket name isn't replace-on-change in the DO schema. You're not getting a rich diff during preview that shows the difference between the old and new resource?
Could you screencap your terminal output by chance?
h
I am not getting a rich diff during preview, even when I ask for one
I posted a screencap about 29 minutes ago
Here’s a fresh one:
Copy code
~/g/pocket-lightning ❯❯❯ pulumi up                          reserved-ips ✚ ✱ ◼
Previewing update (prod)

View Live: <https://app.pulumi.com/mathuin/pocket-lightning/prod/previews/c21bd27c-45f7-40a1-b2d4-191db031c3e3>

     Type                                Name                   Plan     Info
     pulumi:pulumi:Stack                 pocket-lightning-prod
     └─ digitalocean:index:SpacesBucket  pl-bucket                       1 erro


Diagnostics:
  digitalocean:index:SpacesBucket (pl-bucket):
    error: unable to replace resource "urn:pulumi:prod::pocket-lightning::digitalocean:index/spacesBucket:SpacesBucket::pl-bucket"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
w
Sure, I wasnt sure if the text you had posted was the full terminal output or if it was a sample.
Thanks for reposting.
h
Copy code
~/g/pocket-lightning ❯❯❯ pulumi up --diff               ✘ 1 reserved-ips ✚ ✱ ◼
Previewing update (prod)

View Live: <https://app.pulumi.com/mathuin/pocket-lightning/prod/previews/2091b01e-1306-4d95-9f66-cb9b74f5bb86>

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:prod::pocket-lightning::pulumi:pulumi:Stack::pocket-lightning-prod]
error: unable to replace resource "urn:pulumi:prod::pocket-lightning::digitalocean:index/spacesBucket:SpacesBucket::pl-bucket"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
Resources:
    2 unchanged
That’s with the diff argument
w
Oh, what's the output of 
pulumi preview
?
h
Copy code
~/g/pocket-lightning ❯❯❯ pulumi preview               ✘ 255 reserved-ips ✚ ✱ ◼
Previewing update (prod)

View Live: <https://app.pulumi.com/mathuin/pocket-lightning/prod/previews/8e643fe8-b1fb-4a25-9457-5a389f7df783>

     Type                                Name                   Plan     Info
     pulumi:pulumi:Stack                 pocket-lightning-prod
     └─ digitalocean:index:SpacesBucket  pl-bucket                       1 erro


Diagnostics:
  digitalocean:index:SpacesBucket (pl-bucket):
    error: unable to replace resource "urn:pulumi:prod::pocket-lightning::digitalocean:index/spacesBucket:SpacesBucket::pl-bucket"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up
`
w
Man, that message really is unhelped! I should open an issue if there isn't one already.
Well, here's my next pitch to try to debug this:
Try changing your code to:
Copy code
bucket = do.SpacesBucket("pl-bucket", region=region, opts=pulumi.ResourceOptions(protect=False))
...and running 
pulumi preview
, just so we can see what Pulumi thinks is supposed to change.
h
Copy code
~/g/pocket-lightning ❯❯❯ pulumi preview               ✘ 255 reserved-ips ✚ ✱ ◼
Previewing update (prod)

View Live: <https://app.pulumi.com/mathuin/pocket-lightning/prod/previews/b8c217ab-59de-41fb-bd79-97fb66a61b0a>

     Type                                        Name                   Plan
     pulumi:pulumi:Stack                         pocket-lightning-prod
 +-  ├─ digitalocean:index:SpacesBucket          pl-bucket              replace
 +-  ├─ digitalocean:index:ReservedIpAssignment  pl-ripa                replace
 +-  └─ digitalocean:index:Droplet               pl-droplet             replace


Outputs:
  ~ bucket_name: "pl-bucket-66ee4ad" => "pl-bucket-4077b39"
  ~ droplet_ip : "138.197.208.46" => output<string>
  - reserved_IP: "138.68.36.145"
  + reserved_ip: "138.68.36.145"

Resources:
    +-3 to replace
    3 unchanged
So this would trash the bucket and the droplet with no change in the config files at all? (IP -> ip was the “minor” non-impacting change I added)
Oooh actually the user data for the droplet has the bucket name
w
Aha! It looks like... the bucket name, which is autogenerated, does indeed trigger replace-on-change.
h
replace-on-change when the change is the hash seems a little unexpected
w
replace-on-change when the change is the hash seems a little unexpected
Yeah, that's a known ergonomic issue with the way we do auto-naming. There's work underway to revamp auto-naming but I'm not sure the current status.
h
I’m now wondering why the domain and reserved IP are not affected
Copy code
# create the reserved IP
rip = do.ReservedIp("pl-rip", region=region, opts=pulumi.ResourceOptions(protect=True))
pulumi.export("reserved_ip", rip.ip_address)

# create the domain
domain = do.Domain(
    "pl-domain",
    name=hostname,
    ip_address=rip.ip_address,
    opts=pulumi.ResourceOptions(protect=True),
)
pulumi.export("domain_name", domain.name)
w
As a general rule, to work around this difficulty with "replace-on-change" and autonaming, if there's a 
name
field for your particular resource, then you should try to specify it. If you're making a one-off resource, then you can hardcode it. If your resource is part of a component, you should try to incorporate the component's name (as provided by the component constructor). You can template the name into a string to do this. So if this bucket was in a component named 
MyDOComponent
you might want to name the resource to be 
MyDOComponent-bucket
vis a vis 
"{0}-{1}".format(component_name, "bucket")
Sorry about the ergonomics there. I'd say this idiosyncrasy is more than a "papercut" due to Pulumi's autonaming...
h
Is there a way to turn off replace-on-change for the bucket?
w
If we've correctly diagnosed the issue, then there's a bug in the DO schema documentation for 
SpacesBucket
where 
name
should be 
replace-on-change
.
No, that's a rule determined by the provider. ReplaceOnChange means "hey this thing is not longer fundamentally the same as it was if you change this property." In the case of a bucket, it's URI is determined by its name. So if you create a bucket 
jack-twilley
, and try to change it's name, you're fundamentally creating a new bucket according to the cloud provider (because you're changing what is expected to be a permanent URI for the bucket).
h
There is a plus to the random tag on the end of the name: it can take a while for a bucket destroy to finish, and I can pulumi up immediately after pulumi destroy because it’ll make a new name for the bucket.
Not sure if that’s worth this downside
w
Put another way, if you had, say, a 
CreditCard
resource, and you wanted to change the 
CreditCard.cc_number
field, your cloud provider would be within their rights to say "hey man this resource isn't editable! You've got to delete the credit card and make a new one!" It's the same idea with buckets and their names. Or, like "database cluster and engine type".
There is a plus to the random tag on the end of the name: it can take a while for a bucket destroy to finish, and I can pulumi up immediately after pulumi destroy because it’ll make a new name for the bucket.
Hmm... yeah I'm not sure I have any insight to offer; it sounds like a personal choice. I personally don't want my infrastructure to have nondeterminism -- I want to minimize the randomness as much as possible to make it as reproducible as possible. But if that's something you like for the purpose of performance an convenience, I won't yuck your yum 😸
h
I’m past the part of my dev cycle where I destroy/up very often so I’m okay moving past that and into the warm comforting arms of determinism
I guess I figured it made a name and would keep that name if nothing else changes, but there’s nothing else except a name for a bucket and how would it necessarily know that the old name was good
And “replace” isn’t quite what it sounds like for a bucket, or a database — it’s not like the new bucket gets created and the files all get copied over from the old one, right?
w
And “replace” isn’t quite what it sounds like for a bucket, or a database — it’s not like the new bucket gets created and the files all get copied over from the old one, right?
Correct! It's not a "move", it destroys the old and creates a fresh one. Which is exactly the opposite of what happens when you change a non-
replace-on-change
property. Conceptually, that's more like a "mv", in that it updates the properties of a resource without needing to destroy it and create a new one.
h
I think I’m going to set the bucket name and embed the stack name in it — I already embed the project name sorta so it’s not that big a stretch
w
For example, you can change the maintenance windows for a database cluster without having to replace the cluster with a new instance.
h
Still don’t know if I’d call it a bug in DO or in my understanding
Now to destroy that stuff I just made hard to destroy, haha
… and now I do think I have a bug, or at least an ugly wart.
Copy code
pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:prod::pocket-lightning::pulumi:pulumi:Stack::pocket-lightning-prod]
    +-digitalocean:index/spacesBucket:SpacesBucket: (replace) 🔓
        [id=pl-bucket-prod]
        [urn=urn:pulumi:prod::pocket-lightning::digitalocean:index/spacesBucket:SpacesBucket::pl-bucket]
        [provider=urn:pulumi:prod::pocket-lightning::pulumi:providers:digitalocean::default_4_16_0::1533d91c-1447-40ca-ae1c-142b5698f63b]
      ~ region: "sfo2-zg02" => "sfo2"
What the heck is that on the region?
Another day’s pain
Open in Slack
PreviousNext
Powered by