Hi everyone, I’m a Data Platform Engineer at Dream11, the largest fantasy platform in India with over 200 million users. Background I’ve been exploring Pulumi for managing infrastructure and initially found its SDK-based approach appealing because it allows writing infrastructure as code in declartive way using programming languages like Python. However, I’ve run into challenges and would like your views on them. Challenges 1. Its hard to think of approach to use pulumi when managing “pet”-like resources—those that require careful handling and cannot simply be replaced, such as stateful services like kafka and pulsar. 2. In my view, if the Pulumi program is very dynamic or not available, it can be inefficient and risky to recreate it just to update a specific resource. For example, I use templates to define EC2 configurations in AWS, and based on some settings, these templates generate the final Pulumi program. In this case, I don’t have a static or fixed Pulumi program that was originally used to create the infrastructure. Questions 1. Does Pulumi allow granular control to update specific resources without needing to reconstruct or recreate the entire stack pulumi program? 2. Is Pulumi suited for managing “pet”-like resources that require careful handling and updates? 3. Are there better workflows or features in Pulumi to address these challenges? P.S. - https://www.pulumi.com/blog/five-years-of-infrastructure-as-code-part-one/ Using the pet and cattle analogy from this article

Does Pulumi allow granular control to update specific resources without needing to reconstruct or recreate the entire stack pulumi program?

You can use --target to target specific resources in a stack for update.

Is Pulumi suited for managing “pet”-like resources that require careful handling and updates?

Yes, generally you would use the protect flag to ensure you don't accidentally delete them.

And also: use smaller projects. If you have a resource (e.g. an EC2 instance) that you intend to be a "pet", then you probably won't want to update it very often. But there may be many related resources (e.g. its security groups, load balancer, etc.) that you want to update more often. So: have them in separate projects.

@echoing-dinner-19531 @little-cartoon-10569 Thank you for your input. I’d appreciate it if you could also assist me with the following scenario. Your perspective would be highly valued. Current State

Current stack resources (3):
    TYPE                          NAME
    pulumi:pulumi:Stack           pulumi_poc-abc
    │  URN: urn:pulumi:abc::pulumi_poc::pulumi:pulumi:Stack::pulumi_poc-abc
    ├─ aws:ec2/instance:Instance  test-4
    │     URN: urn:pulumi:abc::pulumi_poc::aws:ec2/instance:Instance::test-4
    └─ pulumi:providers:aws       default_6_59_1
          URN: urn:pulumi:abc::pulumi_poc::pulumi:providers:aws::default_6_59_1

Code To Run

def new_instance():
    # Import the existing EC2 instance
    existing_instance = aws.ec2.Instance.get(
        resource_name='existingInstance',
        id='xxxxxxxxx',
    )
    instance = aws.ec2.Instance(
        f"test-5",
        ami="xxxxxxxxxx",
        instance_type="c5n.4xlarge",
        key_name="xxxxxxxxxx",
        subnet_id="xxxxxxxxxx",
        vpc_security_group_ids=["xxxxxxx"],
        iam_instance_profile="xxxxxxxxx",
    )
    pulumi.export(
        f"ip",
        pulumi.Output.all(instance.private_ip)
    )
    pulumi.export('instance_id', instance.id)
    pulumi.export('instance_type', instance.instance_type)


stack = auto.create_or_select_stack(
    stack_name="abc",
    project_name="pulumi_poc",
    program=new_instance
)

Result

Updating (abc):
@ updating....
 +  aws:ec2:Instance test-5 creating (0s)
@ updating.......
    pulumi:pulumi:Stack pulumi_poc-abc running read aws:ec2:Instance existingInstance
@ updating................
 +  aws:ec2:Instance test-5 created (16s)
 -  aws:ec2:Instance test-4 deleting (0s)
@ updating.......................................................
 -  aws:ec2:Instance test-4 deleted (52s)
@ updating....
    pulumi:pulumi:Stack pulumi_poc-abc
Outputs:
  ~ instance_id  : "xxxxxx" => "xxxxxx"
    instance_type: "c5n.4xlarge"
  ~ ip           : [
      ~ [0]: "xxxxx" => "xxxxxxx"
    ]

Required Behaviour test-4 instance should also exists test-5 instance should also exists Question What could be the possible solution if i wanted the required behaviour? I looked at stack reference (but as far i understand it is only for getting output from stacks), importing resources (this is also i think to get resources not managed by pulumi)

Just use multiple stacks? Your only using one stack state "abc" so each time you run it it will update everything stored in that state. If you want multiple copies of the same program just have multiple stacks.

The goal I am trying to achieve is to provide a set of APIs that enable users to manage the nodes within their cluster. Users should be able to add, remove, or update instances in their cluster. Each stack consists of nodes within a cluster, and we need to perform add, remove, or modify operations based on API requests from users, ensuring that these operations happen within the same cluster/stack.

Then have a loop in that code to loop over all the descriptions you've had from users. Pulumi is a goal state based system, you have to tell it everything you want for the stack each time you run it. Not just what new things to add.

Understood, in above program i made a mistake Mistake

existing_instance = aws.ec2.Instance.get(
        resource_name='existingInstance',
        id='xxxxxxxxx',
    )

I put the name of resource as

existingInstance

instead in my stack the resource name was test-4, after modifying my program to below it didn’t delete the existing resources. Updated

def new_program():
    # Import the existing EC2 instance
    test_4 = aws.ec2.Instance.get(
        resource_name='test-4',
        id='xxxxx',
    )
    test_6 = aws.ec2.Instance.get(
        resource_name='test-6',
        id='xxxxxx',
    )
    test_7 = aws.ec2.Instance.get(
        resource_name='test-7',
        id='xxxxxxx',
    )
    instance = aws.ec2.Instance(
        f"test-8",
        ami="xxxxxxxxxx",
        instance_type="c5n.4xlarge",
        key_name="xxxxxxxxxx",
        subnet_id="xxxxxxxxxx",
        vpc_security_group_ids=["xxxxxxx"],
        iam_instance_profile="xxxxxxxxx",
    )

    pulumi.export(
        f"ip",
        pulumi.Output.all(instance.private_ip)
    )

    pulumi.export('instance_id', instance.id)
    pulumi.export('instance_type', instance.instance_type)

This kept all my previous resources, and adds only the new one. Just one last question: Do you recommend

test_4 = aws.ec2.Instance.get(
        resource_name='test-4',
        id='xxxxx',
    )

this way of putting my existing resources in the program, from what i have understood if give the same name and id it treats it as same resource.

yes same id and same name == same resource. But this is a really odd way of managing it. You don't need to change them to 'get' methods you can just leave them as ec2.Instance methods with the same name

aws.ec2.Instance(resource_name="test-8",id="xxxxxxxxxxxx")

Error:

TypeError: Instance._internal_init() got an unexpected keyword argument 'id'

python SDK version: 3.138.0 Am i using some older version?

No, as in when you first made "test-8" you did it with a block of code like:

instance = aws.ec2.Instance(
        f"test-8",
        ami="xxxxxxxxxx",
        instance_type="c5n.4xlarge",
        key_name="xxxxxxxxxx",
        subnet_id="xxxxxxxxxx",
        vpc_security_group_ids=["xxxxxxx"],
        iam_instance_profile="xxxxxxxxx",
    )

Just leave that code exactly as is. The state file tracks the id for "test-8" you don't need it in your program as well.

oh okay, i got it. Thanks for all the help @echoing-dinner-19531. Much appreciated.