https://pulumi.com logo
Powered by
#python
Title
i

important-magician-41327

11/29/2021, 8:56 AM
Dear Experts, I want to design a proper python class for a Pulumi AWS deployment, but I am struggling to find the right design. Can someone please help me and tell me what would be the best approach to design a pulumi class that has the following functionality. My design goal is to reuse as much as possible and modularise everything. Those are the Functions which needs to be nested into a Class or Classes:
Copy code
Region:
+ VPC
+ IGW

Site
- VPC(region A)
- IGW (region A)
+ Subnet
+ Route Table

Site Type A
- VPC(region A)
- IGW (region A)
- Subnet (site 1)
- Route Table (site 1)
+ EC2

Site Type B
- VPC(region A)
- IGW (region A)
- Subnet (site 1)
- Route Table (site 1)
+ ECK
In the above example, resources in Region A is provisioned by the Region Class. The Site Class will import these resources and configure additional resources. Site Type A will import everything from Site 1 and configure some additional resources like EC2 or ECK. How should I structure this with Python Inheritance and Composition?
p

prehistoric-activity-61023

11/29/2021, 8:57 AM
I’d say in general, you should favor composition over inheritance.
I’d simply create a ComponentResource (e.g. 
Site
) that expects another component resource (e.g. 
Region
) as its dependency (you can pass it in the constructor).
🙏 1
i

important-magician-41327

11/29/2021, 9:01 AM
do you have an example how you would do it? How can you access the resources from Sites from Region? I could not find a proper example
p

prehistoric-activity-61023

11/29/2021, 9:02 AM
Use dependency injection
you can pass instance of 
Region
class to the constructor of 
Site
Copy code
class Site(pulumi.ComponentResource):
    def __init__(
        self,
        name: str,
        region: Region,
        opts: Optional[pulumi.ResourceOptions] = None,
    ):
        super().__init__("something:something:Site", name, {}, opts)
        ...
then you can create the region resource first and pass it to 
Site
constructor:
Copy code
region = Region(...)
site = Site("my-site", region=region)
pulumi does not enforce any dependency injection mechanism or design patterns so you’re free to use anything you want
I strongly recommend using ComponentResource. It’s a very thin layer (all it does is creating parent-child relationships and allow you to use parent providers) but it greatly improves readability of the stack resources.
i

important-magician-41327

11/29/2021, 9:07 AM
excellent let me try this! When I understand it right, the Site Class will call the Region class and then It can access all the ressoruces?
Copy code
site = Site("my-site", region=region)
and this is the “trick”?
p

prehistoric-activity-61023

11/29/2021, 9:08 AM
I wouldn’t call it a trick (you can do real magic in python). You simply pass the 
Region
object to the 
Site
class instance so it can access its properties.
Copy code
class Site(pulumi.ComponentResource):
    def __init__(
        self,
        name: str,
        region: Region,
        opts: Optional[pulumi.ResourceOptions] = None,
    ):
        super().__init__("something:something:Site", name, {}, opts)
=>      self.region_name = region.name
you’re gonna get the object in 
__init__
method so based on that you can create additional resources
let me copy’n’paste real life example from one of my repos:
Copy code
class GKECluster(ComponentResource):
    def __init__(
        self,
        name: str,
        network: gcp.compute.Network,
        opts: Optional[ResourceOptions] = None,
    ):
        super().__init__("jakub-bacic:gcp:GKECluster", name, {}, opts)

        self.subnetwork = gcp.compute.Subnetwork(
            name,
            ip_cidr_range=subnetwork_cidr_block,
            network=network.name,
            secondary_ip_ranges=[
                gcp.compute.SubnetworkSecondaryIpRangeArgs(
                    range_name="pods", ip_cidr_range="10.2.0.0/16"
                ),
                gcp.compute.SubnetworkSecondaryIpRangeArgs(
                    range_name="services", ip_cidr_range="10.3.0.0/16"
                ),
            ],
            private_ip_google_access=True,
            opts=ResourceOptions(parent=self),
        )
        ...
it’s just a part but it should help with get the idea
i

important-magician-41327

11/29/2021, 9:11 AM
ok great! Again thanks for your help, let me try to adapt to my use-case!!!!
p

prehistoric-activity-61023

11/29/2021, 9:12 AM
basically, my 
GKECluster
component resource expects the user to pass the already existing 
gcp.compute.Network
and then it’s gonna use it to create additional resources (dedicated subnetwork for GKE and the cluster itself of course)
Copy code
gke = GKECluster(
    common.default_resource_name,
    network=common.vpc_network,
    opts=pulumi.ResourceOptions(
        provider=common.gcp_provider,
    ),
)
^ here’s how I create an instance of this resource
how you’re gonna pass the required 
network
argument is up to you (as you can see, I import it from another module called 
common
)
b

billowy-army-68599

11/29/2021, 4:23 PM
@important-magician-41327 here's a great python component resource example: https://github.com/jen20/pulumi-aws-vpc/tree/master/python
🙏 1
i

important-magician-41327

11/29/2021, 4:46 PM
@billowy-army-68599 can you also send me some example, how to call the Vpc Class?
b

billowy-army-68599

11/29/2021, 4:47 PM
i

important-magician-41327

11/29/2021, 4:48 PM
ahhh…sorry I only looked in the Python folder
Great Example! Thanks, relates directly what I’m currently creating
I’ve done my “homework” and I have created a simple AWS Region and Subnet ComponentResource The goal of this code is to create in a region a VPC and trigger a Site class which creates Subnets per site. I want to illustrate the problem here: Because VPC can only exist once per Region, I have to pass the Region Class to the Site, do avoid Duplicate creation. As you can see this code does not Scale because I have to create the Region Class for every region. How Can I do this better, to loop thru every region and than call it from the Site Class?
Copy code
import pulumi as pulumi
import pulumi_aws as aws


class Region(pulumi.ComponentResource):
    def __init__(self,name,region,aws_providers,opts: pulumi.ResourceOptions = None):
        self
        name: str
        super().__init__('Region', name, None, opts)

        # #Get all available AZ's in VPC
        self.available = aws.get_availability_zones(state="available",
                                                opts=pulumi.ResourceOptions(
                                                provider=aws_providers.get(region)
                                                )
        )

        #Create VPC
        self.vpc = aws.ec2.Vpc(f"vpc_sdwan_{region}", 
                            cidr_block=f"10.0.0.0/16",
                            tags={
                                "Name": f"vpc_sdwan_{region}", 
                                "Owner": "cloudPOC"},
                            opts=pulumi.ResourceOptions(
                            provider=aws_providers.get(region),
                            )
        )        


class Site(pulumi.ComponentResource):
    def __init__(self,name,site,region,aws_providers,region_eu,opts: pulumi.ResourceOptions = None):
        self
        name: str
        self.region = region,
        self.aws_providers = aws_providers
        self.region_eu = region_eu
        super().__init__('Site', name, None, opts)

        self.subnet_lan = aws.ec2.Subnet(f"subnet_lan_site{site}",
            vpc_id=region_eu.vpc.id,
            cidr_block=f"10.0.0.0/28",
            availability_zone=region_eu.available.names[0],
            tags={
                    "Name": f"subnet_lan_site{site}",
                    "Owner": "cloudPOC"
                },
            opts=pulumi.ResourceOptions(
            provider=aws_providers.get(region),
            )
        )

#create AWS providers for every region
regions=["eu-central-1","us-east-1"]
aws_providers = {
        region: aws.Provider(f"aws-provider-{region}", region=region)
        for region in regions
    }

#create Region Object for every region
region_eu = Region("Region_eu-central-1","eu-central-1",aws_providers)
region_us = Region("Region_us-east-1","us-east-1",aws_providers)

#create sites
site1 = Site("Site1",1,"eu-central-1",aws_providers,region_eu)
site2 = Site("Site2",2,"eu-central-1",aws_providers,region_eu)
site3 = Site("Site3",3,"us-east-1",aws_providers,region_eu)
site4 = Site("Site4",4,"us-east-1",aws_providers,region_eu)
p

prehistoric-activity-61023

11/30/2021, 6:36 PM
Can you elaborate why it doesn’t scale? I don’t see any issue with the fact you’re gonna create multiple 
Region
instances if that fits the real life scenario.
I think you’d have to pinpoint the issue you’re worried about to further improve that.
One thing I can suggest (but it’s just a refactor) is to rely on implicit provider injection available in pulumi. If you specify no provider -> resource will try to use the provider assigned to the parent resource. You can write your 
Region
class like this:
Copy code
class Region(pulumi.ComponentResource):
    def __init__(self, name: str, region: str,opts: Optional[pulumi.ResourceOptions] = None):
        super().__init__('Region', name, None, opts)

        ...

        #Create VPC
        self.vpc = aws.ec2.Vpc(
            f"vpc_sdwan_{region}", 
            cidr_block="10.0.0.0/16",
            tags={
                "Name": f"vpc_sdwan_{region}", 
                "Owner": "cloudPOC",
            },
            opts=pulumi.ResourceOptions(
==>              parent=self,
            ),
        )
and you can assign the provider when creating an instance of this class:
Copy code
region_eu = Region(
    "Region_eu-central-1", 
    region="eu-central-1", 
=>  opts=pulumi.ResourceOptions(
=>      provider=aws_providers_for_this_region,
=>  ),
)
🙌 1
Benefits of that approach: • you can reuse your 
Region
component resource with default provider (in case you have a much simpler project where you don’t create providers explicitly) • assigning 
parent
gives you a nice, hierarchical view over your resources instead of flat/one-level mess (just implement the changes above and see the difference in 
pulumi preview
) BTW, the way you’re gonna assign parents is totally up to you. You can even make 
Region
a parent of 
Site
resource if that fits your infrastructure design 🙂.
i

important-magician-41327

12/01/2021, 8:47 AM
Thanks Jakub! Great advice with the AWS providers, much cleaner. I still have a minor error, maybe you see the issue?
Copy code
import pulumi as pulumi
import pulumi_aws as aws


class Region(pulumi.ComponentResource):
    def __init__(self, name: str, region: str, opts: pulumi.ResourceOptions = None):
        super().__init__('Region', name, None, opts)

        # #Get all available AZ's in VPC
        self.available = aws.get_availability_zones(state="available",
                                                opts=pulumi.ResourceOptions(
                                                parent=self
                                                )
        )

        #Create VPC
        self.vpc = aws.ec2.Vpc(f"vpc_sdwan_{region}", 
                            cidr_block=f"10.0.0.0/16",
                            tags={
                                "Name": f"vpc_sdwan_{region}", 
                                "Owner": "cloudPOC"},
                            opts=pulumi.ResourceOptions(
                            parent=self,
                            )
        )        



class Site(pulumi.ComponentResource):
    def __init__(self,name,site,region,aws_providers,region_eu,opts: pulumi.ResourceOptions = None):
        self
        name: str
        self.region = region,
        self.aws_providers = aws_providers
        self.region_eu = region_eu
        super().__init__('Site', name, None, opts)

        self.subnet_lan = aws.ec2.Subnet(f"subnet_lan_site{site}",
            vpc_id=region_eu.vpc.id,
            cidr_block=f"10.0.0.0/28",
            availability_zone=region_eu.available.names[0],
            tags={
                    "Name": f"subnet_lan_site{site}",
                    "Owner": "cloudPOC"
                },
            opts=pulumi.ResourceOptions(
            provider=aws_providers.get(region),
            )
        )


#create Region Object for every region
region_eu = Region("Region_eu-central-1", "eu-central-1", opts= pulumi.ResourceOptions(provider="eu-central-1",))
I got this error:
Copy code
File "/home/coder/myproject/venv/lib/python3.8/site-packages/pulumi/runtime/stack.py", line 133, in __init__
        func()
      File "/home/coder/.pulumi/bin/pulumi-language-python-exec", line 106, in <lambda>
        coro = pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
      File "/usr/lib/python3.8/runpy.py", line 282, in run_path
        return _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/home/coder/myproject/./__main__.py", line 59, in <module>
        region_eu = Region("Region_eu-central-1", "eu-central-1", opts= pulumi.ResourceOptions(provider="eu-central-1",))
      File "/home/coder/myproject/./__main__.py", line 7, in __init__
        super().__init__('Region', name, None, opts)
      File "/home/coder/myproject/venv/lib/python3.8/site-packages/pulumi/resource.py", line 913, in __init__
        Resource.__init__(self, t, name, False, props, opts, remote, False)
      File "/home/coder/myproject/venv/lib/python3.8/site-packages/pulumi/resource.py", line 772, in __init__
        providers = convert_providers(opts.provider, opts.providers)
      File "/home/coder/myproject/venv/lib/python3.8/site-packages/pulumi/runtime/resource.py", line 651, in convert_providers
        return convert_providers(None, [provider])
      File "/home/coder/myproject/venv/lib/python3.8/site-packages/pulumi/runtime/resource.py", line 661, in convert_providers
        result[p.package] = p
    AttributeError: 'str' object has no attribute 'package'
    error: an unhandled error occurred: Program exited with non-zero exit code: 1
Regarding scaling, this is now the solution, I don’t need to manually create a variable per region, I will now use a dictionary instead:
Copy code
#create Region Object for every region
for region in regions:
    aws_region[region] = Region(f"Region_{region}", region, opts= pulumi.ResourceOptions(provider=region))

#create sites
site1 = Site("Site1",1,"eu-central-1",aws_region["eu-central-1"])
site2 = Site("Site2",1,"eu-central-1",aws_region["eu-central-1"])
site3 = Site("Site3",1,"eu-central-1",aws_region["us-east-1"])
site4 = Site("Site4",1,"eu-central-1",aws_region["us-east-1"])
creating a dictionary with all the unique Region objects and call it with the Site object, to avoid duplicate resource creation
p

prehistoric-activity-61023

12/01/2021, 10:14 AM
Two things I can quickly comment: • minor thing but I have to say it: 
opts: pulumi.ResourceOptions = None
is a wrong type hint, it should be 
opts: Optional[pulumi.ResourceOptions]
(if it’s not optional, it shouldn’t be 
None
at any point) • you’re passing a 
str
value instead of provider instance in:
Copy code
region_eu = Region("Region_eu-central-1", "eu-central-1", opts= pulumi.ResourceOptions(provider="eu-central-1",))
creating a dict definitely sounds like a plan
if it’s something constant for your projects, define that in code
if it can vary from stack to stack, read these values from stack config file so you can easily edit that in one place if you need
Powered by