pulumi-community #python

hallowed-beach-15050

08/31/2020, 9:10 PM

@astonishing-quill-88807 what version of pulumi? and can you paste the error you are getting along with the code that you think may be generating the error?

hallowed-beach-15050

08/31/2020, 9:11 PM

inline or a gist would be awesome

astonishing-quill-88807

08/31/2020, 9:11 PM

I figured out the problem. I'm using v2.9.2 and my issue was trying to address the input properties as an attribute, rather than via dictionary access. Here's the code for posterity: https://github.com/mitodl/ol-infrastructure/blob/dagster_instance/src/ol_infrastructure/providers/salt/minion.py

astonishing-quill-88807

08/31/2020, 9:12 PM

It was throwing attribute errors about

salt_api_url

hallowed-beach-15050

08/31/2020, 9:47 PM

makes sense

damp-elephant-82829

09/02/2020, 6:36 AM

Guys when you write custom components in Python, does the framework offers standard logging facilities or do you use plain python logging?

damp-elephant-82829

09/02/2020, 7:24 AM

Awesome. One additional question related to ComponentResources: is it an anti pattern to give external access to child resources instead of registering their output?

strong-musician-98782

09/02/2020, 8:39 AM

Hey guys, i'm trying to create a class to create ec2 instance and then call it from my main but pulumi keeps thinking my userdata is changing even when its not, here's part of my class(tried to edit out as much as i can so some args might be missing):

Copy code

import pulumi
from pulumi_aws import ec2

class KSDBServerArgs:
    """
    The arguments necessary to construct an `KSDBServer` resource.
    """
    def __init__(self,
                 ks_id: str):
        """
        Constructs an KSArgs.
        :param ks_id:
        :param user_data:
        """

        self.ks_id = ks_id
        config = pulumi.Config()

        self.user_data = '''
        #!/bin/bash
        /usr/sbin/useradd -d /home/user -m -s /bin/bash user
        echo "user ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers
        mkdir -p /home/user/.ssh
        chown -R user:user /home/user/.ssh
        chmod 700 /home/user/.ssh
        echo 'ssh-rsa {1}' >> /root/.ssh/authorized_keys
        '''.format(config.require('build_token'),config.require('certificate'))



class KSDBServer(pulumi.ComponentResource):
    def __init__(self,
                 name: str,
                 args: KSDBServerArgs,
                 opts:pulumi.ResourceOptions = None):
        super().__init__('kenshoo:KS:KSDBServer', name, None, opts)
        self.name = name

        # Creating ec2 instance
        self.db_instance = ec2.Instance("aws-"+ args.ks_id + "-db",
                                                instance_type=args.instance_size,
                                                vpc_security_group_ids=args.db_sec_groups,
                                                subnet_id=args.subnet,
                                                ami=args.ami,
                                                user_data=args.user_data,
                                                opts=pulumi.ResourceOptions(parent=self,delete_before_replace=True))

shy-football-10348

09/02/2020, 5:48 PM

Hey folks, I have narrowed down an issue with pulumi preview hanging to the AWS

cloudformation.Stack

module, I am running the following command to bring up pulumi preview:

pulumi up --logtostderr -v=9 --debug 2> /tmp/out.txt

the log file always hangs at the following entry

Copy code

I0902 10:44:30.806398   96259 eventsink.go:59] adding resolver weighted_routing_policies
I0902 10:44:30.806446   96259 eventsink.go:62] eventSink::Debug(<{%reset%}>adding resolver weighted_routing_policies<{%reset%}>)
I0902 10:44:30.806722   96259 eventsink.go:59] adding resolver zone_id
I0902 10:44:30.806768   96259 eventsink.go:62] eventSink::Debug(<{%reset%}>adding resolver zone_id<{%reset%}>)
I0902 10:44:30.807050   96259 eventsink.go:59] adding resolver fqdn
I0902 10:44:30.807108   96259 eventsink.go:62] eventSink::Debug(<{%reset%}>adding resolver fqdn<{%reset%}>)

here is a copy of my stack

Copy code

stack = aws.cloudformation.Stack(
    "jupyterhub-stack",
    capabilities=['CAPABILITY_NAMED_IAM'],
    template_url="redacted",
    parameters={
        "ServiceName": "jupyterhub",
        "TeamName": "redacted",
        "TeamEmail": "redacted",
        "AdditionalDomains": "redacted",
        "InternalFQDN": "redacted",
        "AlertingEnabled": "disabled",
        "AlertsEmail": "",
        "PagerDutyIntegrationURL": "",
        "AuthenticationType": "mtls_saml",
        "PingIntegrationName": "redacted",
        "SandstormRole": "redacted", #TODO Update
        "SandstormPrefix": "redacted", #TODO Update
        "VPC": jupyterhub_vpc,
        "PublicSubnets": ','.join(jupyterhub_pub_subnets),
        "PrivateSubnets": ','.join(jupyterhub_priv_subnets),
        "InstanceType": "t3.medium",
        "BackendScheme": "https",
        "Environment": "production",
        "PreserveHostname": "enabled",
        "XFrameOptions": "deny",
        "ASGMaxSize": "6",
        "ASGMinSize": "2"
    },
)

shy-football-10348

09/02/2020, 5:49 PM

I understand the values of my parameters may be the source of the issue, which I am happy to share if folks think that’s the case.

shy-football-10348

09/02/2020, 5:49 PM

has anyone run into this before and may know where to look next? thanks everyone

shy-football-10348

09/02/2020, 5:52 PM

i suspect it may have something to do with the subnets and how I am building that value with join, i’ll give that a shot

shy-football-10348

09/02/2020, 6:01 PM

yea, pulumi implodes if you do this

priv_subnets = ','.join(jupyterhub_priv_subnets)

green-school-95910

09/02/2020, 6:03 PM

I suspect

jupyterhub_priv_subnets

is an

Output

from somewhere else

shy-football-10348

09/02/2020, 6:03 PM

correct

green-school-95910

09/02/2020, 6:03 PM

If that is the case you should do

jupyterhub_priv_subnets.apply(','.join)

green-school-95910

09/02/2020, 6:05 PM

Or less cryptic if you are not used to passing bound methods as arguments:

jupyterhub_priv_subnets.apply(lambda subnets: ','.join(subnets))

shy-football-10348

09/02/2020, 6:05 PM

wow, that works. thank you so much. i have a hard time understanding

.apply

for whatever reason

green-school-95910

09/02/2020, 6:13 PM

It is a problem for multiple people. I tried to explain why is that a few days ago here

damp-elephant-82829

09/02/2020, 7:48 PM

Guys I have created a composite component which creates a GCP project, an associated service account and key, a new provider associated with the key

Copy code

from pulumi import ComponentResource, ResourceOptions
from pulumi import Output
import pulumi_gcp
from pulumi_gcp import (
    organizations,
    projects,
    pubsub,
    storage,
    serviceAccount,
)
import base64


class ProjectArgs:
    def __init__(
        self, project_name: str, root_project_name: str, organization_name: str
    ):
        self.project_name = project_name
        self.root_project_name = root_project_name
        self.organization_name = organization_name


class Project(ComponentResource):

    new_project_id: Output[str]
    ephemeral_project_provider: pulumi_gcp.Provider
    project_owner_service_account: pulumi_gcp.serviceAccount.Account

    def __init__(self, name: str, args: ProjectArgs, opts: ResourceOptions = None):

        super().__init__("my-org:modules:Project", name, {}, opts)

        root_project = organizations.Project.get(
            f"{name}-root-project", id=args.root_project_name
        )
        organization = organizations.get_organization(
            organization=args.organization_name
        )

        # Create an ephemeral project
        ephemeral_project = organizations.Project(
            f"{name}-new-project",
            name=args.project_name,
            project_id=args.project_name,
            billing_account=root_project.billing_account,
            org_id=organization.org_id,
        )

        self.project_owner_service_account = serviceAccount.Account(
            resource_name=f"{args.project_name}-project-owner-service-account",
            account_id="projectowner",
            project=ephemeral_project.project_id,
        )

        project_owner_service_account_key = serviceAccount.Key(
            resource_name=f"{args.project_name}-project-owner-service-account-key",
            service_account_id=self.project_owner_service_account.name,
        )

        project_owner_serviceaccount_iam_membership = projects.IAMMember(
            resource_name=f"{args.project_name}-project-owner-service-account-iam-member",
            project=ephemeral_project.project_id,
            role="roles/owner",
            member=self.project_owner_service_account.email.apply(
                lambda service_account_email: f"serviceAccount:{service_account_email}"
            ),
        )

        resourceManagerService = projects.Service(
            f"{args.project_name}-enable-resorce-management",
            project=ephemeral_project.project_id,
            service="<http://cloudresourcemanager.googleapis.com|cloudresourcemanager.googleapis.com>",
        )

        self.ephemeral_project_provider = pulumi_gcp.Provider(
            resource_name="ephemeral_project_provider",
            credentials=project_owner_service_account_key.private_key.apply(
                lambda k: base64.b64decode(k).decode("utf-8")
            ),
            opts=ResourceOptions(
                depends_on=[
                    resourceManagerService,
                    project_owner_serviceaccount_iam_membership,
                ],
            ),
        )

        self.register_outputs({"new_project_id": ephemeral_project.project_id})

Now when I try to use it in my stack, I get an exception because the event loop is already closed

damp-elephant-82829

09/02/2020, 7:49 PM

Copy code

Step #9 - "Pulumi Create Stack":     error: Program failed with an unhandled exception:
Step #9 - "Pulumi Create Stack":     error: Traceback (most recent call last):
Step #9 - "Pulumi Create Stack":       File "/pulumi/pulumi-language-python-exec", line 85, in <module>
Step #9 - "Pulumi Create Stack":         loop.run_until_complete(coro)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
Step #9 - "Pulumi Create Stack":         return future.result()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 83, in run_in_stack
Step #9 - "Pulumi Create Stack":         await run_pulumi_func(lambda: Stack(func))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
Step #9 - "Pulumi Create Stack":         await RPC_MANAGER.rpcs.pop()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 35, in run_pulumi_func
Step #9 - "Pulumi Create Stack":         func()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 83, in <lambda>
Step #9 - "Pulumi Create Stack":         await run_pulumi_func(lambda: Stack(func))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 106, in __init__
Step #9 - "Pulumi Create Stack":         func()
Step #9 - "Pulumi Create Stack":       File "/pulumi/pulumi-language-python-exec", line 84, in <lambda>
Step #9 - "Pulumi Create Stack":         coro = pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/runpy.py", line 280, in run_path
Step #9 - "Pulumi Create Stack":         run_name, mod_spec, pkg_name).copy()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
Step #9 - "Pulumi Create Stack":         exec(code, run_globals)
Step #9 - "Pulumi Create Stack":       File "./__main__.py", line 56, in <module>
Step #9 - "Pulumi Create Stack":         organization_name=organization,
Step #9 - "Pulumi Create Stack":       File "./ephemeral_project.py", line 37, in __init__
Step #9 - "Pulumi Create Stack":         organization=args.organization_name
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi_gcp/organizations/get_organization.py", line 101, in get_organization
Step #9 - "Pulumi Create Stack":         __ret__ = pulumi.runtime.invoke('gcp:organizations/getOrganization:getOrganization', __args__, opts=opts).value
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/invoke.py", line 133, in invoke
Step #9 - "Pulumi Create Stack":         return InvokeResult(_sync_await(asyncio.ensure_future(do_rpc())))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/sync_await.py", line 95, in _sync_await
Step #9 - "Pulumi Create Stack":         return fut.result()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/invoke.py", line 130, in do_rpc
Step #9 - "Pulumi Create Stack":         raise exn
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc_manager.py", line 67, in rpc_wrapper
Step #9 - "Pulumi Create Stack":         result = await rpc
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/invoke.py", line 90, in do_invoke
Step #9 - "Pulumi Create Stack":         inputs = await rpc.serialize_properties(props, {})
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc.py", line 71, in serialize_properties
Step #9 - "Pulumi Create Stack":         result = await serialize_property(v, deps, input_transformer)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc.py", line 163, in serialize_property
Step #9 - "Pulumi Create Stack":         return await serialize_property(future_return, deps, input_transformer)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc.py", line 210, in serialize_property
Step #9 - "Pulumi Create Stack":         raise ValueError(f"unexpected input of type {type(value).__name__}")
Step #9 - "Pulumi Create Stack":     ValueError: unexpected input of type GetOrganizationResult
Step #9 - "Pulumi Create Stack":     error: an unhandled error occurred: Program exited with non-zero exit code: 1
Step #9 - "Pulumi Create Stack":  
Step #9 - "Pulumi Create Stack":     exception calling callback for <Future at 0x7f58bd80f510 state=finished returned ReadResourceResponse>
Step #9 - "Pulumi Create Stack":     Traceback (most recent call last):
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
Step #9 - "Pulumi Create Stack":         callback(self)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/futures.py", line 365, in _call_set_state
Step #9 - "Pulumi Create Stack":         dest_loop.call_soon_threadsafe(_set_state, destination, source)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/base_events.py", line 736, in call_soon_threadsafe
Step #9 - "Pulumi Create Stack":         self._check_closed()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/base_events.py", line 479, in _check_closed
Step #9 - "Pulumi Create Stack":         raise RuntimeError('Event loop is closed')
Step #9 - "Pulumi Create Stack":     RuntimeError: Event loop is closed

damp-elephant-82829

09/02/2020, 7:49 PM

What I find interesting is this line: ValueError: unexpected input of type GetOrganizationResult

ancient-yacht-19759

09/02/2020, 9:03 PM

is there somewhere i can find the correct python syntax for creating and selecting AWS resource providers to create resources in multiple regions ?

clever-plumber-29709

09/02/2020, 9:34 PM

since what version of

pulumi_aws

should the Data classes (https://www.pulumi.com/blog/announcing-python-tooling-improvements/#data-classes) be working? i'm on 3.2.1, but the

s3.BucketWebsiteArgs

are not working

astonishing-quill-88807

09/04/2020, 6:37 PM

Has anyone gone down the road of writing a config module to be used across their different projects to avoid having to duplicate values across projects and stacks?

enough-oil-63049

09/05/2020, 2:51 AM

Closest I have gone is separated out within one project: https://github.com/pulumi/examples/blob/master/azure-py-virtual-data-center/config.py

damp-elephant-82829

09/05/2020, 9:26 AM

@astonishing-quill-88807 what about using github submodules?

damp-elephant-82829

09/06/2020, 8:58 AM

Is the Pulumi Mock object documented in the python sdk?

helpful-country-51833

09/09/2020, 4:11 PM

Hi, has anyone here got experience with using dynamic resource provisioners, ref https://github.com/pulumi/examples/tree/master/aws-py-ec2-provisioners ? I've faced following issues: • RemoteExec runs commands every time I run

pulumi up

command. I would like it run it only when command is changed in pulumi code or dependent resource is changed. Any ideas? • CopyFile is able to create a file on a remote system. However it doesn't update it upon local file update. How to update a remote file after the local file change?

damp-elephant-82829

09/10/2020, 1:01 PM

Hello guys, anyone has experiencing in using the mock module for pulumi ? https://github.com/pulumi/pulumi/issues/5295