pulumi-community #python

python

worried-needle-99800

08/30/2020, 8:32 AM

I believe it’ll start from

__main__.py

worried-needle-99800

08/30/2020, 8:34 AM

Not 100% sure but I don’t think there’s any particular naming convention for multiple files

worried-needle-99800

08/30/2020, 8:36 AM

Most programs in the examples repo (https://github.com/pulumi/examples/) seem pretty basic, just putting Python files under the same directory as the

__main__.py

file

hallowed-beach-15050

08/31/2020, 3:40 AM

yeah,

__main__.py

is the way to go

astonishing-quill-88807

08/31/2020, 7:14 PM

I'm trying to use a dynamic provider and it's throwing an exception, but it doesn't show the full traceback. Is there a way to see the full Python traceback to understand where and why it's actually erroring?

hallowed-beach-15050

08/31/2020, 9:10 PM

@astonishing-quill-88807 what version of pulumi? and can you paste the error you are getting along with the code that you think may be generating the error?

hallowed-beach-15050

08/31/2020, 9:11 PM

inline or a gist would be awesome

astonishing-quill-88807

08/31/2020, 9:11 PM

I figured out the problem. I'm using v2.9.2 and my issue was trying to address the input properties as an attribute, rather than via dictionary access. Here's the code for posterity: https://github.com/mitodl/ol-infrastructure/blob/dagster_instance/src/ol_infrastructure/providers/salt/minion.py

astonishing-quill-88807

08/31/2020, 9:12 PM

It was throwing attribute errors about

salt_api_url

hallowed-beach-15050

08/31/2020, 9:47 PM

makes sense

damp-elephant-82829

09/02/2020, 6:36 AM

Guys when you write custom components in Python, does the framework offers standard logging facilities or do you use plain python logging?

damp-elephant-82829

09/02/2020, 7:24 AM

Awesome. One additional question related to ComponentResources: is it an anti pattern to give external access to child resources instead of registering their output?

strong-musician-98782

09/02/2020, 8:39 AM

Hey guys, i'm trying to create a class to create ec2 instance and then call it from my main but pulumi keeps thinking my userdata is changing even when its not, here's part of my class(tried to edit out as much as i can so some args might be missing):

import pulumi
from pulumi_aws import ec2

class KSDBServerArgs:
    """
    The arguments necessary to construct an `KSDBServer` resource.
    """
    def __init__(self,
                 ks_id: str):
        """
        Constructs an KSArgs.
        :param ks_id:
        :param user_data:
        """

        self.ks_id = ks_id
        config = pulumi.Config()

        self.user_data = '''
        #!/bin/bash
        /usr/sbin/useradd -d /home/user -m -s /bin/bash user
        echo "user ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers
        mkdir -p /home/user/.ssh
        chown -R user:user /home/user/.ssh
        chmod 700 /home/user/.ssh
        echo 'ssh-rsa {1}' >> /root/.ssh/authorized_keys
        '''.format(config.require('build_token'),config.require('certificate'))



class KSDBServer(pulumi.ComponentResource):
    def __init__(self,
                 name: str,
                 args: KSDBServerArgs,
                 opts:pulumi.ResourceOptions = None):
        super().__init__('kenshoo:KS:KSDBServer', name, None, opts)
        self.name = name

        # Creating ec2 instance
        self.db_instance = ec2.Instance("aws-"+ args.ks_id + "-db",
                                                instance_type=args.instance_size,
                                                vpc_security_group_ids=args.db_sec_groups,
                                                subnet_id=args.subnet,
                                                ami=args.ami,
                                                user_data=args.user_data,
                                                opts=pulumi.ResourceOptions(parent=self,delete_before_replace=True))

shy-football-10348

09/02/2020, 5:48 PM

Hey folks, I have narrowed down an issue with pulumi preview hanging to the AWS

cloudformation.Stack

module, I am running the following command to bring up pulumi preview:

pulumi up --logtostderr -v=9 --debug 2> /tmp/out.txt

the log file always hangs at the following entry

I0902 10:44:30.806398   96259 eventsink.go:59] adding resolver weighted_routing_policies
I0902 10:44:30.806446   96259 eventsink.go:62] eventSink::Debug(<{%reset%}>adding resolver weighted_routing_policies<{%reset%}>)
I0902 10:44:30.806722   96259 eventsink.go:59] adding resolver zone_id
I0902 10:44:30.806768   96259 eventsink.go:62] eventSink::Debug(<{%reset%}>adding resolver zone_id<{%reset%}>)
I0902 10:44:30.807050   96259 eventsink.go:59] adding resolver fqdn
I0902 10:44:30.807108   96259 eventsink.go:62] eventSink::Debug(<{%reset%}>adding resolver fqdn<{%reset%}>)

here is a copy of my stack

stack = aws.cloudformation.Stack(
    "jupyterhub-stack",
    capabilities=['CAPABILITY_NAMED_IAM'],
    template_url="redacted",
    parameters={
        "ServiceName": "jupyterhub",
        "TeamName": "redacted",
        "TeamEmail": "redacted",
        "AdditionalDomains": "redacted",
        "InternalFQDN": "redacted",
        "AlertingEnabled": "disabled",
        "AlertsEmail": "",
        "PagerDutyIntegrationURL": "",
        "AuthenticationType": "mtls_saml",
        "PingIntegrationName": "redacted",
        "SandstormRole": "redacted", #TODO Update
        "SandstormPrefix": "redacted", #TODO Update
        "VPC": jupyterhub_vpc,
        "PublicSubnets": ','.join(jupyterhub_pub_subnets),
        "PrivateSubnets": ','.join(jupyterhub_priv_subnets),
        "InstanceType": "t3.medium",
        "BackendScheme": "https",
        "Environment": "production",
        "PreserveHostname": "enabled",
        "XFrameOptions": "deny",
        "ASGMaxSize": "6",
        "ASGMinSize": "2"
    },
)

shy-football-10348

09/02/2020, 5:49 PM

I understand the values of my parameters may be the source of the issue, which I am happy to share if folks think that’s the case.

shy-football-10348

09/02/2020, 5:49 PM

has anyone run into this before and may know where to look next? thanks everyone

shy-football-10348

09/02/2020, 5:52 PM

i suspect it may have something to do with the subnets and how I am building that value with join, i’ll give that a shot

shy-football-10348

09/02/2020, 6:01 PM

yea, pulumi implodes if you do this

priv_subnets = ','.join(jupyterhub_priv_subnets)

green-school-95910

09/02/2020, 6:03 PM

I suspect

jupyterhub_priv_subnets

is an

Output

from somewhere else

shy-football-10348

09/02/2020, 6:03 PM

correct

green-school-95910

09/02/2020, 6:03 PM

If that is the case you should do

jupyterhub_priv_subnets.apply(','.join)

green-school-95910

09/02/2020, 6:05 PM

Or less cryptic if you are not used to passing bound methods as arguments:

jupyterhub_priv_subnets.apply(lambda subnets: ','.join(subnets))

shy-football-10348

09/02/2020, 6:05 PM

wow, that works. thank you so much. i have a hard time understanding

.apply

for whatever reason

green-school-95910

09/02/2020, 6:13 PM

It is a problem for multiple people. I tried to explain why is that a few days ago here

damp-elephant-82829

09/02/2020, 7:48 PM

Guys I have created a composite component which creates a GCP project, an associated service account and key, a new provider associated with the key

from pulumi import ComponentResource, ResourceOptions
from pulumi import Output
import pulumi_gcp
from pulumi_gcp import (
    organizations,
    projects,
    pubsub,
    storage,
    serviceAccount,
)
import base64


class ProjectArgs:
    def __init__(
        self, project_name: str, root_project_name: str, organization_name: str
    ):
        self.project_name = project_name
        self.root_project_name = root_project_name
        self.organization_name = organization_name


class Project(ComponentResource):

    new_project_id: Output[str]
    ephemeral_project_provider: pulumi_gcp.Provider
    project_owner_service_account: pulumi_gcp.serviceAccount.Account

    def __init__(self, name: str, args: ProjectArgs, opts: ResourceOptions = None):

        super().__init__("my-org:modules:Project", name, {}, opts)

        root_project = organizations.Project.get(
            f"{name}-root-project", id=args.root_project_name
        )
        organization = organizations.get_organization(
            organization=args.organization_name
        )

        # Create an ephemeral project
        ephemeral_project = organizations.Project(
            f"{name}-new-project",
            name=args.project_name,
            project_id=args.project_name,
            billing_account=root_project.billing_account,
            org_id=organization.org_id,
        )

        self.project_owner_service_account = serviceAccount.Account(
            resource_name=f"{args.project_name}-project-owner-service-account",
            account_id="projectowner",
            project=ephemeral_project.project_id,
        )

        project_owner_service_account_key = serviceAccount.Key(
            resource_name=f"{args.project_name}-project-owner-service-account-key",
            service_account_id=self.project_owner_service_account.name,
        )

        project_owner_serviceaccount_iam_membership = projects.IAMMember(
            resource_name=f"{args.project_name}-project-owner-service-account-iam-member",
            project=ephemeral_project.project_id,
            role="roles/owner",
            member=self.project_owner_service_account.email.apply(
                lambda service_account_email: f"serviceAccount:{service_account_email}"
            ),
        )

        resourceManagerService = projects.Service(
            f"{args.project_name}-enable-resorce-management",
            project=ephemeral_project.project_id,
            service="<http://cloudresourcemanager.googleapis.com|cloudresourcemanager.googleapis.com>",
        )

        self.ephemeral_project_provider = pulumi_gcp.Provider(
            resource_name="ephemeral_project_provider",
            credentials=project_owner_service_account_key.private_key.apply(
                lambda k: base64.b64decode(k).decode("utf-8")
            ),
            opts=ResourceOptions(
                depends_on=[
                    resourceManagerService,
                    project_owner_serviceaccount_iam_membership,
                ],
            ),
        )

        self.register_outputs({"new_project_id": ephemeral_project.project_id})

Now when I try to use it in my stack, I get an exception because the event loop is already closed

damp-elephant-82829

09/02/2020, 7:49 PM

Step #9 - "Pulumi Create Stack":     error: Program failed with an unhandled exception:
Step #9 - "Pulumi Create Stack":     error: Traceback (most recent call last):
Step #9 - "Pulumi Create Stack":       File "/pulumi/pulumi-language-python-exec", line 85, in <module>
Step #9 - "Pulumi Create Stack":         loop.run_until_complete(coro)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
Step #9 - "Pulumi Create Stack":         return future.result()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 83, in run_in_stack
Step #9 - "Pulumi Create Stack":         await run_pulumi_func(lambda: Stack(func))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
Step #9 - "Pulumi Create Stack":         await RPC_MANAGER.rpcs.pop()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 35, in run_pulumi_func
Step #9 - "Pulumi Create Stack":         func()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 83, in <lambda>
Step #9 - "Pulumi Create Stack":         await run_pulumi_func(lambda: Stack(func))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/stack.py", line 106, in __init__
Step #9 - "Pulumi Create Stack":         func()
Step #9 - "Pulumi Create Stack":       File "/pulumi/pulumi-language-python-exec", line 84, in <lambda>
Step #9 - "Pulumi Create Stack":         coro = pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/runpy.py", line 280, in run_path
Step #9 - "Pulumi Create Stack":         run_name, mod_spec, pkg_name).copy()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
Step #9 - "Pulumi Create Stack":         exec(code, run_globals)
Step #9 - "Pulumi Create Stack":       File "./__main__.py", line 56, in <module>
Step #9 - "Pulumi Create Stack":         organization_name=organization,
Step #9 - "Pulumi Create Stack":       File "./ephemeral_project.py", line 37, in __init__
Step #9 - "Pulumi Create Stack":         organization=args.organization_name
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi_gcp/organizations/get_organization.py", line 101, in get_organization
Step #9 - "Pulumi Create Stack":         __ret__ = pulumi.runtime.invoke('gcp:organizations/getOrganization:getOrganization', __args__, opts=opts).value
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/invoke.py", line 133, in invoke
Step #9 - "Pulumi Create Stack":         return InvokeResult(_sync_await(asyncio.ensure_future(do_rpc())))
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/sync_await.py", line 95, in _sync_await
Step #9 - "Pulumi Create Stack":         return fut.result()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/invoke.py", line 130, in do_rpc
Step #9 - "Pulumi Create Stack":         raise exn
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc_manager.py", line 67, in rpc_wrapper
Step #9 - "Pulumi Create Stack":         result = await rpc
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/invoke.py", line 90, in do_invoke
Step #9 - "Pulumi Create Stack":         inputs = await rpc.serialize_properties(props, {})
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc.py", line 71, in serialize_properties
Step #9 - "Pulumi Create Stack":         result = await serialize_property(v, deps, input_transformer)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc.py", line 163, in serialize_property
Step #9 - "Pulumi Create Stack":         return await serialize_property(future_return, deps, input_transformer)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/site-packages/pulumi/runtime/rpc.py", line 210, in serialize_property
Step #9 - "Pulumi Create Stack":         raise ValueError(f"unexpected input of type {type(value).__name__}")
Step #9 - "Pulumi Create Stack":     ValueError: unexpected input of type GetOrganizationResult
Step #9 - "Pulumi Create Stack":     error: an unhandled error occurred: Program exited with non-zero exit code: 1
Step #9 - "Pulumi Create Stack":  
Step #9 - "Pulumi Create Stack":     exception calling callback for <Future at 0x7f58bd80f510 state=finished returned ReadResourceResponse>
Step #9 - "Pulumi Create Stack":     Traceback (most recent call last):
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
Step #9 - "Pulumi Create Stack":         callback(self)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/futures.py", line 365, in _call_set_state
Step #9 - "Pulumi Create Stack":         dest_loop.call_soon_threadsafe(_set_state, destination, source)
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/base_events.py", line 736, in call_soon_threadsafe
Step #9 - "Pulumi Create Stack":         self._check_closed()
Step #9 - "Pulumi Create Stack":       File "/usr/local/lib/python3.7/asyncio/base_events.py", line 479, in _check_closed
Step #9 - "Pulumi Create Stack":         raise RuntimeError('Event loop is closed')
Step #9 - "Pulumi Create Stack":     RuntimeError: Event loop is closed

damp-elephant-82829

09/02/2020, 7:49 PM

message has been deleted

damp-elephant-82829

09/02/2020, 7:49 PM

What I find interesting is this line: ValueError: unexpected input of type GetOrganizationResult

ancient-yacht-19759

09/02/2020, 9:03 PM

is there somewhere i can find the correct python syntax for creating and selecting AWS resource providers to create resources in multiple regions ?

clever-plumber-29709

09/02/2020, 9:34 PM

since what version of

pulumi_aws

should the Data classes (https://www.pulumi.com/blog/announcing-python-tooling-improvements/#data-classes) be working? i'm on 3.2.1, but the

s3.BucketWebsiteArgs

are not working

Title

clever-plumber-29709

09/02/2020, 9:34 PM

since what version of

pulumi_aws

should the Data classes (https://www.pulumi.com/blog/announcing-python-tooling-improvements/#data-classes) be working? i'm on 3.2.1, but the

s3.BucketWebsiteArgs

are not working

gentle-diamond-70147

09/02/2020, 9:36 PM

When you say they are not working, what do you mean?

I believe they're available as of 3.2.0 - https://github.com/pulumi/pulumi-aws/blob/master/CHANGELOG.md#320-2020-08-24.

microscopic-pilot-97530

09/02/2020, 9:43 PM

Best results with the latest

pulumi

2.9.2 and

pulumi_aws

3.2.1 packages.

clever-plumber-29709

09/02/2020, 9:45 PM

i think my pulumi cli was not updated, not sure if that affects, i'm testing again

do i need to configure something on vs code? i installed mypyls and the extension

microscopic-pilot-97530

09/02/2020, 9:59 PM

I use VS Code with Pylance installed. https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance I also have Pylance’s basic type checking enabled via the VS code editor setting (it’s not perfect, but I’ve found it to be helpful):

"python.analysis.typeCheckingMode": "basic"

When I open a project, after the Python extension is finished loading (shown in the status bar of VS Code), I click the Python interpreter that it is using and set it to the

venv

for my project.

clever-plumber-29709

09/02/2020, 10:14 PM

yes. i saw the post and installed pylance and set it as default. also the interpreter is correctly set to the venv. but when typing i dont see the box with the types indication (image 1) And i see an error that BucketWebsiteArgs is not member of s3 module (image 2)

microscopic-pilot-97530

09/02/2020, 10:19 PM

You’re sure you have

pulumi_aws

3.2.1 installed in the venv? I don’t get any red squiggles and this is what I see when hovering over

BucketWebsiteArgs

What does

venv/bin/python -m pip list

show?

This is for the project I’m currently working in:

Package    Version
---------- -------
Arpeggio   1.9.2
attrs      20.1.0
dill       0.3.2
grpcio     1.31.0
parver     0.3.0
pip        20.2.2
protobuf   3.13.0
pulumi     2.9.2
pulumi-aws 3.2.1
semver     2.10.2
setuptools 50.0.3
six        1.15.0
wheel      0.35.1

clever-plumber-29709

09/02/2020, 11:05 PM

versions are fine on my project. i switched to a newly created project just to check, and now I dont see the previous error. but i see a new one related to mypy (image 1) hovering BucketWebsiteArgs says unknown (image 2)

View count: 1