https://pulumi.com logo
Powered by
Title
b

better-actor-92669

02/24/2020, 9:15 AM
@here, hey guys. I need to access an Output object's value as type 
str
and not 
Output
. I know that it is asynchronous operation and the value may not be available right away, so I wrap it in a function as @white-balloon-205 recommended. For instance, I need to set up environmental variables to connect to a CloudSQL Postge Instance. I created a function, that does that, and then I want to pass an Output object as a string, not as an Output type: 
import os
from pulumi_gcp.sql import DatabaseInstance, SslCert, User
from pulumi import Config, export, get_project, ResourceOptions, Output

def set_env_vars(env_var_name, env_var_value):
    os.environ[env_var_name] = env_var_value

service_user_certificate = SslCert(
        'service_user_certificate',
        common_name=service_user_name,
        opts=ResourceOptions(
            depends_on=[cloud_pgsql_main_1, service_user],
            protect=False,
        ),
        instance=cloud_pgsql_main_1.name,
    )

    # Creating a Database Provider
    Output.all(['PGSSLCERT', service_user_certificate.cert]).apply(set_env_vars)
    Output.all(['PGSSLKEY', service_user_certificate.private_key]).\
        apply(set_env_vars)
    Output.all(['PGSSLROOTCERT', service_user_certificate.server_ca_cert]).\
        apply(set_env_vars)

    # set_env_vars('PGSSLCERT', Output.all([service_user_certificate.cert]).apply(lambda l: f"{l}"))
    # set_env_vars('PGSSLKEY', str(service_user_certificate.private_key))
    # set_env_vars('PGSSLROOTCERT', str(service_user_certificate.server_ca_cert))

    print(os.environ['PGSSLCERT'])
    print(os.environ['PGSSLKEY'])
    print(os.environ['PGSSLROOTCERT'])
I tried different methods, however, environmental variables are not set up, because 
os.environ
can't find a key as it is not set up. Nevertheless, I can obviously check output of 
service_user_certificate.cert
via 
pulumi.export()
. Can you please help me figure out how to get a string value of an Output object?
g

gentle-diamond-70147

02/24/2020, 4:35 PM
Due to the async nature of Outputs, I don't think you can set and then use environment variables like this.
You could potentially use the set environment variables from within the .apply() after you immediately set them. But setting them in the apply and then attempting to use them outside won't work I believe.
👍 1
How are you wanting to use those values once they are available? In some custom code to connect to the database?
b

better-actor-92669

02/24/2020, 4:59 PM
It is for connecting to a GCP CloudSQL Instance via pulumi-postgresql module and specifically the Provider class: 
postgre_provider = pulumi_postgresql.Provider(
        'cloud_pgsql_main_1_provider',
        host=cloud_pgsql_main_1.public_ip_address,
        opts=ResourceOptions(
            depends_on=[cloud_pgsql_main_1, service_user,
                        service_user_certificate],
        ),
        username=service_user.name,
        password=service_user.password,
        port=5432,
        sslmode='verify-ca',
        expected_version='11',
    )

    for role in ROLES_TO_CREATE:
        role_object = pulumi_postgresql.Role(
            'postge_user_' + role.replace('-', '_'),
            login=True,
            name=role,
            opts=ResourceOptions(
                depends_on=[cloud_pgsql_main_1, service_user,
                            service_user_certificate, postgre_provider],
                protect=False,
                provider=postgre_provider,
            ),
            password=CLOUDSQL_POSTGRE_APPLICATION_USERS_AND_PASSWORDS.get(
                role,
                password_generator.random_password_gen(to_base64_string=False)),
        )
Since I want to use HTTPS, I need to provide environmental variables for that, as it uses the libpq-ssl https://www.postgresql.org/docs/current/libpq-ssl.html
https://github.com/pulumi/pulumi-postgresql
m

many-garden-84306

02/24/2020, 9:00 PM
Modifying the OS environment in a concurrent/asynchronous process is inherently risky since os.environ is a singleton across threads/tasks, and multiple tasks may be competing to change it. IMO you should capture the environment variables you need at startup (before overlapping/asynchronous activity begins), then pass the captured or computed values to your async tasks, then explicitly pass them down to operations as variables rather than through the environment. If the library you are using requires per-instance values to be passed through environment variables, it may not be safe to use it within an asynchronous task, unless appropriate locks are placed around updating and using the os environment. One thing that is safe to do is to create a new dict representing the environment you wish to run your operation in, then spawn a child process (with subprocess.check_call, etc) passing the new dict to the child process with 
env=
. You could do that within apply()
In the case of libpq-ssl, all of the parameters that can be passed as environment variables can also be passed as connection parameters https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS . However, the terraform provider for postgresql (from which the pulumi provider is derived) does not seem to expose these parameters, which is why you are stuck passing environment variables around. One solution that may or may not work for you is to create a Connection Service File that contains all of the parameter settings, then create the connection using the logical service name: https://www.postgresql.org/docs/current/libpq-pgservice.html
b

better-actor-92669

02/25/2020, 9:20 AM
Hello, @many-garden-84306. Thanks a lot for the explanation! "However, the terraform provider for postgresql (from which the pulumi provider is derived) does not seem to expose these parameters" - exactly, there is no other way. In addition, I don't know if connection service file is going to solve the deal, as I create a SslCert object during the same runtime. Nevertheless, I will definitely take a look at that. Thank you once again. Yeah, I agree that using env variables is not the best approach, but it is better to use an encrypted connection to the instance to create the users in 🙂 and there were no other option as far as I understood
@many-garden-84306, @white-balloon-205 I don't know why, but
import os
import pulumi
import network.vpc
from cloud_api import apis
import cloud_sql.postgre_sql
# from cloud_sql import postgre_sql
from service_scripts.mandrill_send import send_message


# Variables for FortiGate Configuration
def pulumi_output_renderer_fortigate(args_to_render) -> None:
    print('I am called, 0000000000000')
    print(args_to_render)

    from service_scripts.template_files import template_config

    jinja2_variables_dict_fortigate = {
        'phase1_interface_1_name': 'vpn-to-pulumi-1',
        'phase1_interface_2_name': 'vpn-to-pulumi-2',
        'phase1_interface_1_comment': 'GCP HA-VPN Interface0',
        'phase1_interface_2_comment': 'GCP HA-VPN Interface1',
        'ha_vpn_if_0_ip': args_to_render[0],
        'ha_vpn_if_1_ip': args_to_render[1],
        'fgt_vpn_if_1_ip': network.vpc.hq_bgp_peer_interface_1_ip,
        'fgt_vpn_if_2_ip': network.vpc.hq_bgp_peer_interface_2_ip,
        'neighbor_ip_1':
            network.vpc.vpn_router_to_hq_1_interface_1_ip.split('/', 1)[0],
        'neighbor_ip_2':
            network.vpc.vpn_router_to_hq_2_interface_1_ip.split('/', 1)[0],
        'remote_asn_1': network.vpc.vpn_router_to_hq_1_bgp_asn,
        'remote_asn_2': network.vpc.vpn_router_to_hq_2_bgp_asn,
    }

    template_config(
        './out_files/fgt_vpn_config.conf',
        './fortigate_vpn_config_template/fgt_vpn_config.j2',
        jinja2_variables_dict_fortigate,
        render_in_place=False,
        write_to_stdout=False,
    )


# libpq_config variables for database connections
def pulumi_output_renderer_postgre(args_to_render) -> None:
    print('I am called, 111111111111')
    print(args_to_render)

    from service_scripts.template_files import template_config

    # jinja2_variables_dict_postgre = {
    #     'postgre_host': args_to_render[0],
    #     'postgre_port': '5432',
    #     'postgre_service_user': args_to_render[1],
    #     'postgre_service_user_password': args_to_render[2],
    #     'postgre_sslmode': 'verify-ca',
    #     'postgre_sslcert': args_to_render[3],
    #     'postgre_sslkey': args_to_render[4],
    #     'postgre_sslrootcert': args_to_render[5],
    # }

    jinja2_variables_dict_postgre = {
        'postgre_host': '1',
        'postgre_port': '5432',
        'postgre_service_user': '2',
        'postgre_service_user_password': '23',
        'postgre_sslmode': 'verify-ca',
        'postgre_sslcert': '5',
        'postgre_sslkey': '66',
        'postgre_sslrootcert': '22',
    }

    template_config(
        f'{os.environ["HOME"]}/.pg_service.conf',
        './postgre_templates/libpq_config.j2',
        jinja2_variables_dict_postgre,
        render_in_place=False,
        write_to_stdout=False,
    )


def apply_and_send(instance_name: str, data_to_send: dict):

    from service_scripts.template_files import template_config

    # Templating Jinja2 Template for Postgre CloudSQL users
    cloudsql_postgre_users_out_file = './out_files/postgre_users_data.txt'

    template_config(
        cloudsql_postgre_users_out_file,
        './service_objects/password_email_template.j2',
        {
            'postgre_users': data_to_send
        },
        render_in_place=False,
        write_to_stdout=False,
    )

    send_message(
        # Send a message via Mandrill API
        f"Postgre CloudSQL users for "
        f"'{instance_name}'"
        f" instance in "
        f"'{pulumi.get_project()}' project",
        None,
        attachment_present=True,
        attachments=[
            {'content': cloudsql_postgre_users_out_file,
             'name': 'postgre_cloudsql_users.html',
             'type': 'text/html',
             },
        ],
    )


pulumi.Output.all(
    network.vpc.ha_vpn_gw_hq.vpn_interfaces[0]['ip_address'],
    network.vpc.ha_vpn_gw_hq.vpn_interfaces[1]['ip_address'],
).apply(pulumi_output_renderer_fortigate)

pulumi.Output.all(
    cloud_sql.postgre_sql.cloud_pgsql_main_1.ip_addresses,
    cloud_sql.postgre_sql.service_user.name,
    cloud_sql.postgre_sql.service_user.password,
    cloud_sql.postgre_sql.service_user_certificate.cert,
    cloud_sql.postgre_sql.service_user_certificate.private_key,
    cloud_sql.postgre_sql.service_user_certificate.server_ca_cert
).apply(pulumi_output_renderer_postgre)
first 
pulumi.Output.all
works perfectly, but second one doesn't. It seems like the values from 
cloud_sql.postgre_sql
cannot be parsed. When I change the second output to ` 
# pulumi.Output.all(
#     network.vpc.ha_vpn_gw_hq.vpn_interfaces[0]['ip_address'],
# ).apply(pulumi_output_renderer_postgre)
it works, so once again, I think that the values from the classes 
from pulumi_gcp.sql import DatabaseInstance, SslCert, User
cannot be derived. Please help 🙂
#pythonJoin Slack
Powered by