https://pulumi.com logo
Powered by
Title
s

shy-bird-55689

10/04/2021, 9:28 PM
hey ya, i'm trying to pass some resource outputs into a docker image build as build args, and keep on getting a json serialisation error as they are passing as outputs rather than the actual value. I've tried using 
apply
and 
Output.concat
anyone got any pointers? 
package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args={'BUCKET': "bucket_name",
                                                       'ACCESS_KEY_ID': package_dash_user_access_key.id,
                                                       'SECRET_ACCESS_KEY': package_dash_user_access_key.secret
                                                       },
                                                 context='source/services/package-dash/',
                                                 ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )
b

billowy-army-68599

10/04/2021, 9:29 PM
apply should work, what code did you try?
you'll need to use 
Output.all
and build the image args inside the apply
s

shy-bird-55689

10/04/2021, 9:30 PM
this is whar i tried 
package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args={'MARV_BUCKET': marv_bucket,
                                                       'ACCESS_KEY_ID': package_dash_user_access_key.id.apply(lambda key: f'{key}'),
                                                       'SECRET_ACCESS_KEY': package_dash_user_access_key.secret.apply(lambda secret: f'{secret}')
                                                       },
                                                 context='source/services/package-dash/',
                                                 ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )
b

billowy-army-68599

10/04/2021, 9:33 PM
you'll need to do the apply higher up, before the Image resource is created
something like: 
build_args = pulumi.Output.all(package_dash_user_access_key.id, package_dash_user_access_key.secret).apply(
lambda arg: f"ACCESS_KEY={arg[0]}. SECRET_ACCESS_KEY={arg[1]|
then pass 
build_args
to your DockerBuild that way
hopefully that makes sense
s

shy-bird-55689

10/04/2021, 9:39 PM
thanks @billowy-army-68599, will give it a go
Think i must be passing it in wrong @billowy-army-68599, still getting 
TypeError: Object of type Output is not JSON serializable
build_args = pulumi.Output.all(package_dash_user_access_key.id, package_dash_user_access_key.secret).apply(
        lambda arg: f"ACCESS_KEY={arg[0]}. SECRET_ACCESS_KEY={arg[1]}")

    package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args=build_args,
                                                 context='source/services/package-dash/',
                                                 ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )
b

billowy-army-68599

10/04/2021, 10:12 PM
i'll try get a repro this evening
s

shy-bird-55689

10/04/2021, 10:13 PM
Thanks @billowy-army-68599 much appreciated
g

great-sunset-355

10/05/2021, 12:37 PM
I use 
Output.from_input
for this. The trick is that simple outputs can be passed right away but templated strings need apply or concat 
Output.from_input(
            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "ListObjectsInBucket",
                        "Effect": "Allow",
                        "Action": ["s3:ListBucket"],
                        "Resource": [
                            stekz_tenant.s3_website.s3_bucket.arn,
                            maingau_frontend.s3_website.s3_bucket.arn,
                        ],
                    },
                    {
                        "Sid": "AllObjectActions",
                        "Effect": "Allow",
                        "Action": "s3:*Object*",
                        "Resource": [
                            Output.concat(stekz_tenant.s3_website.s3_bucket.arn, "/*"),
                            Output.concat(maingau_frontend.s3_website.s3_bucket.arn, "/*"),
                        ],
                    },
                ],
            }
        ).apply(json.dumps)
s

shy-bird-55689

10/05/2021, 2:04 PM
Thanks @great-sunset-355 Still no luck here though, thrrowing the same 
TypeError
g

great-sunset-355

10/05/2021, 2:11 PM
Which resource are you using? Are you sure that 
build_args
isn't an array? I've had a lot of these problems and it took me several days to wrap my head around them. So make sure you check the AWS API or Terraform docs on which data structure does it expect. eg: ECS has parameter 
container_definitions
and it's an array of JSON objects. so I built like a so: 
container_definition = {
            "name": config.container_name,  # is output Config.require
            "image": config.image_name,  # just str
            "portMappings": [{"containerPort": 80, "hostPort": 80, "protocol": "tcp"}],
            "environment": [{"name": k, "value": v} for k, v in config.env_vars.items()],  # config.env_vars is Dict[str,Any]
        }
        container_definitions = Output.from_input([container_definition]).apply(json.dumps)
s

shy-bird-55689

10/05/2021, 2:29 PM
I'm using the 
docker.Image
resource. i've been going by this example that passes at dict https://github.com/pulumi/pulumi-docker/blob/master/examples/aws-py/__main__.py
b

billowy-army-68599

10/05/2021, 5:45 PM
can you share your full code?
s

shy-bird-55689

10/05/2021, 6:01 PM
import base64
import json

import pulumi
from pulumi import Output, ResourceOptions
from pulumi_aws import ecr, ec2, iam, sqs
from pulumi_docker import ImageRegistry, Image, DockerBuild

from .config import env, region, current, marv_bucket


def get_registry_info(rid):
    creds = ecr.get_credentials(registry_id=rid)
    decoded = base64.b64decode(creds.authorization_token).decode()
    parts = decoded.split(':')
    if len(parts) != 2:
        raise Exception("Invalid credentials")
    return ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])


def package_dash(worker_vpc_subnet_01, worker_vpc_sg, scratch_efs, origin_bucket):
    package_dash_user = iam.User('package-dash-user',
                                 path='/',
                                 tags={
                                     "env": env,
                                     'product': 'marv'
                                 })

    package_dash_user_access_key = iam.AccessKey('package-dash-user-access-key', user=package_dash_user.name)

    package_dash_repository = ecr.Repository("package-dash-service")

    package_dash_image_name = package_dash_repository.repository_url
    package_dash_repository_info = package_dash_repository.registry_id.apply(get_registry_info)

    package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args={
                                   "ACCESS_KEY": package_dash_user_access_key.id.apply(lambda key: f'{key}'),
                                   "SECRET_ACCESS_KEY": package_dash_user_access_key.secret.apply(
                                       lambda secret: f'{secret}')
                               },
                                   context='source/services/package-dash/',
                               ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )

    package_dash_user_policy = iam.UserPolicy('package-dash-user-policy',
                                              user=package_dash_user.name,
                                              policy=json.dumps({
                                                  "Version": "2012-10-17",
                                                  "Statement": [
                                                      {
                                                          "Effect": "Allow",
                                                          "Action": [
                                                              "ecr:GetAuthorizationToken",
                                                              "ecr:BatchCheckLayerAvailability",
                                                              "ecr:GetDownloadUrlForLayer",
                                                              "ecr:GetRepositoryPolicy",
                                                              "ecr:DescribeRepositories",
                                                              "ecr:ListImages",
                                                              "ecr:DescribeImages",
                                                              "ecr:BatchGetImage",
                                                              "ecr:GetLifecyclePolicy",
                                                              "ecr:GetLifecyclePolicyPreview",
                                                              "ecr:ListTagsForResource",
                                                              "ecr:DescribeImageScanFindings"
                                                          ],
                                                          "Resource": "*"
                                                      },
                                                      {
                                                          "Action": [
                                                              "sqs:DeleteMessage",
                                                              "sqs:ReceiveMessage",
                                                              "sqs:GetQueueAttributes"
                                                          ],
                                                          "Resource": f'arn:aws:sqs:{region}:{current.account_id}:package-dash-queue*',
                                                          "Effect": "Allow"
                                                      },
                                                      {
                                                          "Effect": "Allow",
                                                          "Action": [
                                                              "s3:GetObject"
                                                          ],
                                                          "Resource": [f"arn:aws:s3:::{marv_bucket}/*"]
                                                      },
                                                      {
                                                          "Effect": "Allow",
                                                          "Action": [
                                                              "s3:PutObject",
                                                              "s3:GetObject"
                                                          ],
                                                          "Resource": [f"arn:aws:s3:::{origin_bucket.id}/*"]
                                                      }
                                                  ]
                                              })
                                              )

    package_dash_queue = sqs.Queue(
        "package-dash-queue",
        name='package-dash-queue',
        visibility_timeout_seconds=600,
        tags={
            "env": env,
            'product': 'marv'
        },
        opts=pulumi.ResourceOptions(delete_before_replace=True)
    )

    with open('infra/config/package_dash_cloud-init.yaml', 'r') as cfg:
        package_dash_instance_user_data_template = cfg.read()

    package_dash_instance_user_data = Output.concat(package_dash_instance_user_data_template.split('{-}')[0],
                                                    package_dash_user_access_key.id,
                                                    package_dash_instance_user_data_template.split('{-}')[1],
                                                    package_dash_user_access_key.secret,
                                                    package_dash_instance_user_data_template.split('{-}')[2],
                                                    scratch_efs.id,
                                                    package_dash_instance_user_data_template.split('{-}')[3],
                                                    region,
                                                    package_dash_instance_user_data_template.split('{-}')[4],
                                                    package_dash_repository.repository_url.apply(
                                                        lambda url: url.split('/')[0]),
                                                    package_dash_instance_user_data_template.split('{-}')[5],
                                                    package_dash_image.image_name,
                                                    package_dash_instance_user_data_template.split('{-}')[6],
                                                    package_dash_queue.url,
                                                    package_dash_instance_user_data_template.split('{-}')[7],
                                                    marv_bucket,
                                                    package_dash_instance_user_data_template.split('{-}')[8],
                                                    origin_bucket.id,
                                                    package_dash_instance_user_data_template.split('{-}')[9]
                                                    )

    package_dash_network_interface = ec2.NetworkInterface("package-dash-network-interface",
                                                          subnet_id=worker_vpc_subnet_01.id,
                                                          security_groups=[worker_vpc_sg.id],
                                                          tags={
                                                              "env": env,
                                                              'product': 'marv'
                                                          })

    package_dash_instance = ec2.Instance("package-dash-instance",
                                         ami='ami-0dbec48abfe298cab',
                                         instance_type='t3.micro',
                                         network_interfaces=[ec2.InstanceNetworkInterfaceArgs(
                                             network_interface_id=package_dash_network_interface.id,
                                             device_index=0
                                         )],
                                         root_block_device=ec2.InstanceRootBlockDeviceArgs(
                                             volume_size=15
                                         ),
                                         key_name=f'marv.{env}',
                                         user_data=package_dash_instance_user_data,
                                         opts=ResourceOptions(delete_before_replace=True))
2 Views
#pythonJoin Slack
Powered by