https://pulumi.com logo
#python
Title
# python
s

shy-bird-55689

10/04/2021, 9:28 PM
hey ya, i'm trying to pass some resource outputs into a docker image build as build args, and keep on getting a json serialisation error as they are passing as outputs rather than the actual value. I've tried using
apply
and
Output.concat
anyone got any pointers?
Copy code
package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args={'BUCKET': "bucket_name",
                                                       'ACCESS_KEY_ID': package_dash_user_access_key.id,
                                                       'SECRET_ACCESS_KEY': package_dash_user_access_key.secret
                                                       },
                                                 context='source/services/package-dash/',
                                                 ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )
b

billowy-army-68599

10/04/2021, 9:29 PM
apply should work, what code did you try?
you'll need to use
Output.all
and build the image args inside the apply
s

shy-bird-55689

10/04/2021, 9:30 PM
this is whar i tried
Copy code
package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args={'MARV_BUCKET': marv_bucket,
                                                       'ACCESS_KEY_ID': package_dash_user_access_key.id.apply(lambda key: f'{key}'),
                                                       'SECRET_ACCESS_KEY': package_dash_user_access_key.secret.apply(lambda secret: f'{secret}')
                                                       },
                                                 context='source/services/package-dash/',
                                                 ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )
b

billowy-army-68599

10/04/2021, 9:33 PM
you'll need to do the apply higher up, before the Image resource is created
something like:
Copy code
build_args = pulumi.Output.all(package_dash_user_access_key.id, package_dash_user_access_key.secret).apply(
lambda arg: f"ACCESS_KEY={arg[0]}. SECRET_ACCESS_KEY={arg[1]|
then pass
build_args
to your DockerBuild that way
hopefully that makes sense
s

shy-bird-55689

10/04/2021, 9:39 PM
thanks @billowy-army-68599, will give it a go
Think i must be passing it in wrong @billowy-army-68599, still getting
TypeError: Object of type Output is not JSON serializable
Copy code
build_args = pulumi.Output.all(package_dash_user_access_key.id, package_dash_user_access_key.secret).apply(
        lambda arg: f"ACCESS_KEY={arg[0]}. SECRET_ACCESS_KEY={arg[1]}")

    package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args=build_args,
                                                 context='source/services/package-dash/',
                                                 ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )
b

billowy-army-68599

10/04/2021, 10:12 PM
i'll try get a repro this evening
s

shy-bird-55689

10/04/2021, 10:13 PM
Thanks @billowy-army-68599 much appreciated
g

great-sunset-355

10/05/2021, 12:37 PM
I use
Output.from_input
for this. The trick is that simple outputs can be passed right away but templated strings need apply or concat
Copy code
Output.from_input(
            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "ListObjectsInBucket",
                        "Effect": "Allow",
                        "Action": ["s3:ListBucket"],
                        "Resource": [
                            stekz_tenant.s3_website.s3_bucket.arn,
                            maingau_frontend.s3_website.s3_bucket.arn,
                        ],
                    },
                    {
                        "Sid": "AllObjectActions",
                        "Effect": "Allow",
                        "Action": "s3:*Object*",
                        "Resource": [
                            Output.concat(stekz_tenant.s3_website.s3_bucket.arn, "/*"),
                            Output.concat(maingau_frontend.s3_website.s3_bucket.arn, "/*"),
                        ],
                    },
                ],
            }
        ).apply(json.dumps)
s

shy-bird-55689

10/05/2021, 2:04 PM
Thanks @great-sunset-355 Still no luck here though, thrrowing the same
TypeError
g

great-sunset-355

10/05/2021, 2:11 PM
Which resource are you using? Are you sure that
build_args
isn't an array? I've had a lot of these problems and it took me several days to wrap my head around them. So make sure you check the AWS API or Terraform docs on which data structure does it expect. eg: ECS has parameter
container_definitions
and it's an array of JSON objects. so I built like a so:
Copy code
container_definition = {
            "name": config.container_name,  # is output Config.require
            "image": config.image_name,  # just str
            "portMappings": [{"containerPort": 80, "hostPort": 80, "protocol": "tcp"}],
            "environment": [{"name": k, "value": v} for k, v in config.env_vars.items()],  # config.env_vars is Dict[str,Any]
        }
        container_definitions = Output.from_input([container_definition]).apply(json.dumps)
s

shy-bird-55689

10/05/2021, 2:29 PM
I'm using the
docker.Image
resource. i've been going by this example that passes at dict https://github.com/pulumi/pulumi-docker/blob/master/examples/aws-py/__main__.py
b

billowy-army-68599

10/05/2021, 5:45 PM
can you share your full code?
s

shy-bird-55689

10/05/2021, 6:01 PM
Copy code
import base64
import json

import pulumi
from pulumi import Output, ResourceOptions
from pulumi_aws import ecr, ec2, iam, sqs
from pulumi_docker import ImageRegistry, Image, DockerBuild

from .config import env, region, current, marv_bucket


def get_registry_info(rid):
    creds = ecr.get_credentials(registry_id=rid)
    decoded = base64.b64decode(creds.authorization_token).decode()
    parts = decoded.split(':')
    if len(parts) != 2:
        raise Exception("Invalid credentials")
    return ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])


def package_dash(worker_vpc_subnet_01, worker_vpc_sg, scratch_efs, origin_bucket):
    package_dash_user = iam.User('package-dash-user',
                                 path='/',
                                 tags={
                                     "env": env,
                                     'product': 'marv'
                                 })

    package_dash_user_access_key = iam.AccessKey('package-dash-user-access-key', user=package_dash_user.name)

    package_dash_repository = ecr.Repository("package-dash-service")

    package_dash_image_name = package_dash_repository.repository_url
    package_dash_repository_info = package_dash_repository.registry_id.apply(get_registry_info)

    package_dash_image = Image('package-dash-image',
                               build=DockerBuild(args={
                                   "ACCESS_KEY": package_dash_user_access_key.id.apply(lambda key: f'{key}'),
                                   "SECRET_ACCESS_KEY": package_dash_user_access_key.secret.apply(
                                       lambda secret: f'{secret}')
                               },
                                   context='source/services/package-dash/',
                               ),
                               image_name=package_dash_image_name,
                               registry=package_dash_repository_info,
                               )

    package_dash_user_policy = iam.UserPolicy('package-dash-user-policy',
                                              user=package_dash_user.name,
                                              policy=json.dumps({
                                                  "Version": "2012-10-17",
                                                  "Statement": [
                                                      {
                                                          "Effect": "Allow",
                                                          "Action": [
                                                              "ecr:GetAuthorizationToken",
                                                              "ecr:BatchCheckLayerAvailability",
                                                              "ecr:GetDownloadUrlForLayer",
                                                              "ecr:GetRepositoryPolicy",
                                                              "ecr:DescribeRepositories",
                                                              "ecr:ListImages",
                                                              "ecr:DescribeImages",
                                                              "ecr:BatchGetImage",
                                                              "ecr:GetLifecyclePolicy",
                                                              "ecr:GetLifecyclePolicyPreview",
                                                              "ecr:ListTagsForResource",
                                                              "ecr:DescribeImageScanFindings"
                                                          ],
                                                          "Resource": "*"
                                                      },
                                                      {
                                                          "Action": [
                                                              "sqs:DeleteMessage",
                                                              "sqs:ReceiveMessage",
                                                              "sqs:GetQueueAttributes"
                                                          ],
                                                          "Resource": f'arn:aws:sqs:{region}:{current.account_id}:package-dash-queue*',
                                                          "Effect": "Allow"
                                                      },
                                                      {
                                                          "Effect": "Allow",
                                                          "Action": [
                                                              "s3:GetObject"
                                                          ],
                                                          "Resource": [f"arn:aws:s3:::{marv_bucket}/*"]
                                                      },
                                                      {
                                                          "Effect": "Allow",
                                                          "Action": [
                                                              "s3:PutObject",
                                                              "s3:GetObject"
                                                          ],
                                                          "Resource": [f"arn:aws:s3:::{origin_bucket.id}/*"]
                                                      }
                                                  ]
                                              })
                                              )

    package_dash_queue = sqs.Queue(
        "package-dash-queue",
        name='package-dash-queue',
        visibility_timeout_seconds=600,
        tags={
            "env": env,
            'product': 'marv'
        },
        opts=pulumi.ResourceOptions(delete_before_replace=True)
    )

    with open('infra/config/package_dash_cloud-init.yaml', 'r') as cfg:
        package_dash_instance_user_data_template = cfg.read()

    package_dash_instance_user_data = Output.concat(package_dash_instance_user_data_template.split('{-}')[0],
                                                    package_dash_user_access_key.id,
                                                    package_dash_instance_user_data_template.split('{-}')[1],
                                                    package_dash_user_access_key.secret,
                                                    package_dash_instance_user_data_template.split('{-}')[2],
                                                    scratch_efs.id,
                                                    package_dash_instance_user_data_template.split('{-}')[3],
                                                    region,
                                                    package_dash_instance_user_data_template.split('{-}')[4],
                                                    package_dash_repository.repository_url.apply(
                                                        lambda url: url.split('/')[0]),
                                                    package_dash_instance_user_data_template.split('{-}')[5],
                                                    package_dash_image.image_name,
                                                    package_dash_instance_user_data_template.split('{-}')[6],
                                                    package_dash_queue.url,
                                                    package_dash_instance_user_data_template.split('{-}')[7],
                                                    marv_bucket,
                                                    package_dash_instance_user_data_template.split('{-}')[8],
                                                    origin_bucket.id,
                                                    package_dash_instance_user_data_template.split('{-}')[9]
                                                    )

    package_dash_network_interface = ec2.NetworkInterface("package-dash-network-interface",
                                                          subnet_id=worker_vpc_subnet_01.id,
                                                          security_groups=[worker_vpc_sg.id],
                                                          tags={
                                                              "env": env,
                                                              'product': 'marv'
                                                          })

    package_dash_instance = ec2.Instance("package-dash-instance",
                                         ami='ami-0dbec48abfe298cab',
                                         instance_type='t3.micro',
                                         network_interfaces=[ec2.InstanceNetworkInterfaceArgs(
                                             network_interface_id=package_dash_network_interface.id,
                                             device_index=0
                                         )],
                                         root_block_device=ec2.InstanceRootBlockDeviceArgs(
                                             volume_size=15
                                         ),
                                         key_name=f'marv.{env}',
                                         user_data=package_dash_instance_user_data,
                                         opts=ResourceOptions(delete_before_replace=True))
2 Views