hello i have a chicken and egg scenario i have to create a d Pulumi Community #aws

hello, i have a chicken-and-egg scenario. i have ...

average-school-38756

03/29/2024, 2:05 AM

hello, i have a chicken-and-egg scenario. i have to create a dynamic number of corresponding SQS queues and Lambda functions, but another Lambda needs to reference the queues, and each queue policy needs to reference that Lambda. i typically would create the queues and functions in the same for-loop, but there is a circular dependency with he additional Lambda resource. i wish i could keep a mapping of resources to reference later, but Outputs don't seem to work that way. here is pseudo-code for what i'm trying to do:

Copy code

# this will be an arbitrary length
config = {"foo": ["foo1", "foo2"], "bar": ["bar1"]}

# create a Queue for each config key
for k in config.keys():
    aws.sqs.Queue()  # creates foo-queue and bar-queue


# create a Lambda which maps config values to queue ARNs
primary_fn = aws.lambda_.Function(
    environment_variables={
        "MY_VAR": json.dumps({  # imagine a method to dynamically construct this
            "foo1": <foo-queue-arn>, 
            "foo2": <foo-queue.arn>, 
            "bar1": <bar-queue.arn>
        })
    }
)

# create a Lambda to process each Queue
for k, v in config.keys():

    aws.sqs.QueuePolicy(
        # when k="foo", references foo-queue.arn
        # also references primary_fn.arn
    )

    f = aws.lambda_.Function()

    aws.lambda_.EventSourceMapping(
        event_source_arn=<k-queue.arn>,  # i.e. foo-queue-arn where k is "foo"
        function_name=f.name
    )

Any advice?

average-school-38756

03/29/2024, 6:00 PM

here's how i solved it, fwiw: first , i asked myself, if i were to spin all these resources up using the CLI, how would i do it? 1. create queues 2. create policies for queue functions 3. create queue functions for reading from queues 4. create event source mappings connecting queue functions to queues 5. create primary function with env vars 6. create a queue policy for each queue which allows primary function to publish to it So that left me with something like this:

Copy code

# this will be an arbitrary length
config = {"foo": ["foo1", "foo2"], "bar": ["bar1"]}

# create a Queue and Function for each config key

key_to_queue = {}

for k in config.keys():
    q = aws.sqs.Queue()  # creates foo-queue and bar-queue

    f = aws.lambda_.Function()

    aws.lambda_.EventSourceMapping(
        event_source_arn=q.arn,
        function_name=f.name
    )
    key_to_queue[k] = q


# create a Lambda which maps config values to queue ARNs
    
def construct_provider_env(*queue_arns):
    # do some matching, transforms, etc... the key is in the resource name and ARN.
    return {}


primary_fn = aws.lambda_.Function(
    environment_variables={
        "MY_VAR": pulumi.Output.all(
            *[x.arn for x in key_to_queue.values()] 
        ).apply(lambda args: json.dumps(construct_env(*args))),
    }
)

# create Queue policies

for key, queue in key_to_queue.items():

    aws.sqs.QueuePolicy(
        queue_url=queue.id,
        # policy references queue.arn and references primary_fn.arn
    )

Open in Slack

Previous Next