Has anyone created a dynamic provider that utilizes a

boto3

client to execute a task? I may be going about this the wrong way (long time SWE, been doing DevOps for a bit under a year though) so I welcome any feedback. I am building a CI/CD pipeline in AWS CodeCommit, CodeBuild, and CodePipeline. CodePipeline requires the use of an S3 bucket to hold the artifacts (i.e. the code associated with a PR that is being merged into Main). The problem is that there is no “clean up” of that bucket and a standard lifecycle policy on the bucket is rather coarse, lacking fine-grained controls. I need to keep the last N artifacts (it varies depending on the team) and clear out the rest. My choices are to do this via a Lambda or via the CI/CD process itself - I’m hoping to do it as part of the pipeline itself and not use a lambda. Ordinarily, I would just write the required functions, using

boto3

to verify the bucket exists, get the list of objects ordered by created date, and delete all but the last N. My concern is that this seems like a workaround when all other actions use Pulumi resources/objects to do the work. When i found out about

Dynamic Providers

, it seemed like the more correct way to integrate

boto3

. So far I have been unsuccessful. I haven’t been able to even just return the results of a

s3.head_bucket

command. I am not a strong python programmer (started using python when I moved to DevOps since it is the company standard), so I could be doing things wrong - but I can’t figure out how to see what the dynamic provider is doing. I’ve tried logging (log files are blank), even adding an

atexit.register(flush_logs)

to my code. I know the dynamic provider runs in a subprocess, but most of what I’ve googled all point to the same way of adding a logger to a subprocess (at the top of the file, add the standard logger = logging.getLogger… type boilerplate and a fileHandler if desired). I’m happy to add some code to this if it helps, but I’m hoping someone can point me to an example or something other than the Pulumi doc examples. I’ve done several Pulumi projects - basics like create an S3 statebucket, deploy EKS, Helm charts, EC2, IAM, ECR - but the dynamic provider just isn’t clicking for me.

have you looked at the

Command

provider? I use that all the time to run simple

aws cli

commands during stack execution https://www.pulumi.com/registry/packages/command/api-docs/local/command/

@wonderful-umbrella-73154 I have not looked at it, and more than happy to check it out. Any good AWS CLI examples you could point to? Can you also provide an example of getting the stdout/stderr as Outputs? I’ve don’t recall running across that before.

aws s3api head-bucket --bucket mybucketname

could be one to start playing with

Oh! So replace boto3 with the corresponding AWS CLI command?

if you're somewhat new to Pulumi though, you should definitely make sure you understand the concept of

Outputs

and all the async stuff going on so that you can reference them correctly

New-ish to pulumi. I’m functionally literate? I know how to (mostly) use

.apply(lambda...)

and

pulumi.Output.all(something).apply(lambda…)

Async in python is still non-sensical to me (it was so much easier to implement in C#).

it's definitely a hack---but if you only need something simple, it might be easier than figuring out the best way to use dynamic providers (and other custom things like that)