https://pulumi.com
Join Slack
Hi, I’m very new to Pulumi. I have a python program that uses boto3 to launch things in AWS. In the ...
# python
g
Hi, I’m very new to Pulumi. I have a python program that uses boto3 to launch things in AWS. In the interest of idempotency, I thought I’d replace all my query/test and try/except call/wait stuff with Pulumi, but I don’t have a notional stack here — I just want to make API calls ad hoc, using Pulumi as an assertion macro rather than doing it all in detail. Is that really a supported use case or am I barking up the wrong forest?
This is a general question but I’m asking in #CDE799L1M because I tried it and it looks like it would like to work, but then I get unexpected monitor.Invoke…NoneType has no attribute ‘Invoke’ errors, so it quickly becomes a Python question.
g
with this is pretty much impossible to answer, please share a gist of the code. It sounds like you are using pulumi wrong because you may lack knowledge on how does it work. It is not a typical python program. See: https://www.pulumi.com/docs/languages-sdks/python/#blocking-and-asynchronous-code https://github.com/pulumi/pulumi/issues/12172 To create conditional resources you need to provide synchronous conditions beforehand. If you/your team can use different languages, my advice is to stay away from python and use TypeScript or Go, perhaps .Net.
g
OK - so basically it should be possible to drive operations from a program without a 
Pulumi.yaml
and a 
pulumi up
command? I’m mainly just asking whether this is solvable. As to language, this is a Python program. Pulumi isn’t the goal, it’s just a means to do some work within an existing Python framework.
g
I see, in this case you need 
pulumi up
and 
Pulumi.yaml
(well kinda). Take a look at the 
automation api
- it is meant for programmatic control over the pulumi runtime. You should find what you need even though some parity differences between languages may exist. https://www.pulumi.com/docs/using-pulumi/automation-api/getting-started-automation-api/ https://www.pulumi.com/docs/using-pulumi/automation-api/concepts-terminology/ https://github.com/pulumi/automation-api-examples
g
For code sample, I don’t even need a gist. This fails:
Copy code
import pulumi
from pulumi_aws import get_caller_identity
pulumi.runtime.set_config('aws:profile', 'control')
caller_identity = get_caller_identity()
OK. Well if I need pulumi.yaml and a pulumi command then I’m using the wrong tool.
I think I get the idea of the automation API — thank you. I found that previously and was hoping I didn’t need to do that either. I just want to drive actions against the AWS API, and not define a platform state.
So the idea of a stack, even if I’m invoking it from code, feels like not what I need
(We do use pulumi for some other things that are more stateful though, and I think auto api will help with that — so this is very helpful information!)
g
I see. You mentioned some assertions. Are you trying to verify what your 
boto
code did? Pulumi can help you deploy resources, same as cloudformation, but much faster because Pulumi is defacto wrapper around terraform which uses AWS API. I am usually trying to avoid programmatic control over AWS resources unless you can store the state somewhere. By 
state
I do not mean pulumi state but rather references to what is running. Eg. keep resource ARNs in DynamoDB. Then on top of that, you can build assertions. With lambda or similar things.
g
Oh, sorry, not that kind of assertions 🙂 I guess I mean declarations of the end goal. But once the end is achieved, we’re done. We’re not going to maintain the state in a state repository, calculate deltas and apply differences, etc. So yes, more like for a single-use deployment.
My boto code is all OK, it does what I need. But I don’t enjoy that I need to maintain code to check whether a role exists, and if not, create it, and so on, before deploying the objects that use the role. So yes, I do need state — but only for the lifetime of the deployment. Other actors could influence the runtime, so if we ever run again we need to rediscover current state and adapt to that. Pulumi is not authoritative.
I want partial updates. So e.g. if I’m deploying a lambda, and I say in my configuration that I want the max runtime to be 30s, it should make that change. If I don’t say what the memory size should be, and the lambda already exists, it should update the runtime but not make any changes to memory.
It’s a contrived example but hope that explains the reason I’m trying to avoid persistent state and authority over the outcome
Got to run for now. Thank you for the help — if you have other thoughts or questions I’ll be happy to catch up later!
g
I am working on a quite drastic refactor of our pulumi codebase and I'll need to import a large amount of resources. You can import resources into the state via code as well: https://www.pulumi.com/docs/using-pulumi/adopting-pulumi/import/#importing-resources-in-code I am doing this in TS where I can get an async context within 
initialize
(atm not properly documented) -> https://github.com/pulumi/pulumi-awsx/blob/master/awsx/ec2/vpc.ts#L68 so the idea could be -> query resources -> dynamically import into the state -> run the rest of the pulumi -> throw away the state.
m
@great-gigabyte-39962 Can you explain in a bit more detail how what you want to do differs from something like https://github.com/pulumi/examples/tree/master/aws-py-secrets-manager? If all you want to do is to ensure that a set of AWS resources exists in a specific state, you don't need the automation API or any other advanced/complex functionality. You can use the structure in the example I linked and use your local file system or an S3 bucket as the backend (instead of Pulumi Cloud): https://www.pulumi.com/docs/concepts/state/#using-a-self-managed-backend Whether the partial updates you describe are possible or not depends on the provider and the specific resource. You'd have to work with https://www.pulumi.com/docs/concepts/options/ignorechanges/ and imports. That might become tedious, though. Pulumi generally assumes that the program encodes the entire desired state.
g
so the idea could be -> query resources -> dynamically import into the state -> run the rest of the pulumi -> throw away the state.
I’ll have to refactor things from the simple procedural approach I have now, but I could probably make this work. Thank you again!
@modern-zebra-45309 I have a tool that creates and updates resources in AWS. I don’t want to get deep into what resources or why. Ultimately I’m investigating whether I can leverage Pulumi to help with this task that I have already built tooling for. I’m not hunting for the tool, I already have it. I have a program that generates configuration and then realizes it, currently through boto, and I’m hoping just to replace boto with something more naturally idempotent. This tool is used many times over deployments of many projects — it is not the actual deployment project itself. (Whether I should scrap the whole tool and use only Pulumi/TF/whatever is another question, and a valid one, but it’s not my question right now. Right now I just want to use short-duration state to make declarative configuration in AWS, without saying “here I have a Pulumi description of infrastructure and I will use Pulumi commands to make it go up and down.“)
g
the local state might be helpful too if you wish to keep it only for a shot time
m
@great-gigabyte-39962 Just out of curiosity: So you've effectively written a custom AWS provider and are looking for ways to use Pulumi as a DSL for it?
g
Strictly speaking yes, but I wouldn’t say exactly that. I’d focus more on the point that writing a resilient, declarative provider is a dodgy business and if Pulumi can provide that in a more or less straightforward, drop-in way then I’m improving the code and reducing the maintenance cost. That’s why my initial question was whether this is even a reasonable goal. 🙂
m
This is why I wondered what exactly is missing in a scenario like https://github.com/pulumi/examples/tree/master/aws-py-secrets-manager? Is it that you want to keep your program code (= not work with Pulumi resource constructs directly) and effectively have a drop-in replacement for boto3? Or that you don't want to use the Pulumi CLI but call Pulumi from your own code?
g
The latter, I want to call Pulumi from code. The existing software already has a CLI driver and I’d like to keep the interface small, without adding additional steps for the end user. But also, the Pulumi/TF model of storing persistent state across invocations adds new overhead to the existing system. There’s no single point of authority for the state of the deployment.
m
Then the automation API maybe is what you're looking for, and I second the idea of a "query resources -> dynamically import into the state -> run the rest of the pulumi -> throw away the state" loop. I have built a CLI application with the automation API, happy to answer questions or share my experience to the extent that I'm able to.
g
Great, thank you both so much!
This is a part-time project so not sure how soon I’ll next be able to work on it but I’m glad I came to ask questions.
7 Views
Open in Slack
PreviousNext
Powered by

Pulumi Community

No matter how you like to participate in developer communities, Pulumi wants to meet you there. If you want to meet other Pulumi users to share use-cases and best practices, contribute code or documentation, see us at an event, or just tell a story about something cool you did with Pulumi, you are part of our community.

Powered by