I'm making my own framework/tool on top of pulumi/the automation API in python. Pulumi doesn't have support for verifying that old resources are setup before replacing them with new ones, making it impossible to have phoenix servers (cattle not pets) with zero downtime/without risk of stuff going down on a bad update. This article advocates for cattle not pets https://www.pulumi.com/blog/deploy-wordpress-aws-pulumi-ansible/, but pulumi leaves gaps in it as is. I'm making this because I can not afford the possibility of deploying software (like a wordpress site for example), and having everything go down before a new server is ready, or having the old server gone because of a bad download or update. I raised a ticket about this issue with current pulumi capabilities here: https://github.com/pulumi/pulumi/issues/20619 . This is not intended to be an advertisement and I'm doing this almost entirely for myself. • Pulumi manages resource and there is no additional lifecycle beyond crud operations. I'm making a framework to add a layer of components in python code that will allow for putting resources in different stacks on a "super stack". (think pulumi_program() -> None) My model to solve this problem is that: You manage stacks on a super stack, stacks are the component system, you create a stack, add some resources, and can add provisioners to a stack component, a stack within a super stack will have it's own state within my tool, it will likely have a flag to say that it is done, based on all of the provisioners running, if any of them fail, a stack component will not be fit to replace an old component. • The framework operates by taking a pulumi program function and capturing the information of every resource created. This charting process effectively creates it's own goal state for later creating a pulumi program for each stack with the required information. This program creation is done entirely in python, with a new function that operates like a normal pulumi automation api program function. • The process of capturing information for resources created in a pulumi/my framework program for later reconstruction is not necessarily clean. The Resource constructor is monkey patched in the charting process in order to discover properties (i.e. arguments that need to be passed to the resources to create them in a pulumi new program), to find only the arguments and not outputs, the signature of the internal init method is read, which is basically private. • This has been developed mostly with integration test driven development, with a behavioral focus. All features have been added by writing tests/example programs, running the custom program runner/pulumi together and verifying the results after. This means that while some pieces could be considered dirty and break due to changes in python pulumi code (i.e. framework code relying on private methods that could change), it would be trivial to check compatibility with a pulumi version by running the tests, and reasonable to use the failing tests to guide fixing them. Scenarios for features can be seen here: https://github.com/RequiemWorld/requiem_omnislash/blob/935be99325abcab916e5f70d3edde8c7dfae1542/tests/test_program_runner_resources.py , and here https://github.com/RequiemWorld/requiem_omnislash/blob/935be99325abcab916e5f70d3edde8c7dfae1542/tests/test_program_runner_relationships.py . • Most importantly, the goal is that writing infrastructure as code does not change. No complicated code using the automation API and using multiple stacks will have be written when writing infrastructure code. The automation API is advertised as being capable of something to use for scripting up blue/green deployment patterns, but it isn't reasonable to exit declarative code for something that a framework could handle more nicely and the edge cases of. I am dead serious about seeing this through so that the type of deployment shown in the article can be achieved safely. I have put 21 hours of work into designing the start of this/writing the initial code for it. I started on the 13th and I've designed and built it up iteratively, one behavior/test at a time. It doesn't do what the goal is yet but a component system and multiple stack creation is there, along with getting outputs and stuff like that right. I'm not saying any of this as a complaint or intended to say anything bad about pulumi. I'm sharing this on the contrary, I love what I have experienced with pulumi and I am dead serious about using it, it's the only option I take seriously. I think what I'm experiencing with the automation API speaks volumes to how composable, useful, and capable pulumi really is. I guess I'm posting this here because I'm interested in sharing my effort/getting reactions if anyone should take the time to read this, I would hope that what I think is a serious effort for writing a framework based on pulumi (which I don't see anyone else doing) would seem interesting. If not, that's fine, just wanted to say my piece and share an update since last time 🙂

Hello @proud-painting-63563 I remember your original slack conversation with @echoing-dinner-19531. Very cool that you got something working. It looks like your original ticket is still being tracked, and I am guessing this is related as well: https://github.com/pulumi/pulumi/issues/21898 ( though I'm not an expert on hooks, so not sure if that solves it )

Yeh we are interested in changing hooks to error the deployment if hooks fail

@modern-spring-15520 You are correct that it is related. I explored hooks thoroughly and failing the deployment would solve part of the problem or enable a different approach to the same end I think. • When you reach the create hook, the resource has already been created on the underlying provider. Pulumi will replace it regardless of hook pass/fail. I don't remember if it is replaced by the time this hook is called or not. • I think there was slightly more possible with the before delete hook doing something a bit obscure but not really useful or safe enough?. Regarding failing the deployment on resource hook failure: • It would have to be durable, meaning that if the program were to crash during or before pulumi got to executing/finishing executing the hooks, it shouldn't consider it a success. Example, you add a resource hook for provisioning the server, uploading some software, starting it, and a health check, but then pulumi creates the resource, it goes to execute your hooks but it crashes and the next time it runs it continues with everything else as if it succeeded. • @echoing-dinner-19531 Said that the part I describe as durability wasn't initially considered but that it makes sense: https://github.com/pulumi/pulumi/issues/20619#issuecomment-3352463442 . I've mostly only reasoned about the resource hooks as they are, not how they will be after but: • The whole program/deployment failing sounds correct, but with the question: If the resource hooks on my resource fails. and the deployment fails, when I try it again, will it try to create a new replacement resource and recover on success/replace the old. • For example in terraform, If I have a provisioner on a resource and it fails, the deployment will fail, the resource will get tainted, and on the next up, it will delete the old and try to create another new one (and I'd assume it's durable).

yeh failing deployments and being durable are both things we're looking at. Might be we just do the first to start, but both are being looked at.

@echoing-dinner-19531 I'm glad to hear that the interest is there and I'm happy if the durability part is at least a longer term goal. But to be clear that's not my only reason for attempting to build my own framework on top. Reasons/relevant information includes: • Components are like resources (I think in python they inherit from the resource class or descendants of it), resources have properties and resources can be replaced with other resources, you can put multiple resources in one component.) • It would be cool if it was possible to manage a group of resources (a component) as a unit and replace them with another unit. An existing cloud service won't always provide what you want in a simple resource, but, it would be trivial to make your own or use existing ones with a better component system. • It would be ideal if there was a component system that worked like regular resources but managing multiple. It would be possible to create a component that for example takes the name of a docker image and the ports to map and when you change the name of the docker image, it creates a new component and replaced it with the old one. In the same way that if you give a different input to a component it will be replaced. I initially tried to achieve similar with properties in the existing component system only to discover that's not how it works. If some of or all of this is out of the scope of pulumi, that's fine. But as a user, I see a resource like a droplet and I see user data, I change the user data and I get a replacement droplet, it leads me to think that if I have a component with it's own properties/inputs, I can change them and build cool and similar stuff myself. I can't even really replace a droplet necessarily for example unless I add comments or something to the user data based on inputs to a component. Just wanted to clarify my reasoning for pursuing this 🙂

yeah that is me 🙂 👋

If some of or all of this is out of the scope of pulumi, that's fine.

I mean this is all very interesting ideas. I'll definitely be giving it all some thought.

@echoing-dinner-19531 I think that you should consider some of these ideas in the context of managing third party software. The issue is that some of these ideas can be (I don't think universally necessarily are) bad ideas for managing software that you write. • Writing/Releasing deployed software: If I write a piece of software I should be testing it in a production like environment using all of the deployment scripts and having everything be the same so that I know if I tweak a detail or two I can have the software released to production. You can conclude frying images (the opposite approach of hashicorp packer for example) is a bad idea because there is no guarantee that the software/server released into production, is the same the production my end to end tests ran against, for example a kernel update could break performance or something, or updates could break networking, who knows. • Third Party Software: There is so much software and stuff that you could want to do with it that you have to get a server and setup and continuously manage. You have to manage infrastructure and you have to do work. I can't imagine any simpler option than replacing an entire server instance with another. I wouldn't have to keep track of IP addresses. I'm a developer and I can not stress enough how much I love code and how much simpler it can make things, if I could say "I want a docker component, and this docker component should have an IP address, and the name of the container" and start building up the infrastructure for tasks I have to do, everything would be so easy, and it would be easy to clean up too. • My actual use-case: I can talk in theory about the use-cases about this but I actually have I'm planning to handle with this. The issue is: In the community I'm in there is a project near mine which uses digitalocean spaces (s3 buckets, they bill you exclusively for bandwidth, $10 per terabyte of downloads). The project came under attack, and it was using roughly 350gbs per day. Unfortunately, no one bothered enabling and reading the logs so we don't know specifics. But I was able to replicate the issue and do a lot worse: In less than 2 hours, and with a droplet that only costed pennies on the hour with a 2gbps connection, I was able to cause $10 in damages to my own bucket with a 1mb file and there was no rate limiting. Why I want immutable servers: I want to protect my project in a way that does not create much additional work or management for me. My ideal solution: I want to continue using spaces s3 for convenience, but I want to have the public servers be droplet instances which replicate the files there and have their own rate limiting. I want updating them to be as simple creating new ones and replacing the old. It would be as simple as deploying a new server, updating it, installing nginx, configuring it, replicating everything to a point, then replacing the old one and switching floating IPs (at least that's how simple I think it would be). If people are disconnected mid-download because of a floating IP switch/server deletion then that's fine because on the scale of project I'm on, I think it's fine to trade a tiny bit of downtime or disruption for significantly less work, when the alternative is manual, error-prone, less reliable labor. What I'm trying to say is that the ideas I present have flaws or trade offs which you'll probably think of but I think that the contexts should be considered. I really want to use the developer skills I have in situations where the alternative really is just managing servers and third-party software, where even If I manage to have a deployment pipeline (all changes including OS updates flow through it) for the software I write, it would still make sense for stuff I have to manage to support that software, where if the only context considered is software you write and deploy, it might not so much. But think about this at least: If I can rent a server and write some playbooks to upgrade it, put the software on it, put a new version of the software on it and that is valid in one way or another, even if it's not as good, why can't i eliminate the work of renting, and setup, and maintenance and just swap out server instances. Even if any of this can be bad, it's still gotta be so much better in some cases to valid enough alternatives. Also, I'm coming from a hobbyist background more or less, and generally people say that if you put more work in, you get to reduce costs and save money, but then it costs human labor, but I feel like there's a chance to automate almost everything, and still save money here within reason (a dedicated server and manual work is always going to be cheaper, so are virtual private servers in comparison to servers available on cloud providers) at least in some contexts. I don't have anything else to say about this. I appreciate you guys taking the time to listen regardless of the timeline on implementation of certain things, and only if it is just thinking about it 🙂