Hi All. Scenario: Google Cloud. Resource: Compute...
# general
r
Hi All. Scenario: Google Cloud. Resource: Compute Instance (but could be any cloud resource). IaC: Terraform (active) Pulumi import Compute Instance (so stack is updated) and, post import output, main configuration file updated with resource definition. Does this mean that both IaC solutions could be used to amend / delete the resource? Does Pulumi overwrite any cloud resource attributes, in the cloud resource definition? Are there any edge cases where an IaC provider can write IaC-centric attributes to a resource and, when importing to Pulumi, would Pulumi overwrite those? Put another way - is the Pulumi import process a read only process? Once the stack is updated; how impactful could a pulumi up action be on that resource where the resource definition hasn't, as yet, changed?
l
Hello @rich-agency-75207, if you import an existing cloud resource into a pulumi setup (code + state), then Pulumi assumes it is owner of that resource from then on, as is the case for Terraform. No, you can’t use both tools at the same time. We do understand that you can’t migrate over all your infrastructure all at once. Now, if you do it project by project, sometimes Pulumi needs to read TF outputs, which can be done with this: https://github.com/pulumi/pulumi-terraform If you talk about the reverse case where TF needs to read Pulumi stack outputs, we recently got the following contributed into the #pulumiverse community: https://github.com/pulumiverse/terraform-provider-pulumi When importing a resource, there are 2 cases: • You already have a resource with a certain name in your Pulumi code and you run `pulumi import`: Pulumi reports which actual properties differ from the code you have in your Pulumi setup. You can import the resource id, but as long as the values differ, you will get a diff presented at the next
pulumi preview
or
pulumi up
• You don’t have the matching code yet: Pulumi will import the resource into your state with all the actual values, and it will display the code snippet you have to add to your Pulumi program. The values in the generated code are the actual values, so you will not get any diff presented at the next
pulumi preview
or
pulumi up
.
👏 1
👍 1
m
Hi @rich-agency-75207, In my vision, using two different approaches to handle the same resource is not a best practice because your resource will have interaction in different ways, for example: Terraform and Pulumi Native Provider, not only the tool but code could be different(tags, context, etc.) For sure that you can survive with Pulumi and Terraform together. However, If you can separate which resource will be handled for each tool will be a better approach. Pulumi import is normally used to add in your Pulumi code/state a resource that you already have orchestrated and don’t want to change right now, adding all resources to your IaC(existent resource + new resources) to allow you to manage all resources using Pulumi. To migrate from Terraform to Pulumi, it will be better to use https://www.pulumi.com/tf2pulumi/ which will allow you to migrate from Terraform in a faster way than import all resources in one. NOTE: I guess that @limited-rainbow-51650 gave you a better description of Pulumi import than my description 😄
l
More resources: • the video from my colleague @powerful-electrician-48045 here: https://www.pulumi.com/terraform/https://www.pulumi.com/tf2pulumi/ (stay tuned where we update this include support for the latest HCL syntax) • https://www.pulumi.com/docs/guides/adopting/from_terraform/
r
Thanks both - super helpful info
@mammoth-electrician-64525 - to clarify - my use case isn't to manage a resource via two distinct IaC providers concurrently. I am faced with a migration scenario from TF
However, although we use TF, our IaC is defined using an in-house yaml schema
Further, we use jinja2 and extensive templating.
I've already developed a reference design re: pulumi yaml via use of jinja2 and templating. I am now focused on
state
and
yaml conversion
. However, it seems to me, that a pulumi import (second bullet point shared by Ringo) would allow me to re-build the source project yaml templates and add the jinja2 conditional code afterwards..
The problem with that approach is that pulumi will become the owner of that resource and break TF ownership which, presumably, would break the TF state file causing an error
Do you guys agree with my last speculation?
l
@rich-agency-75207 just an idea here: can your in-house yaml schema, and the templating tool be adapted to spit out Pulumi Yaml? This might be the shortest path for you to migrate everything over, at least code wise already. Once you are there, you can use our
pulumi convert
cli command to convert the YAML to any of our other supported programming languages. This latest conversion is only a conversion of the code without any impact on the Pulumi state at that moment.
r
@limited-rainbow-51650 - that was my first instinct too.
wrt "Once you are there, you can use our pulumi convert cli command to convert the YAML to any of our other supported programming languages." I would not want to do that. I am a subscriber to the yaml philosophy where resources are declared. Given that jinja2 already caters to all conditional logic requirements etc. I would not want to introduce a programming language paradigm.
fyi - our in-house iac tooling (templating tool) is a python based program. It parses a primary 'product' yaml, references 'project' yamls, interprets jinja2 logic and spits out a master 'product file' yaml that contains all associated projects in scope of environment. This is then converted to TF etc.
l
I understand this. Now, let me sketch that you are missing out on a lot of Pulumi power if you stick to YAML for all the raw cloud resources. With Multi-language Components, you can implement your abstractions in a programming language, package it up like a provider and still instantiate your component from YAML. You will not force any programming language paradigm onto your users this way.
👍 1
r
@limited-rainbow-51650 - Writing a yaml converter, using in-house schemas, to pulumi yaml does seem the best approach. This will allow me to first 'translate' the code only w/o impact to current state. Then, using the first bullet point approach you shared ^^ - I would, in effect, perform a migration of TF state controlled assets to Pulumi state control.
To clarify:
Copy code
You don't have the matching code yet: Pulumi will import the resource into your state with all the actual values, and it will display the code snippet you have to add to your Pulumi program. The values in the generated code are the actual values, so you will not get any diff presented at the next pulumi preview or pulumi up.
Is there a way to get Pulumi to generate the code w/o updating stack or cloud resource state?
m
I finished some weeks ago a huge migration from Terraform to Pulumi. Maybe my case could be useful to you. I know that every migration project takes time to redefine the same standards applied in Terraform to Pulumi, mostly security and governance. We had a repository for the SRE team, to provisioning all resources for all environments and multiple repositories with Terraform code into the applications, just to promote applications from dev to prod. Interacting with pre-existent resources. We first migrated our SRE Terraform repository because the impact was internal to the SRE team. We use Pulumi and Python to do that. After we migrated everything, we archived the same level of automation of Terraform in Pulumi but with many improvements because we added Python, business rules, governance, security, Jinja2 temples, etc. We deprecated the Terraform code when we were confident to use Pulumi, we avoid interacting with both at any time. For the developer’s side, we move all of our Terraform code to Pulumi YAML; in that case, we can give developers the autonomy to maintain how they will release and deploy their services into environments; without our management, we support them however we aren’t an owner of that. If you try to adopt immutable infrastructure, your problems with Terraform and Pulumi state will reduce drastically. In my case, I don’t care about Terraform or Pulumi state. We change the legos(resources) and use the reproductive way than caring about the current or previous state of any resource. We use the Pulumi state to save time by adding a new resource that depends on other resources already in place
In my vision managing the state of IaC didn’t bring any results to the company, however, automation and reproducible capability are results that the company expected from us. In our roadmap, we are planning to add Pulumi Automation API and create the culture of self-service infrastructure, when developers can consume an API or CLI, created by SRE to provision any environment to be used in POCs or tests, exactly as we have in our environments.
r
Thanks @mammoth-electrician-64525 - that is very helpful. I will need to spend sometime to unpack everything. I would look to preserve the current product / project (parent / child) yaml template structure since this is what devs are already accustomed to. One of the first things I considered was simply taking the final generated TF configuration and using tf2pulumi but relational logic, is, of course, lost in translation and I would then have to reverse engineer the pulumi yaml to reconstruct the original start state.
"automation and reproducible capabilities" are our mantra too
as too is self-serve
l
r
Could this be indirectly achieved by simply never executing a pulumi up operation? I.E. Actual values are pulled, the active cloud resource state does not change at all and, so, one could iterate through this process to generate pulumi yaml (given a small number of resources involved)?
m
I guess that your plan is exactly what we did. We don’t expect a perfect migration using tf2pulumi, but it was helpful to have good progress in the first interactions. If you try to have the same behavior in an application written with Go using Python the results could be the same but the perspective will be different because both have different concepts. If you can save 30% or 40% of your time using tf2pulumi will be amazing. Another point that probably your team will discover in that migration journey is that you will change your IaC when you are migrating to eliminate the some pain points that we had with Terraform like the evil “depends on”, and code too much verbose. Today our IaC is too tiny compared to Terraform, and we have much more intelligence, we definitely have a clean code; for example, we check the costs on the fly to decide if we will create a resource. We check the budget defined by the business before creating a new resource for a specific project, both of these improvements are dreams with Terraform.
🚀 1
👍 1
r
like the evil "depends on", and code too much verbose.
👍👍
l
Indeed,
tf2pulumi
will convert your code, but we are in progress of updating it to support the latest syntax, so we can increase that coverage way beyond the 40%. You can follow up on our public roadmap: https://github.com/orgs/pulumi/projects/44
👏 1
This is the ticket most to your interest: https://github.com/pulumi/tf2pulumi/issues/186
👍 1
r
@limited-rainbow-51650 - I need to clarify - I have considered tf2pulumi but it doesn't really help my use case.
Let me try to explain:
Product/Service X
Comprised of n cloud projects
Different services in the E2E data processing pipline
applications, etc.
IaC at rest:
Product X IaC template
contains parent metadata as well as project yaml template references + metadata
Both parent and child yaml files contains jinja2
Once parsed then final product yaml is 1000s lines long
This is used to produce a TF configuration used to produce build plan etc.
Using tf2pulumi doesn't help here because it would product a monolithic output
That I would then need to decouple and reconstitute
l
@rich-agency-75207, as @mammoth-electrician-64525 mentioned, if you look at your current code, and you leverage core features of Pulumi in the migration, your code will reduce drastically. A template in your current custom tool, wrapped around TF, could be (given the current insight I have) converted to a multi-language component (in real code) that you then invoke in YAML with a few lines. The amount of lines only depends on how much configuration you have to pass to that.
r
I will consider your proposition. I do see the practical and technical benefits 👍
However, first and foremost, I need to determine a path that persists the current design so that a proposition exists that outlines minimal impact to devs who are used to maintaining a specific iac source structure
A change in design will also be submitted ( with benefits outlined )
Further, I was hoping to drop the python codebase.
l
Are the devs using the Python templating tool you mentioned earlier? As a transition path, you can use the Automation API and embed Pulumi within that same tool. The CLI remains the same for the devs, but the rollout is Pulumi at that moment.
r
Not directly - the tool is invoked as part of Gitlab CD - it has been abstracted away
Only product / project yaml files are updated that are conformant to the in-house schemas
Thanks @limited-rainbow-51650 @mammoth-electrician-64525 though - you have given me a lot of good information to review and unpack. Will likely be on this channel again soon with some updates.
l
Yet another way is to implement a Pulumi program which reads your product/project yaml files and uses that as input directly. This means, no pre/post processing before you run Pulumi (on Gitlab CD). Complementing that with the multi-language components is something which would simplify your setup a lot.
👍 1
m
@rich-agency-75207, I guess that you are getting the same issues as us, but the solution that we used was not to try to convert all Terraform bases of code. However, we convert module by module, and after that, we create a final glue. Maybe one day we can have a 100% compatible solution. However, I can’t see a different path nowadays. @limited-rainbow-51650, First of all, congrats on the excellent work with tf2pulumi. In our case, our efficiency was around 24% which is not bad because our Terraform code + Terragrunt had a lot of challenges that we didn’t expect to be addressed with that tool. We are discussing a feature request to add in the tf2pulumi to give insights or suggestions in the converting process, I guess that it could be helpful to some uses, but it is still in the brainstorm phase :D