Hello! 👋 We are building an internal developer platform using the open-source Pulumi + Automation API. Our application uses

LocalWorkspace

to run stacks. We'd like to understand: Is there any way for

LocalWorkspace

to directly consume a Pulumi program (or ComponentResource template) from a remote Git repository? Or is it inevitable that we must always bundle Pulumi code inside our application source (or manually clone the repo before calling

LocalWorkspace

)? • (we're also considering using npm packages (or similar package managers) to distribute Pulumi components, but we're curious if this is a generally recommended best practice for this use case) To rephrase: In the OSS-only setup (without Pulumi Deployments or RemoteWorkspace), are users expected to always embed Pulumi programs locally, or is there a recommended pattern for reusing templates/components maintained in separate repositories? Any guidance or best practices would be greatly appreciated.

I've not tried so I can't confirm. However I can confirm that non-inline projects and git submodules do combine to solve this problem, albeit not as elegantly as a Pulumi-native solution might.

directly consume a Pulumi program (or ComponentResource template) from a remote Git repository?

Internally it would just have to download it locally to run it anyway. You can just scaffold that around AutomationAPI yourself, with the benefit of downloading whatever from where ever rather than just sources that we've added support for.

Thank you for your help. I have an additional question about another issue! Is there an accurate way to monitor the computing resource usage of Pulumi's plugin processes? The problem situation is as follows: We're creating n number of AWS resources within a single componentResource. We're using 3-4 providers to create resources across different AWS accounts. Additionally, we're running a Node.js-based application that utilizes the Pulumi Automation API as a container on Kubernetes. We haven't specified the parallel option for stack operations. During resource provisioning operations, CPU/memory usage becomes very high, which seems to create the possibility of issues like OOM (Out of Memory) errors. Currently, the Node.js process inside the container doesn't show high computer resource usage, but we suspect that the resource usage of multiple plugin processes is the root cause of the problem. We think adjusting the parallel option value might resolve the issue. However, it would be great if we could accurately understand how Pulumi plugins (including the AWS plugin) work or what operations they perform. We would really appreciate your help! 😄

However, it would be great if we could accurately understand how Pulumi plugins (including the AWS plugin) work or what operations they perform.

The engine starts a new provider process up for each provider declared in the program, plus the default provider. Most plugins are pretty simple just transforming pulumi resource requests into the given clouds API requests. I know they take a lot of memory to hold all the mappings of how to go from pulumi shapes to api shapes but that should be a fixed size. I don't think they should have a lot of dynamic memory requirements.

he memory profiling results

Did you look at that with pprof? I thought it collected samples for the whole program not just the end.

Thank you for your help every time! As you mentioned, I confirmed that memory usage during provisioning operations is somewhat fixed. I also confirmed that the basic computing resource usage is quite high. The issue is that I'm wondering if there are ways to reduce or optimize the basic computing resource usage itself. Currently, we're creating various resources to run ECS within a single componentResource, and since these are created across different AWS accounts, we have to use 3-4 providers. In this situation, when I briefly checked the memory usage, I found that plugin processes account for a very large portion as shown below. This makes me think that if we change to a method of handling multiple stacks within a single container in the future, resource usage would increase even more and become less stable. We're currently running with cpu/mem: 1024/2048 allocated to the container and not using the parallel option separately, using the default value of core * 4. I tried intentionally reducing the parallel value, but it doesn't seem to be a fundamental improvement method. Perhaps running separate containers for processing each request might be a good approach (like GitHub runners :) ). If there are any good solutions or improvement methods for this issue, I would really appreciate it if you could let me know!

/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1  80m 1.2g pulumi
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1  92m 1.2g pulumi
  474   450  45m 750m pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 112m 1.3g pulumi
  474   450  62m 751m pulumi-resource
  491   450 289m 2.0g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 112m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 310m 2.0g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 112m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 350m 2.0g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 117m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 422m 2.0g pulumi-resource
  519   450 106m 1.9g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 117m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 427m 2.0g pulumi-resource
  519   450 309m 2.0g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 117m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 430m 2.0g pulumi-resource
  519   450 322m 2.0g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 117m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 430m 2.0g pulumi-resource
  519   450 428m 2.0g pulumi-resource
  544   450 210m 1.9g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 117m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 430m 2.0g pulumi-resource
  519   450 428m 2.0g pulumi-resource
  544   450 319m 2.0g pulumi-resource
/ # ps -eo pid,ppid,rss,vsz,comm | grep pulumi
  450     1 117m 1.3g pulumi
  474   450  63m 751m pulumi-resource
  491   450 430m 2.0g pulumi-resource
  519   450 428m 2.0g pulumi-resource
  544   450 317m 2.0g pulumi-resource

I think the best thing would be to raise issues on the repos for the providers your using to ask about memory performance optimisation. Recorded issues for problems like this are the best way to get some company time put towards improving it. I don't think there's anything in the system today to really help here.

If you're able to upgrade to the AWS v7 plugin, you can use a single provider to create resources in any region. That would side-step the problem, by the looks of it?

Thank you for the suggestion. I did check that part, but I'm currently in a situation where I need to create various resources across multiple accounts, not multiple regions. That's why I needed multiple AWS providers, and I believe the main cause is having multiple AWS plugin processes running