Hello, I am integrating our Pulumi python code in the Azure DevOps pipeline. In order to do it I am using the Pulumi task that is available for free on the MarketPlace. However I have struggled to make it work properly. The biggest problem that we have so far is the impossibility of installing the required Pulumi plugins. We are trying by using the Azure Cloud hosting agent for the moment, but we feel that (since the task is downloading Pulumi), the task should also install the required plugins (and maybe it could also do a pip install -r requirement.txt 😛 ). Am I missing something or it is indeed a missing functionality?

Hi @curved-alarm-76356 , first off thank you for reaching out to us!

Am I missing something or it is indeed a missing functionality?

Nope you are not missing something. 🙂 I can definitely understand the need for having the Pulumi Task restore the deps. However, because Azure DevOps has ways to cache dependencies and especially for self-hosted agents, it is common to do so, it is easier if the Pulumi Task does not “get in the way” of doing that. Given that there are built-in tasks within the Azure Pipelines that you can use to restore dependencies, I didn’t want to duplicate that inside the task.

However I have struggled to make it work properly. The biggest problem that we have so far is the impossibility of installing the required Pulumi plugins.

I’d love to understand more about the problem you are facing and help you to resolve them. Would you mind opening an issue on https://github.com/pulumi/pulumi-az-pipelines-task/issues?

Hello @clever-sunset-76585, Thank you for the prompt reply. I will open up an issue, but my problems were related to not accurate docs on the options available (maybe there is but I did not find it), python venv and pulumi plugins. I was kinda expecting (since I specify it in the Pulumi.yaml runtime options) to have the python env automatically provisioned. But I can understand why it would not. Do you have any pointers to an azure pipeline which used the methods mentioned by you?

I will open up an issue, but my problems were related to not accurate docs on the options available (maybe there is but I did not find it), python venv and pulumi plugins.

Ah I see. Did you manage to get past the problems now?

Do you have any pointers to an azure pipeline which used the methods mentioned by you?

I do not have any examples at hand. But have you seen some of the built-in tasks from Az Pipelines? Here’s a doc for the Python Script task: https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops

Yes, I am familiar with it but sadly (at least on Cloud Hosted agents) does not cache anything. I will try to use the Cache@2 task to see if it works and will let you know. However it is a pity, because if I have to manually script: 1. Python env 2. Pulumi plugins install then I can easily add Pulumi install (and login) and not use the Pulumi task at all. On top of that, do we have a way to install Pulumi plugins from a file? It would come in handy for having the full blow automation (and avoid manually installing them on the self-hosted agent) Maybe I can open an issue and propose it as optional feature.

Yes, tasks themselves do not caching. You have to be explicit about caching anything you care about not repeating in your pipeline. The only type of cache that the cloud-hosted agents do is, tool cache. That is, they cache the installed tools in a path in a single execution. Because the cloud-hosted agents are “shared” I don’t know what level of caching you can do with them. For self-hosted build agents you are more in control over what sort of tools you have installed on your agent and caching folders. For more about pipeline caching, you may want to look at https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops.

However it is a pity, because if I have to manually script:

1. Python env

2. Pulumi plugins install

then I can easily add Pulumi install (and login) and not use the Pulumi task at all.

Yeah I totally understand that. The thing with restoring dependencies is that, there are multiple packaging tools in each supported language that could bloat the Task and then again it still wouldn’t be able to do everything. So it is best for users to use the built-in mechanisms of each Az Pipelines. I do agree with what you say, though, that you’d have to separate tasks for each step. On top of that, do we have a way to install Pulumi plugins from a file? It would come in handy for having the full blow automation (and avoid manually installing them on the self-hosted agent) The Pulumi plugins installation step should happen automatically when you initialize your Python env and run

pip install -r requirements.txt

. Are you seeing something different? So you shouldn’t have to do it separately. As for installing the plugins from a file, the

pulumi plugin install …

command takes in a URL (

--server

) from which you would like to install plugins, as well as a tarball file (

-f

) from which you can install the plugins. https://www.pulumi.com/docs/reference/cli/pulumi_plugin_install/

"The Pulumi plugins installation step should happen automatically when you initialize your Python env and run 

pip install -r requirements.txt

. Are you seeing something different? So you shouldn’t have to do it separately." It does not. Let me explain the workflow. We have created the Pulumi stack on a local machine using`pulumi new ...` Then we push to the git repo the files. On the pipeline we run:

pip install -r requirements.txt

However when we run

pulumi preview

we get the message of missing plugins.

@curved-alarm-76356 we are looking into this. There is also this issue that was opened within the last week that I have just been made aware of. https://github.com/pulumi/pulumi-azure/issues/601 You may want to follow that issue.

Hello @clever-sunset-76585 Thank you for the reply. I was the issue. I have to say that in my case it is not installing the plugins at all. I might forgot to mention that I am using Ubuntu 18.04 LTS as distro and not Windows.