pulumi-community #automation-api

automation-api

lemon-agent-27707

09/02/2020, 3:18 PM

set the channel topic: 🏭 Happy Automating! 🏭 Full announcement, docs, details linked here: https://github.com/pulumi/pulumi/issues/3901#issuecomment-685803282

lemon-agent-27707

09/02/2020, 3:19 PM

Full announcement here: https://github.com/pulumi/pulumi/issues/3901#issuecomment-685803282

lemon-agent-27707

09/02/2020, 3:19 PM

A list of my favorite resources to get an overview and get started: • GoDoc with comprehensive examples across the API surface area • Automation API Examples: a repo with end to end walkthroughs for multiple facets including inline programs, git workflows, inline/cli hybrid programs, and multi-stack orchestration • The source code for Automation API • halloumi: a prototype of a heroku-like tools that shows what sort of powerful abstractions can be built when infrastructure is just another part of your application code. • The Pulumi Kubernetes Operator: a real world example of exposing Pulumi behind an API.

🎉 9

lemon-agent-27707

09/02/2020, 10:23 PM

👋 Open invitation for anyone who's taken the time to kick the tires to schedule a 30 minute zoom call to provide feedback: https://calendly.com/evan-148/30min We'd love to hear from you!

lemon-agent-27707

09/03/2020, 11:48 PM

👋 Another open invitation. I'll help you setup automation API in your project: https://calendly.com/evan-148/30min

lemon-agent-27707

09/04/2020, 2:32 PM

A great video from @faint-table-42725 showing an overview of automation api in action:

https://www.youtube.com/watch?v=m92HpG-JlUs&ab_channel=PulumiTV▾

lemon-agent-27707

09/04/2020, 11:42 PM

@chilly-rainbow-79265 @important-appointment-55126 I put together an example of Automation API for provisioning temporary VMs. You'll probably have to make some adjustments to parameterize the VMs to your specifications but I think this should be a good starting point. It's a higher level CLI called

vmgr

that has two functions: •

vmgr add

creates a new VM (webserver in this instance) and prints out the public IP. Given that this is just plain Go code, you could extend it to email VM credentials to someone, put them in a queue somewhere, etc. •

vmgr cron <expiration>

ie `vmgr cron 5d`: runs a minutely job scanning for VMs older than the specified expiration. It will retry on failures, etc. You could run this on your laptop, or on a VM/container somewhere https://github.com/EvanBoyle/automation-api-examples/tree/main/vm_manager_azure

clever-byte-21551

09/06/2020, 11:07 AM

Hey I've been trying to play with the new Automation API and I tried running one of the example and I get the following error

Successfully setup workspace
Installing the AWS plugin
Successfully installed AWS plugin
Logged user: yarin
Failed to create or select stack: failed to create stack: code: 255
, stdout: 
, stderr: error: stack names may not contain slashes

: exit status 255: failed to select stack: code: 255
, stdout: 
, stderr: error: no stack named 'yarin/project-name/stack-name' found

: exit status 255

clever-byte-21551

09/06/2020, 11:07 AM

the “stack name must not contain slashes” comes from the CLI tool (v2.9.2)

clever-byte-21551

09/06/2020, 11:08 AM

I'm creating a new stack using:

fqsn := auto.FullyQualifiedStackName(user, projectName, stackName)
s, err := auto.NewStack(ctx, fqsn, w)

clever-byte-21551

09/06/2020, 11:08 AM

What am I missing?

clever-byte-21551

09/06/2020, 12:22 PM

After some debugging it seems that I can’t use a local backend ? It requires some changes in the CLI / automation so it would work

clever-byte-21551

09/06/2020, 12:22 PM

@lemon-agent-27707 when you are up please tell me if there’s a workaround or this is intended

clever-byte-21551

09/07/2020, 5:07 PM

How should we handle recovery from interrupted operations using the new Automation API? Even in the CLI approach the support to mitigate this automatically is done very poorly.

lemon-agent-27707

09/07/2020, 7:10 PM

Yes a pulumi cancel is the function that will stop the update. This is currently missing from the automation API, I'll put it in the queue for tomorrow

clever-byte-21551

09/07/2020, 7:21 PM

Also we should consider using custom exit codes from the pulumi CLI to detect certain errors like “interrupted while creating”

lemon-agent-27707

09/07/2020, 7:44 PM

Yes, distinct error codes is a long-term project. The error design was intentional in allowing that to evolve over time. The current set of errors are opaque and can be updated to use the more reliable error codes as they are introduced. If you have other error cases you'd like to detect please open issues.

lemon-agent-27707

09/07/2020, 7:44 PM

https://github.com/pulumi/pulumi/blob/master/sdk/go/x/auto/errors.go#L44

enough-winter-19192

09/09/2020, 10:14 PM

Wrapping the pulumi CLI with the automation-api. Working nicely so far. Main issue so far is the binary size is quite large "~480mb". Is there any way around this? I've thought about breaking out local workspaces into other languages (Nodejs) to avoid importing

<http://github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1|github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1>

, but looks like
/x/auto
depends on this.

enough-winter-19192

09/09/2020, 10:37 PM

My bad, I was wrong about the dependency graph. Removing imports for

kubernetes/core

brought the file size down to 61M. Going to go with a hybrid approach

broad-dog-22463

09/10/2020, 9:18 PM

hey all - just thought i'd let you know in here that we just launched support for non-default secrets providers in v2.10.0 for the automation api

broad-dog-22463

09/10/2020, 9:19 PM

this means you can use awskms / azurekeyvault / gcpkms / hashivault or passphrase

bitter-dentist-28132

09/11/2020, 4:36 PM

does the fact that the automation api is written in go mean it can only load projects written in go?

victorious-xylophone-55816

09/12/2020, 5:31 PM

I don't have anything to contribute, I just wanted to say the Automation API is neat and seems really useful. Particularly enjoyed reading this example repo:

halloumi: a prototype of a heroku-like tools that shows what sort of powerful abstractions can be built when infrastructure is just another part of your application code.

😛artypus: 1

lemon-agent-27707

09/16/2020, 6:41 PM

A bunch of new features just landed in Automation API. This will require updating your CLI to v2.10.1. You can see the full changelog here, but some highlights: • Stack name validations have been relaxed at the Automation API layer. This change was made to enable local backends. • Output from Up/Refresh/Preview can now be streamed to a list of

io.Writer

example • Support added for private git repo authentication: examples • Helper functions to extract permalink from update result: example • cancel/import/export recovery APIs: example • upsert stack helper methods (create or select): example • workspace scoped environment variables (useful for setting passphrase envs for local backends): example • Support for non-default secrets providers: example Docs and examples are up to date as well: • godoc • automation-api-examples • halloumi 🏭 Thanks for all of the great feedback so far! 🏭

🤖 3

🎉 5

clever-byte-21551

09/22/2020, 6:22 AM

Not sure if this is automation API related but I have a stack where calling refresh just stucks. I added logging of the refresh command output, it list all the resources needed for refresh and after that it’s just stuck on

@ refreshing....

for many minutes

clever-byte-21551

09/23/2020, 6:00 AM

Is there a way to sync my plugins with the current SDKs I’m using? Currently I have to manually sync my

go.mod

with the hard-coded plugins versions i’m installing via the automation API

limited-rainbow-51650

09/24/2020, 2:51 PM

Using the automation API, I want to execute a Git based Pulumi NodeJS project. After using

auto.UpsertStackRemoteSource

, how must I trigger

npm install

to retrieve all dependencies before invoking

stack.Up(…)

red-energy-90711

09/24/2020, 3:05 PM

Hi, with automation-api would be possible to run inside a serverless function like AWS lambda or Azure Functions. I mean receive args and then run the provisioning process without "pulumi up" command.

proud-pizza-80589

09/25/2020, 7:18 PM

@lemon-agent-27707 the automation api looks awesome. With a bit more context on where i’m investigating to use it. We have built a BPaaS (blockchain platform as a service https://settlemint.com/blockchain-platform-features/ #buzzwordbingo) which is a large collection of microservices to create and manage k8s clusters on AWS/Google/Azure and deploy on top of this blockchain nodes, api’s, management ui’s etc. Currently it is all hand coded TS + Helmfile + Helmcharts. But we are reworking some of them and I kind of feel like i can ditch all my cloud provider code (and gain a lot more clouds) and helmfile with pulumi. All clusters are set up on demand and settings etc are stored in a database. I think i can cut thousands of lines of code by building on pulumi. Just one thing i need to investigate and that is state management. each cluster/services set can be a stack (we have a unique name for each). What are the benefits of using the could hosted TEAM PRO over an s3 backend?

🎉 1

Title

proud-pizza-80589

09/25/2020, 7:18 PM

🎉 1

Follow up question, how would I use your PR to try it out?

lemon-agent-27707

09/27/2020, 1:11 PM

Hey @proud-pizza-80589, glad to hear this is going to help solve some of your problems! Deploying infrastructure on the fly is exactly what we had in mind when designing the api, so your use case sounds perfect. Trying it out depends on what language you'd like to use. Go is published in alpha and available to use today. There is a full examples repo here that has multiple sample projects: https://github.com/evanboyle/automation-api-examples If you're looking for typescript, we do have a branch but there will be a bunch of changes over the next few days before it goes out for final code review: https://github.com/pulumi/pulumi/pull/5347 I'd recommend waiting a few days for node if that's what you're interested in, but happy to help get you set up if you don't mind handling the incoming changes. The short of it is to: 1. enlist in that branch 2.

cd ...pulumi/pulumi/sdk/node

make build && make install

(this will yarn link the pulumi pacakge globally 4. Start a node project somewhere and run

yarn link "@pulumi/pulumi"

Python and .NET will be started once we finish up node.

There's a few key differences between S3 and the Pulumi hosted backend: 1. S3 has no concurrency control. This is a big deal for your use case as you run the risk of automated deployments colliding and potentially corrupting state/making simultaneous edits to your infra. If you want to use the S3 backend, you'll have to implement locking around your deployments to guard against this. The pulumi backend has locking and concurrency control built in. 2. The pulumi backend is designed for software teams, and has features like stack access contorl, RBAC, built-in concurrency control, etc. 3. The pulumi backend has some interesting state storage capabilities that aren't in the other backends. You can refresh config from the last deployment, meaning you don't have to worry about storing it yourself. I've created a temporary vm manager that queries that pulumi backend for all ephemeral stacks matching time based criteria. This just kinda works and doesn't require me to maintain any state myself. It will work either way, but the S3 backend will require more manual effort on your part.

proud-pizza-80589

09/27/2020, 4:09 PM

Hmm, the concurrency is indeed a good selling point!

View count: 1