https://pulumi.com logo
Docs
Join the conversationJoin Slack
Channels
announcements
automation-api
aws
azure
blog-posts
built-with-pulumi
cloudengineering
cloudengineering-support
content-share
contribex
contribute
docs
dotnet
finops
general
getting-started
gitlab
golang
google-cloud
hackathon-03-19-2020
hacktoberfest
install
java
jobs
kubernetes
learn-pulumi-events
linen
localstack
multi-language-hackathon
office-hours
oracle-cloud-infrastructure
plugin-framework
pulumi-cdk
pulumi-crosscode
pulumi-deployments
pulumi-kubernetes-operator
pulumi-service
pulumiverse
python
registry
status
testingtesting123
testingtesting321
typescript
welcome
workshops
yaml
Powered by Linen
automation-api
  • l

    lemon-agent-27707

    09/02/2020, 3:18 PM
    set the channel topic: šŸ­ Happy Automating! šŸ­ Full announcement, docs, details linked here: https://github.com/pulumi/pulumi/issues/3901#issuecomment-685803282
  • l

    lemon-agent-27707

    09/02/2020, 3:19 PM
    Full announcement here: https://github.com/pulumi/pulumi/issues/3901#issuecomment-685803282
  • l

    lemon-agent-27707

    09/02/2020, 3:19 PM
    A list of my favorite resources to get an overview and get started: • GoDocĀ with comprehensive examples across the API surface area • Automation API Examples: a repo with end to end walkthroughs for multiple facets includingĀ inline programs,Ā git workflows,Ā inline/cli hybrid programs, andĀ multi-stack orchestration • TheĀ source codeĀ for Automation API • halloumi: a prototype of a heroku-like tools that shows what sort of powerful abstractions can be built when infrastructure is just another part of your application code. • TheĀ Pulumi Kubernetes Operator: a real world example of exposing Pulumi behind an API.
    šŸŽ‰ 9
  • l

    lemon-agent-27707

    09/02/2020, 10:23 PM
    šŸ‘‹ Open invitation for anyone who's taken the time to kick the tires to schedule a 30 minute zoom call to provide feedback: https://calendly.com/evan-148/30min We'd love to hear from you!
  • l

    lemon-agent-27707

    09/03/2020, 11:48 PM
    šŸ‘‹ Another open invitation. I'll help you setup automation API in your project: https://calendly.com/evan-148/30min
  • l

    lemon-agent-27707

    09/04/2020, 2:32 PM
    A great video from @faint-table-42725 showing an overview of automation api in action:

    https://www.youtube.com/watch?v=m92HpG-JlUs&ab_channel=PulumiTVā–¾

  • l

    lemon-agent-27707

    09/04/2020, 11:42 PM
    @chilly-rainbow-79265 @important-appointment-55126 I put together an example of Automation API for provisioning temporary VMs. You'll probably have to make some adjustments to parameterize the VMs to your specifications but I think this should be a good starting point. It's a higher level CLI called
    vmgr
    that has two functions: •
    vmgr add
    creates a new VM (webserver in this instance) and prints out the public IP. Given that this is just plain Go code, you could extend it to email VM credentials to someone, put them in a queue somewhere, etc. •
    vmgr cron <expiration>
    ie `vmgr cron 5d`: runs a minutely job scanning for VMs older than the specified expiration. It will retry on failures, etc. You could run this on your laptop, or on a VM/container somewhere https://github.com/EvanBoyle/automation-api-examples/tree/main/vm_manager_azure
    i
    c
    t
    • 4
    • 5
  • c

    clever-byte-21551

    09/06/2020, 11:07 AM
    Hey I've been trying to play with the new Automation API and I tried running one of the example and I get the following error
    Successfully setup workspace
    Installing the AWS plugin
    Successfully installed AWS plugin
    Logged user: yarin
    Failed to create or select stack: failed to create stack: code: 255
    , stdout: 
    , stderr: error: stack names may not contain slashes
    
    : exit status 255: failed to select stack: code: 255
    , stdout: 
    , stderr: error: no stack named 'yarin/project-name/stack-name' found
    
    : exit status 255
  • c

    clever-byte-21551

    09/06/2020, 11:07 AM
    the ā€œstack name must not contain slashesā€ comes from the CLI tool (v2.9.2)
  • c

    clever-byte-21551

    09/06/2020, 11:08 AM
    I'm creating a new stack using:
    fqsn := auto.FullyQualifiedStackName(user, projectName, stackName)
    s, err := auto.NewStack(ctx, fqsn, w)
  • c

    clever-byte-21551

    09/06/2020, 11:08 AM
    What am I missing?
  • c

    clever-byte-21551

    09/06/2020, 12:22 PM
    After some debugging it seems that I can’t use a local backend ? It requires some changes in the CLI / automation so it would work
  • c

    clever-byte-21551

    09/06/2020, 12:22 PM
    @lemon-agent-27707 when you are up please tell me if there’s a workaround or this is intended
    l
    e
    • 3
    • 5
  • c

    clever-byte-21551

    09/07/2020, 5:07 PM
    How should we handle recovery from interrupted operations using the new Automation API? Even in the CLI approach the support to mitigate this automatically is done very poorly.
  • l

    lemon-agent-27707

    09/07/2020, 7:10 PM
    Yes a pulumi cancel is the function that will stop the update. This is currently missing from the automation API, I'll put it in the queue for tomorrow
  • c

    clever-byte-21551

    09/07/2020, 7:21 PM
    Also we should consider using custom exit codes from the pulumi CLI to detect certain errors like ā€œinterrupted while creatingā€
  • l

    lemon-agent-27707

    09/07/2020, 7:44 PM
    Yes, distinct error codes is a long-term project. The error design was intentional in allowing that to evolve over time. The current set of errors are opaque and can be updated to use the more reliable error codes as they are introduced. If you have other error cases you'd like to detect please open issues.
  • l

    lemon-agent-27707

    09/07/2020, 7:44 PM
    https://github.com/pulumi/pulumi/blob/master/sdk/go/x/auto/errors.go#L44
  • e

    enough-winter-19192

    09/09/2020, 10:14 PM
    Wrapping the pulumi CLI with the automation-api. Working nicely so far. Main issue so far is the binary size is quite large "~480mb". Is there any way around this? I've thought about breaking out local workspaces into other languages (Nodejs) to avoid importing
    <http://github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1|github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1>
    , but looks like
    /x/auto
    depends on this.
  • e

    enough-winter-19192

    09/09/2020, 10:37 PM
    My bad, I was wrong about the dependency graph. Removing imports for
    kubernetes/core
    brought the file size down to 61M. Going to go with a hybrid approach
    l
    • 2
    • 1
  • b

    broad-dog-22463

    09/10/2020, 9:18 PM
    hey all - just thought i'd let you know in here that we just launched support for non-default secrets providers in v2.10.0 for the automation api
    c
    • 2
    • 1
  • b

    broad-dog-22463

    09/10/2020, 9:19 PM
    this means you can use awskms / azurekeyvault / gcpkms / hashivault or passphrase
  • b

    bitter-dentist-28132

    09/11/2020, 4:36 PM
    does the fact that the automation api is written in go mean it can only load projects written in go?
    l
    • 2
    • 8
  • v

    victorious-xylophone-55816

    09/12/2020, 5:31 PM
    I don't have anything to contribute, I just wanted to say the Automation API is neat and seems really useful. Particularly enjoyed reading this example repo:
    halloumi: a prototype of a heroku-like tools that shows what sort of powerful abstractions can be built when infrastructure is just another part of your application code.
    šŸ˜›artypus: 1
    l
    • 2
    • 2
  • l

    lemon-agent-27707

    09/16/2020, 6:41 PM
    A bunch of new features just landed in Automation API. This will require updating your CLI to v2.10.1. You can see the full changelog here, but some highlights: • Stack name validations have been relaxed at the Automation API layer. This change was made to enable local backends. • Output from Up/Refresh/Preview can now be streamed to a list of
    io.Writer
    example • Support added for private git repo authentication: examples • Helper functions to extract permalink from update result: example • cancel/import/export recovery APIs: example • upsert stack helper methods (create or select): example • workspace scoped environment variables (useful for setting passphrase envs for local backends): example • Support for non-default secrets providers: example Docs and examples are up to date as well: • godoc • automation-api-examples • halloumi šŸ­ Thanks for all of the great feedback so far! šŸ­
    šŸ¤– 3
    šŸŽ‰ 5
  • c

    clever-byte-21551

    09/22/2020, 6:22 AM
    Not sure if this is automation API related but I have a stack where calling refresh just stucks. I added logging of the refresh command output, it list all the resources needed for refresh and after that it’s just stuck on
    @ refreshing....
    for many minutes
    l
    w
    • 3
    • 12
  • c

    clever-byte-21551

    09/23/2020, 6:00 AM
    Is there a way to sync my plugins with the current SDKs I’m using? Currently I have to manually sync my
    go.mod
    with the hard-coded plugins versions i’m installing via the automation API
    l
    • 2
    • 1
  • l

    limited-rainbow-51650

    09/24/2020, 2:51 PM
    Using the automation API, I want to execute a Git based Pulumi NodeJS project. After using
    auto.UpsertStackRemoteSource
    , how must I trigger
    npm install
    to retrieve all dependencies before invoking
    stack.Up(…)
    g
    • 2
    • 2
  • r

    red-energy-90711

    09/24/2020, 3:05 PM
    Hi, with automation-api would be possible to run inside a serverless function like AWS lambda or Azure Functions. I mean receive args and then run the provisioning process without "pulumi up" command.
    l
    g
    p
    • 4
    • 16
  • p

    proud-pizza-80589

    09/25/2020, 7:18 PM
    @lemon-agent-27707 the automation api looks awesome. With a bit more context on where i’m investigating to use it. We have built a BPaaS (blockchain platform as a service https://settlemint.com/blockchain-platform-features/ #buzzwordbingo) which is a large collection of microservices to create and manage k8s clusters on AWS/Google/Azure and deploy on top of this blockchain nodes, api’s, management ui’s etc. Currently it is all hand coded TS + Helmfile + Helmcharts. But we are reworking some of them and I kind of feel like i can ditch all my cloud provider code (and gain a lot more clouds) and helmfile with pulumi. All clusters are set up on demand and settings etc are stored in a database. I think i can cut thousands of lines of code by building on pulumi. Just one thing i need to investigate and that is state management. each cluster/services set can be a stack (we have a unique name for each). What are the benefits of using the could hosted TEAM PRO over an s3 backend?
    šŸŽ‰ 1
    l
    • 2
    • 4
Powered by Linen
Title
p

proud-pizza-80589

09/25/2020, 7:18 PM
@lemon-agent-27707 the automation api looks awesome. With a bit more context on where i’m investigating to use it. We have built a BPaaS (blockchain platform as a service https://settlemint.com/blockchain-platform-features/ #buzzwordbingo) which is a large collection of microservices to create and manage k8s clusters on AWS/Google/Azure and deploy on top of this blockchain nodes, api’s, management ui’s etc. Currently it is all hand coded TS + Helmfile + Helmcharts. But we are reworking some of them and I kind of feel like i can ditch all my cloud provider code (and gain a lot more clouds) and helmfile with pulumi. All clusters are set up on demand and settings etc are stored in a database. I think i can cut thousands of lines of code by building on pulumi. Just one thing i need to investigate and that is state management. each cluster/services set can be a stack (we have a unique name for each). What are the benefits of using the could hosted TEAM PRO over an s3 backend?
šŸŽ‰ 1
Follow up question, how would I use your PR to try it out?
l

lemon-agent-27707

09/27/2020, 1:11 PM
Hey @proud-pizza-80589, glad to hear this is going to help solve some of your problems! Deploying infrastructure on the fly is exactly what we had in mind when designing the api, so your use case sounds perfect. Trying it out depends on what language you'd like to use. Go is published in alpha and available to use today. There is a full examples repo here that has multiple sample projects: https://github.com/evanboyle/automation-api-examples If you're looking for typescript, we do have a branch but there will be a bunch of changes over the next few days before it goes out for final code review: https://github.com/pulumi/pulumi/pull/5347 I'd recommend waiting a few days for node if that's what you're interested in, but happy to help get you set up if you don't mind handling the incoming changes. The short of it is to: 1. enlist in that branch 2.
cd ...pulumi/pulumi/sdk/node
3.
make build && make install
(this will yarn link the pulumi pacakge globally 4. Start a node project somewhere and run
yarn link "@pulumi/pulumi"
Python and .NET will be started once we finish up node.
There's a few key differences between S3 and the Pulumi hosted backend: 1. S3 has no concurrency control. This is a big deal for your use case as you run the risk of automated deployments colliding and potentially corrupting state/making simultaneous edits to your infra. If you want to use the S3 backend, you'll have to implement locking around your deployments to guard against this. The pulumi backend has locking and concurrency control built in. 2. The pulumi backend is designed for software teams, and has features like stack access contorl, RBAC, built-in concurrency control, etc. 3. The pulumi backend has some interesting state storage capabilities that aren't in the other backends. You can refresh config from the last deployment, meaning you don't have to worry about storing it yourself. I've created a temporary vm manager that queries that pulumi backend for all ephemeral stacks matching time based criteria. This just kinda works and doesn't require me to maintain any state myself. It will work either way, but the S3 backend will require more manual effort on your part.
p

proud-pizza-80589

09/27/2020, 4:09 PM
Hmm, the concurrency is indeed a good selling point!
View count: 1