Channels
pulumi-ai
package-authoring
pulumi-cdk
oracle-cloud-infrastructure
learn-pulumi-events
linen
registry
built-with-pulumi
pulumi-cloud
contribex
testingtesting321
hacktoberfest
cloudengineering
yaml
pulumi-crosscode
content-share
finops
multi-language-hackathon
office-hours
workshops
blog-posts
localstack
welcome
gitlab
pulumi-kubernetes-operator
jobs
aws
pulumi-deployments
dotnet
golang
announcements
java
pulumiverse
python
install
getting-started
cloudengineering-support
testingtesting123
hackathon-03-19-2020
typescript
google-cloud
contribute
azure
kubernetes
docs
automation-api
status
general
Title
g
glamorous-jelly-86558
10/20/2021, 2:35 AMIdentify all non-Pulumi managed resources in an AWS Account?
Our DevOps team is somewhat new to Pulumi and occasionally find it easier to manually create certain resources in the AWS console to speed up the iteration cycle. Most of the time they also manually remove these resources when they are done. Sometimes, though, there will be a leftover resource that shouldn't exist.
So is there any way to do a diff on an AWS account and see what resources exist that are not managed by Pulumi?
l
little-cartoon-10569
10/20/2021, 2:35 AMNo. There is no way, even in AWS, to list all resources across all services.
g
glamorous-jelly-86558
10/20/2021, 2:36 AMAhhh... bummer. If given some type of scope, like databases or cidr blocks or s3 buckets... would it be any more feasible?
l
little-cartoon-10569
10/20/2021, 2:37 AMYes, but you'd have to write the code yourself, using the AWS SDK.
I know it sounds trite, but if "losing" resources is a genuine issue, the correct solution is to remove console access and programmatic access from all members of this team.
g
glamorous-jelly-86558
10/20/2021, 2:38 AMGreat... so the general code for that would be...
1. Use Pulumi to output the managed resources...
2. Use the AWS SDK to enumerate all resources of given types...
3. Manually write some diffing layer
... does that sound about right?
l
little-cartoon-10569
10/20/2021, 2:38 AMIf there isn't reviewed, accepted and merged code that created the resource, then the resource won't exist.
Yes that would work. It wouldn't be too hard, since you've got
pulumi state --show-urns👍 1
g
glamorous-jelly-86558
10/20/2021, 2:39 AMThanks a bunch!
👍 1
b
billowy-army-68599
10/20/2021, 2:46 AM@glamorous-jelly-86558 you could scrape the API with the AWS SDK and then use
importbut I agree with tenwit, having the ability to create resources in the console is going to create chaos down the line unfortunately, I've been there myself
g
glamorous-jelly-86558
10/20/2021, 5:39 AMFor the record: I agree. We have some locked down environments that everything needs to be reviewed and deployed through CI and IaC. But we have some mixed mode environments and I was mainly curious if it was possible.
It could also be helpful as a tool for identifying if there was a breach and if there were any resources created by a malicious attacker.
b
bumpy-grass-54508
10/20/2021, 1:50 PMIt isn't airtight but you could also make sure your IaC / CD use default tags on any resource that takes tags - include things like ManagedBy=Pulumi and some build ID or git sha sort of thing. And mix in some AWS Config tagging policies could get you part of the way there. not immune to malicious spoofing and not every resource has tags, but makes browsing the console easier to tell what is and isn't pulumi
m
millions-umbrella-34765
10/20/2021, 3:55 PM@glamorous-jelly-86558 You may want to look at https://steampipe.io/ as a tool to query for AWS resources.
🙌 1
l
little-cartoon-10569
10/20/2021, 8:54 PM@glamorous-jelly-86558 Look what I just found... https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/resource-operations-list.html
🙌 1
It doesn't cover all resources (yet) but it looks like a big step in the right direction...
g
glamorous-jelly-86558
10/20/2021, 9:22 PMThis community is amazing.
🕺 1
l
little-cartoon-10569
10/20/2021, 9:25 PMI'm playing with the the cloud control API right now. It's nowhere near fit for solving this problem right now. Loads of stuff isn't supported.
For example: there is support for listing NACLs, but no support for listing subnets. And there is no support of any kind for EC2 instances....
b
billowy-army-68599
10/21/2021, 1:22 AM@little-cartoon-10569 the AWS native provider is driven by the cloud control API. It'll add more resources as time goes on
l
little-cartoon-10569
10/21/2021, 2:51 AMYep I'm following it 🙂 The same AWS API can help solve this problem, eventually. Though afaik there will never be a "discover all resources" API. You'll need to script "discover all resources of type X" for all values of X.
Fortunately, there's another API for discovering all values of X (cloudformation).
However, it doesn't seem to include the "list" feature. You can limit to types that are FULLY_MUTABLE or IMMUTABLE, but neither of those 100% overlaps with "listable".