Hi, folks! I’m having a devil of a time determini...
# aws
e
Hi, folks! I’m having a devil of a time determining which missing AWS IAM action(s) are causing a fail when I run
pulumi up
or
pulumi destroy
. Is there any way to log the exact AWS API request or AWS error reply ? Pulumi gives me an error like “*Error reading new Lambda permissions: AccessDeniedException: status code: 403, request id: 22b39a15-7ff2-4fda-abf4-e7b057e073ef*” , but I’m looking for something to tell me which IAM action(s) I need to add to my IAM role (such as “`lambda:listFunctions`”). I don’t want to just use “`lambda:*`” for security reasons. I don’t know if the vague error message is coming from AWS or Pulumi. Any ideas? (-v=9 did not log the details I seek.) Thanks!
l
CloudWatch. Check for UnauthorizedApiCall.
You may even be able to send all those to a Slack channel or something, for easier and faster investigation.
e
Thanks for the swift reply! I’m familiar with CloudWatch, but not with UnauthorizedApiCall. I’ll do some research into that, guessing it’s something that has to be set up to trace API calls on our account. I’m trying to avoid posting all my errors - if I did, I’d be doing it all day because after getting around one error, another one pops up. My question is more general so I can help myself. Intuitively it seems that if a failure occurs on the AWS side, AWS would send the reason for failure, and Pulumi would log it. In the past, I use to get more lengthy and informative errors displayed (sometimes would even state the exact action that was lacking), but maybe they went away when I upgrade Pulumi, or maybe those were just for specific resources.
r
You might need to use
--logflow
along with
--logtostderr -v=9
e
I will try that …
Hmm… did not help much. Lowest level error I’m getting in the verbosity is the same as I’m getting from standard Pulumi interactive output. Seems like no level of logging reveals what precisely AWS is complaining about. Frustrating, because if I were using the AWS CLI, I usually get something back that gives me a clue. Also, --logflow wrote a bunch of stuff to stdout, which makes it hard to see the standard Pulumi outout. I still think I’m must be missing something here - AWS rejects a request; what error message did AWS return? Don’t know how other users are getting by without this basic feature. But thank you for your ideas!
l
UnauthorizedApiCall is just a normal error that gets logged to CloudWatch in the normal way. You don't have to turn it on or anything.
I use this in my loggroup query:
{ $.eventType = "AwsApiCall" && $.errorCode = "AccessDenied" }
. You can use
{ $.eventType = "AwsApiCall" && $.errorCode = "*" }
and narrow it from there.
e
Is it supposed to be under Log groups?
I see the logs for our custom apps, Lambdas, Glue, etc. I don’t see anything (or remember ever seeing anything) that looks like an automatic log for our API calls. That’s why I’m suspecting it’s something that needs to be set up somewhere.
Doing a bit more digging, looks like there’s an “Insights” page that let’s one run log group queries. But, one still must select a log group, so one must still have a log group that logs the API calls. Which I don’t think I have, unless someone can tell me the name of it? Thanks! 🤔 Of course, now this thread is far removed from Pulumi, so no need to respond. Not trying to spam up the channel with non-Pulumi side-discussions. :)
Doing more digging - it looks like maybe this is indeed something that has to be enabled via CloudTrail Insights. Even then, the docs make it sound like only write events get logged, incurs extra charges, and could take up to 36 hours t o see ’em. Not going to work for me. 😆 I guess I’ll just go back to trial-and-error. But thank you, everyone, for your assistance.
f
@early-rose-86563 this was 2 years ago but were you able to fix it? I am having the exact same error.