https://pulumi.com logo
Join Slack
Powered by
I'm now getting `"PROBLEM: Lambda internal error. ...
# aws
r
I'm now getting 
"PROBLEM: Lambda internal error. Please contact Lambda customer support."
in the LastProcessingResult output for the ESM
b
Can you post any more information?
r
Here is the lambda consumer function...
Copy code
export async function writeOnTopic(
  event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> {
  return {
    statusCode: 200,
    body: JSON.stringify(event)
  };
}
I also have 
numberOfNatGateways: 2
in the vpc config
and PRIVATE_SUBNETS = the 2 private subnet IDs from the vpc config
b
Are you seeing any invocations happening in the lambda?
in your lambda code can you remove the two types: the one for event and the one for the return type. It might be that the lambda is rejecting the messages because I don't think that 
APIGatewayProxyEvent
is the correct input type for this type of message
If you MUST have a type, use 
any
for now
r
gotcha let me try that really quick
What I'm testing now
Copy code
export async function writeOnTopic(
  event: any
): Promise<any> {
  return {
    statusCode: 200,
    body: JSON.stringify(event)
  };
}
Same thing 
PROBLEM: Lambda internal error. Please contact Lambda customer support.
b
It sounds like the MSK can't talk to the lambda. Are you sure that the lambda has the correct security group attached?
r
quick question the security group for the lambda isn't the same instance of security group for msk but they are both open...does it actually need to be the same instance of the security group for both?
b
The sg for the lambda needs to allow access from the security group of the MSK... or the IP address if you know it
r
the consumer lambda and msk both have separate security groups with this configuration
Copy code
const securityGroup = new aws.ec2.SecurityGroup('ciii-kafka-sg-2', {
    vpcId: vpc,
    ingress: [
      {
        fromPort: 0,
        toPort: 0,
        protocol: '-1',
        cidrBlocks: ['0.0.0.0/0']
      }
    ],
    egress: [
      {
        fromPort: 0,
        toPort: 0,
        protocol: '-1',
        cidrBlocks: ['0.0.0.0/0']
      }
    ]
  });
my thought was this should open it all up on both ends
b
I've done a quick search and it looks like you need to give access using an IAM role, not a security group: https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/
r
My role definition looks like this
Copy code
const lambdaConsumerRole = new aws.iam.Role('development-ciii-kafka-lambda-consumer-role-2', {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
      Service: '<http://lambda.amazonaws.com|lambda.amazonaws.com>'
    })
  });

  const policyAttachment = new aws.iam.PolicyAttachment('development-ciii-kafka-policy-attachment=2', {
    roles: [lambdaConsumerRole.name],
    policyArn: 'arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole'
  });
I was thinking that 
AWSLambdaMSKExecutionRole
should be enough
I set it here in the definition of the lambda
Copy code
const consumerLambda = new aws.lambda.CallbackFunction('ciii-development-consumer-lambda-5', {
    runtime: aws.lambda.NodeJS12dXRuntime,
    role: lambdaConsumerRole,
    callback: writeOnTopic,
    vpcConfig: {
      securityGroupIds: [securityGroup.id],
      subnetIds: privateSubnets.apply(s => s.map((x: {id: string}) => x.id))
    }
  });
b
So yes the lambda doesn't need to be in a VPC so it's definitely an IAM thing (snapshot from my link above):
r
I see...do you see anything wrong with the way I'm creating the lambda consumer role and defining the policy attachment?
b
I'm looking at that now
👍 1
No, that's how I'd do it
It might be worth removing
Copy code
vpcConfig: {
      securityGroupIds: [securityGroup.id],
      subnetIds: privateSubnets.apply(s => s.map((x: {id: string}) => x.id))
    }
from the lambda to remove it from the vpc. I don't know if it can't be in a vpc for it to work or it just doesn't need to be
r
ahh ok...let me remove it and try that...
b
One more thing... how many policy attachments do you have? Just one for the MSK policy?
r
yep
b
You'll need one to actually run the lambda (I know how silly that sounds, but it's an AWS thing).
Copy code
const policyAttachment2 = new aws.iam.PolicyAttachment('development-ciii-kafka-policy-attachment=3', {
    roles: [lambdaConsumerRole.name],
    policyArn: aws.iam.ManagedPolicies.AWSLambdaFullAccess
  });
when I'm doing testing I use the full access one like above
(by the way, we have strongly typed the managed policies)
We don't have one for MSK yet but for a lot of them we have them
r
ahh
Yea I didn't see the MSK policy in that list of constants
I'll add the lambda full access one and retry really quick....
b
Sorry, my mistake it should be 
aws.iam.ManagedPolicy.AWSLambdaFullAccess
(singular 
ManagedPolicy
)
I suspect it's the same, but we're deprecating the plural ones
r
👍 testing it now
Should I leave the securityGroup undefined? Looks like it's always under the vpcConfig and since I'm not defining a vpcConfig on the 
CallbackFunction
just remove it?
b
Are you talking about the lambda sg?
r
yep
just making sure I don't need a security group for the lambda
b
that AWS page says that the lambda doesn't need to be in a vpc for MSK to talk to it. If the lambda doesn't need access to anything outside of AWS or things like elasticache it doesn't need to be in a VPC and therefore can't have an SG
authorisation happens through IAM
r
gotcha ok testing...
same thing
b
One thing you could do (and I've done this before when I couldn't get something working just in Pulumi) is to follow the instructions on here: https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/ and then do pulumi import on the lambda and MSK to start with and work your way until you've got everything imported and then you can start changing things so you're referencing the resources in code rather than with the names
I've never seen that particular error message, but it sounds like your MSK instance can't talk to the lambda for whatever reason. Have you raised it with AWS Support?
r
I see...I'm opening a support ticket with them
One quick thing...when defining the MSK cluster I should only be using the private subnets right?
b
That's what AWS' recommendation is
so you should spread your MSK across the two subnets
Open in Slack
PreviousNext
Powered by