https://pulumi.com logo
Powered by
Title
r

rhythmic-student-36708

01/07/2021, 2:32 PM
I'm now getting 
"PROBLEM: Lambda internal error. Please contact Lambda customer support."
in the LastProcessingResult output for the ESM
b

brave-planet-10645

01/07/2021, 2:33 PM
Can you post any more information?
r

rhythmic-student-36708

01/07/2021, 2:34 PM
Yep here's a description of the issue https://pulumi-community.slack.com/archives/CRH5ENVDX/p1609964584123200
Here is the lambda consumer function...
export async function writeOnTopic(
  event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> {
  return {
    statusCode: 200,
    body: JSON.stringify(event)
  };
}
I also have 
numberOfNatGateways: 2
in the vpc config
and PRIVATE_SUBNETS = the 2 private subnet IDs from the vpc config
b

brave-planet-10645

01/07/2021, 2:51 PM
Are you seeing any invocations happening in the lambda?
in your lambda code can you remove the two types: the one for event and the one for the return type. It might be that the lambda is rejecting the messages because I don't think that 
APIGatewayProxyEvent
is the correct input type for this type of message
If you MUST have a type, use 
any
for now
r

rhythmic-student-36708

01/07/2021, 2:59 PM
gotcha let me try that really quick
What I'm testing now 
export async function writeOnTopic(
  event: any
): Promise<any> {
  return {
    statusCode: 200,
    body: JSON.stringify(event)
  };
}
Same thing 
PROBLEM: Lambda internal error. Please contact Lambda customer support.
b

brave-planet-10645

01/07/2021, 3:26 PM
It sounds like the MSK can't talk to the lambda. Are you sure that the lambda has the correct security group attached?
r

rhythmic-student-36708

01/07/2021, 3:27 PM
quick question the security group for the lambda isn't the same instance of security group for msk but they are both open...does it actually need to be the same instance of the security group for both?
b

brave-planet-10645

01/07/2021, 3:30 PM
The sg for the lambda needs to allow access from the security group of the MSK... or the IP address if you know it
r

rhythmic-student-36708

01/07/2021, 3:31 PM
the consumer lambda and msk both have separate security groups with this configuration 
const securityGroup = new aws.ec2.SecurityGroup('ciii-kafka-sg-2', {
    vpcId: vpc,
    ingress: [
      {
        fromPort: 0,
        toPort: 0,
        protocol: '-1',
        cidrBlocks: ['0.0.0.0/0']
      }
    ],
    egress: [
      {
        fromPort: 0,
        toPort: 0,
        protocol: '-1',
        cidrBlocks: ['0.0.0.0/0']
      }
    ]
  });
my thought was this should open it all up on both ends
b

brave-planet-10645

01/07/2021, 3:33 PM
I've done a quick search and it looks like you need to give access using an IAM role, not a security group: https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/
r

rhythmic-student-36708

01/07/2021, 3:33 PM
My role definition looks like this 
const lambdaConsumerRole = new aws.iam.Role('development-ciii-kafka-lambda-consumer-role-2', {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
      Service: '<http://lambda.amazonaws.com|lambda.amazonaws.com>'
    })
  });

  const policyAttachment = new aws.iam.PolicyAttachment('development-ciii-kafka-policy-attachment=2', {
    roles: [lambdaConsumerRole.name],
    policyArn: 'arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole'
  });
I was thinking that 
AWSLambdaMSKExecutionRole
should be enough
I set it here in the definition of the lambda 
const consumerLambda = new aws.lambda.CallbackFunction('ciii-development-consumer-lambda-5', {
    runtime: aws.lambda.NodeJS12dXRuntime,
    role: lambdaConsumerRole,
    callback: writeOnTopic,
    vpcConfig: {
      securityGroupIds: [securityGroup.id],
      subnetIds: privateSubnets.apply(s => s.map((x: {id: string}) => x.id))
    }
  });
b

brave-planet-10645

01/07/2021, 3:35 PM
So yes the lambda doesn't need to be in a VPC so it's definitely an IAM thing (snapshot from my link above):
r

rhythmic-student-36708

01/07/2021, 3:36 PM
I see...do you see anything wrong with the way I'm creating the lambda consumer role and defining the policy attachment?
b

brave-planet-10645

01/07/2021, 3:36 PM
I'm looking at that now
👍 1
No, that's how I'd do it
It might be worth removing 
vpcConfig: {
      securityGroupIds: [securityGroup.id],
      subnetIds: privateSubnets.apply(s => s.map((x: {id: string}) => x.id))
    }
from the lambda to remove it from the vpc. I don't know if it can't be in a vpc for it to work or it just doesn't need to be
r

rhythmic-student-36708

01/07/2021, 3:39 PM
ahh ok...let me remove it and try that...
b

brave-planet-10645

01/07/2021, 3:43 PM
One more thing... how many policy attachments do you have? Just one for the MSK policy?
r

rhythmic-student-36708

01/07/2021, 3:44 PM
yep
b

brave-planet-10645

01/07/2021, 3:45 PM
You'll need one to actually run the lambda (I know how silly that sounds, but it's an AWS thing).
const policyAttachment2 = new aws.iam.PolicyAttachment('development-ciii-kafka-policy-attachment=3', {
    roles: [lambdaConsumerRole.name],
    policyArn: aws.iam.ManagedPolicies.AWSLambdaFullAccess
  });
when I'm doing testing I use the full access one like above
(by the way, we have strongly typed the managed policies)
We don't have one for MSK yet but for a lot of them we have them
You can see the ones we do have here: https://github.com/pulumi/pulumi-aws/blob/master/sdk/nodejs/iam/managedPolicies.ts
r

rhythmic-student-36708

01/07/2021, 3:49 PM
ahh
Yea I didn't see the MSK policy in that list of constants
I'll add the lambda full access one and retry really quick....
b

brave-planet-10645

01/07/2021, 3:52 PM
Sorry, my mistake it should be 
aws.iam.ManagedPolicy.AWSLambdaFullAccess
(singular 
ManagedPolicy
)
From https://github.com/pulumi/pulumi-aws/blob/master/sdk/nodejs/types/enums/iam/index.ts
I suspect it's the same, but we're deprecating the plural ones
r

rhythmic-student-36708

01/07/2021, 3:54 PM
👍 testing it now
Should I leave the securityGroup undefined? Looks like it's always under the vpcConfig and since I'm not defining a vpcConfig on the 
CallbackFunction
just remove it?
b

brave-planet-10645

01/07/2021, 3:59 PM
Are you talking about the lambda sg?
r

rhythmic-student-36708

01/07/2021, 3:59 PM
yep
just making sure I don't need a security group for the lambda
b

brave-planet-10645

01/07/2021, 4:03 PM
that AWS page says that the lambda doesn't need to be in a vpc for MSK to talk to it. If the lambda doesn't need access to anything outside of AWS or things like elasticache it doesn't need to be in a VPC and therefore can't have an SG
authorisation happens through IAM
r

rhythmic-student-36708

01/07/2021, 4:03 PM
gotcha ok testing...
same thing
b

brave-planet-10645

01/07/2021, 4:24 PM
One thing you could do (and I've done this before when I couldn't get something working just in Pulumi) is to follow the instructions on here: https://aws.amazon.com/blogs/compute/using-amazon-msk-as-an-event-source-for-aws-lambda/ and then do pulumi import on the lambda and MSK to start with and work your way until you've got everything imported and then you can start changing things so you're referencing the resources in code rather than with the names
I've never seen that particular error message, but it sounds like your MSK instance can't talk to the lambda for whatever reason. Have you raised it with AWS Support?
r

rhythmic-student-36708

01/07/2021, 4:26 PM
I see...I'm opening a support ticket with them
One quick thing...when defining the MSK cluster I should only be using the private subnets right?
b

brave-planet-10645

01/07/2021, 5:31 PM
That's what AWS' recommendation is
so you should spread your MSK across the two subnets
#awsJoin Slack
Powered by