s
w
cc @lemon-spoon-91807 for thoughts. Could you share a
du
of the contents of your zip? Curious what node_modules ended up getting packaged, and how much code ended up getting directly serialized? Ultimately, we should only be bringing in the things your code actually uses - so they size shouldn't be any larger than it would need to be if you zipped this up manually.
s
Here's my main function: queue.onEvent('send-to-stackdriver', new aws.lambda.CallbackFunction("stackdriver-writer", { memorySize: 512, // this is an excessive amount of RAM, but its running very slow with less. callback: (e) => { logger.logger(e, 'client-events') }, environment: { variables: { gcpKey: keyFile, projectId: projectId } }, policies: [ 'arnawsiam:awspolicy/service-role/AWSLambdaSQSQueueExecutionRole' ], runtime: "nodejs10.x" // required to use NodeJS 8.13+ runtime for GCP SDK } ))
and the logging function it calls: import * as Logging from "@google-cloud/logging"; export async function logger(event: any, logName = "client-events") { const keyFile = JSON.parse(
${process.env.gcpKey}
) const projectId =
${process.env.projectId}
const logging = new Logging.Logging({ projectId: projectId, credentials: { client_email: keyFile.client_email, private_key: keyFile.private_key }}); const log = logging.log(logName); const metadata = { resource: { type: "global" } }; event.Records.forEach(async (record: { body: {} | undefined; }) => { const entry = log.entry(metadata, record.body); await log.write(entry); console.log(
Logged: ${record}
); }); } just one dependency
l
hi. investigating
so we have a way to help out here.
Here's the way Pulumi works (and we desperately must documentthis somewhere).
1. when you serialize a JS function (like you have with
callback: (e) => { logger.logger(e, 'client-events') },
) by default we include all your node_modules brought in by your package.json.
This is clearly not optimal in many scenarios (but it makes things super simple for common cases).
To improve this, we allow you to do the following:
1. in your package.json file add a new
"pulumi": { ... }
section.
in there, add a
"runtimeDependencies": { ... }
where you list only the node_modules that you want to be sucked up into the cloud.
note: this will still need to be in your normal
"dependencies"
section so that tools like npm/yarn know about it and pull it down.
so, based on your description, it should be sufficient to just pull in
@google-cloud/logging
.
Note: this will impact all lambdas you create. If you want to exercise fine-grained control over your individual lambdas, you can do so when calling
new aws.lambda.CallbackFunction("stackdriver-writer", {
There is a
.codePathOptions
property that can be used to flexibly control things here.
@swift-painter-31084 Let me know if this helps you out!
s
That is awesome, I'll give that a try!
l
great!
s
So it went down to 23.8 MB from 23.9... How does this look: "dependencies": { "@google-cloud/language": "^3.2.1", "@google-cloud/logging": "^5.1.3", "@pulumi/aws": "^0.18.18", "@pulumi/awsx": "^0.18.6", "@pulumi/cloud": "^0.18.0", "@pulumi/gcp": "^0.18.11", "@pulumi/pulumi": "^0.17.21", "aws-sdk": "^2.489.0", "eslint": "^6.0.1", "npm": "^6.10.0", "typescript": "^3.5.2" }, "pulumi": { "runtimeDependencies": { "@google-cloud/logging": "^5.1.3" } }
l
seems reasonable
did it not work though? what's in the final uploaded zip?
oh. i see the problem
we respect this sectoin in referenced packages, but not your own.
s
looks like Typescript (should probably be a dev dependancy) and more.
l
darnit
for now, it's likely you'll have to setup the
codePathOptions: {     extraExcludePackages... }
yourself
😞
sorry
we should make this work for your package as well, i'll file bug
s
hey I was thinking about using something like Lerna for keeping all of our projects in one repo, but having multiple package.json's is that worth pursuing?
l
Hard to say. I don't have experience with lerna myself
s
How do I format the codePathOptions property in this case? I'm not able to find that in docs.
l
hrmm, are you using TypeScript/vscode?
it should be able to give intellisense.
s
yes
l
ok strange!
i believe you should be able to do
codePathOptions: { extraExcludePackages: ["@..."] }
if that doesn't work, i'll have to investigate
(where you fill in the part in the quotes btw)
i believe we literally check the names of the properties in your 'dependencies' section, i.e.
Copy code
"dependencies": {
   "@google-cloud/language": "^3.2.1",
so you'd put in
"@google-cloud/language"
👍 1
s
codePathOptions: { extraExcludePackages: [ "aws-sdk", "eslint", ... Worked perfectly!
l
ok great!
ideally we'll make the former method work as well
so you don'thave to have this big goop for all your functions