Is it possible to breakout the api gateway definition from the endpoint definitions? Have an "infra" project to manage usage plans and authorizers, etc. but then have a "services" project that references the stack and adds apigateway endpoints on top of the infra stack?

yep! using stack references https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences

I saw this, just not sure how to break apart the api gateway endpoints into separate services.