https://pulumi.com logo
Join Slack
Powered by
Hey is there a way to upload some code to an ec2 i...
# general
h
Hey is there a way to upload some code to an ec2 instance? I’m using userData but it’s throwing a gRPC error specifying that my file is larger than max
c
hmm cc @stocky-spoon-28903?
I think no?
h
What’s the typical approach towards spinning up a basic server with an app? I’ve tried ECS but as noted yesterday, I can’t connect to external services.
c
uh, are you doing this from inside a security group and vpc?
h
The ecs? I’m just creating a cloud.Service and not using ECS directly.
c
Hmm, @lemon-spoon-91807 knwos more about cloud.Service, but if you can’t connect to external services that sounds like you are launching into a VPC that has restrictive permissions.
oh when you say you CAN’T you mean that you’re not allowed to, or you are but something is stopping you.
l
could you clarify what you mean by: "upload some code to an ec2 instance? " 🙂
h
That’s what I figured but I’m not specifying the VPC or anything - just letting pulumi do it’s thing. And by can’t I mean I get a timeout when trying to GET a basic webpage (which works locally)
c
you’d deploy into a default vpc
in that case
you always deploy into a vpc.
h
@lemon-spoon-91807 I’m trying to upload a go app to an EC2 instance and getting 
8 RESOURCE_EXHAUSTED: grpc: received message larger than max (8762430 vs. 4194304)
Is the default VPC fairly open or?
c
depends on how it was set up
it can be, typically in a realistic corporate setting you’d really lock down or delete it.
h
I’m not setting up, that’s the thing. It looks like pulumi (or ECS) is automatically setting up the VPC for me.
c
I mean, it comes with your AWS account.
l
Hi Andy. Let's step back for a second 🙂
c
If you don’t specify a VPC, EC2 runs the instance inside the that default VPC
l
Could you clarify, at a higher level, what you're tryin to accomplish? Knowing htat, we can better help determine what the right approach is to go forward.
c
oh maybe cloud.service does
h
I’m trying to use pulumi to launch a site and a web scrapper (two services).
Trying to use cloud.Service prevents me from making external requests (hence the web scraper breaks since it can’t talk to the world)
l
Ok. Removing 'pulumi' for a moment, what would you normally be doing with AWS?
h
I then attempted to spin up an EC2 server and use userData or userDataBase64 to upload some code
I haven’t used ECS directly. But with EC2 I would spin up a server and just upload some code (SFTP or some other mechanism) which is what I’m trying to avoid.
l
Ok. So this is more a question of how to use ECS, then pulumi in particular. For ECS, the idea is generally that it's a 'container orchestrator'
i.e. you provide what you want to run in Docker containers, and you use ECS as the orchestration system that ensures that the right configuration of machiens are running your containers.
h
Shouldn’t the containers be able to access external services (make an HTTP request to google for example)?
l
it really depends on how you've configured your network.
so, for example, we have some really helpful APIs to help here in 
@pulumi/awsx
(note: the 
x
at the end of 
awsx
)
There, you can easily make some services, pointing at some containers, using ECS as the orchestrator. ECS then has two main strategies for managing instances you can use. 'Fargate', or 'EC2'
h
So you’re saying use the ecs api there
l
'EC2' gives a lot more control, but is a lot more work on your part. 'Fargate' takes away from control, but makes it a lot easier to just have things work.
h
What about 
@pulumi/cloud-aws
cloud.Service then? My understanding was that it’s an abstraction above the lower level stuff.
l
It is. But it will also hide a lot of stuff.
h
So by default it looks to create a restrictive VPC. Is that correct? And this prevents external access??
l
no. by default i beleive it is using your account's default-VPC
unless you've specified otherwise. a nd it will try to use the public subnets of that VPC by defualt if you have any
but i'm not sure what the configuration of your VPC is.
in pulumi/awsx you have a lot more control here. including specifying an entirely independent VPC to run things in.
h
How do I attach a new VPC to a cloud.Service? Or is that not possible?
l
pulumi/cloud was more an attempt to show how you could create cross-cloud attachments.
but when you really need to be able to configure the individual cloud stuff, packages like 
@pulumi/awsx
are hte recommended way to go.
How do I attach a new VPC to a cloud.Service? Or is that not possible?
For that, i would recommend using awsx. let me try to find an example
h
so 
@pulumi/cloud-aws
is fairly basic and takes a lot of (secure) defaults I’m assuming (hence no custom VPC’s)?
l
Here are some basic examples of making your own VPCs: https://github.com/pulumi/pulumi-awsx/blob/master/nodejs/awsx/examples/vpc/index.ts Here are examples of how you could create an ECS Service sitting on top of Fargate: https://github.com/pulumi/pulumi-awsx/blob/master/nodejs/awsx/examples/fargate/fargate.ts
I would put it more like this: cloud-aws is not an area of high investment 🙂
h
lol. Gotcha
now what about spinning up a raw EC2 server and uploading a binary and running it? It seems userData isn’t a way to go.
l
that isn't something i'm personally familiar with
my question would first start with: how would you normally do this with AWS's apis?
h
typically spin up a server and then SSH into it 😅
Then git clone, or download a release (curl) or similar
l
so... you could do all that if you wanted 🙂
it's more of a really low level way of doing things
i think more modern practices would be recommending you dockerize things
that said, you could cetainly use pulumi to create your EC2 server...
and then still ssh into it
h
ya. I’ve managed to create the server, just not upload data. I guess then use sftp (separate lib) to upload
l
that should still likely work.
h
But pulumi doesn’t provide any wrappers or conveniences in order to do so?
l
for that? no. But if you dockerize things, then definitely yes.
For example, we'll shell out to docker, build everything for you, upload your container to ECR (AWS's container registry), and point your services at those containers
see, for example:
Here, we show an example of creating a Service based off of some container that is built locally (from the 
./app
folder)
the Service will sit on fargate to manage instances automatically, and ECS will ensure you always have at least to isntances running.
and the Service will be exposed through an NLB to the internet through port 80.
Does that make sense?
The benefit of the Docker/containerized approach is taht it's very clear what you're running on these instances. as opposed to whatever random changes were made when ssh'ed in. this keeps things highly repeatable and managable
it's also much easier to test as you literally can just use the same containers locally that are in AWS.
h
ya. I’ve used docker before so I’m familiar with it (just not ECS directly). I’ll try playing around with awsx and avoid 
@pulumi/could-aws
I have one other thing to note. I’ve tried using pulumi to build docker images and it works great locally, but fails in github actions (it looks like pulumi uses a blocked docker command, can’t remember off the top of my head). My current solution is to have a separate dockerize action which uploads to ECR and then use that image in pulumi.
l
but fails in github actions (it looks like pulumi uses a blocked docker command, can’t remember off the top of my head)
h
Is there a plan to address this issue in the future?
l
hrmm... it works successfully for us in our CI. I can pull in someone else to help you with that
taggign @bitter-oil-46081
Matt, could you possibly help out Andy with https://pulumi-community.slack.com/team/UGR3PG0FR when you have some time?
h
thanks for the help @lemon-spoon-91807. Appreciate it!
b
@lemon-spoon-91807 The problem is that the execution environment that github actions run in disallow certain docker commands that 
@pulumi/docker
uses. Specifically, 
docker image inspect
is blocked in this environment, so https://github.com/pulumi/pulumi-docker/blob/ecddfc468c7baf63e61afbc8dc57ea06247daf13/sdk/nodejs/docker.ts#L360-L363 fails. I think we need to rework 
@pulumi/docker
to do something else when this command doesn't work.
h
ya, that’s the command!
l
I apologize. I completely misunderstood. I thought this was about running 
pulumi
in a CI scenario, not specifically the 'github actions' workflow.
Matt thinks we have a bug on this. If not, we'll open one.
h
no worries
b
I moved a bug we had in cloud (that andy opened, sorry for the delay, andy) to pulumi/docker: https://github.com/pulumi/pulumi-docker/issues/84
l
Can't guarantee we'll necessarily have a solution though. I'd have to understand the restrictions of 'github actions' and what (if any) recommendations there are for doing this sort of thing.
h
Not a problem. I’m sure you guys are busy
Thanks for the help @bitter-oil-46081 and @lemon-spoon-91807. Just want to say that I’m really enjoying pulumi
b
👍
l
That's great to hear!
30 Views
Open in Slack
PreviousNext
Powered by