Hey is there a way to upload some code to an ec2 instance I

Question

Hey is there a way to upload some code to an ec2 instance I m using userData but it s throwing a gRPC error specifying that my file is larger than max

creamy-potato-29402 · Answer

hmm cc < stocky spoon 28903>

creamy-potato-29402 · Answer

I think no

hallowed-king-98702 · Answer

What s the typical approach towards spinning up a basic server with an app I ve tried ECS but as noted yesterday I can t connect to external services

creamy-potato-29402 · Answer

uh are you doing this from inside a security group and vpc

hallowed-king-98702 · Answer

The ecs I m just creating a cloud Service and not using ECS directly

creamy-potato-29402 · Answer

Hmm < lemon spoon 91807> knwos more about cloud Service but if you can t connect to external services that sounds like you are launching into a VPC that has restrictive permissions

creamy-potato-29402 · Answer

oh when you say you CAN T you mean that you re not allowed to or you are but something is stopping you

lemon-spoon-91807 · Answer

could you clarify what you mean by upload some code to an ec2 instance slightly smiling face

hallowed-king-98702 · Answer

That s what I figured but I m not specifying the VPC or anything just letting pulumi do it s thing And by can t I mean I get a timeout when trying to GET a basic webpage which works locally

creamy-potato-29402 · Answer

you d deploy into a default vpc

creamy-potato-29402 · Answer

in that case

creamy-potato-29402 · Answer

you always deploy into a vpc

hallowed-king-98702 · Answer

< lemon spoon 91807> I m trying to upload a go app to an EC2 instance and getting `8 RESOURCE EXHAUSTED grpc received message larger than max 8762430 vs 4194304 `

hallowed-king-98702 · Answer

Is the default VPC fairly open or

creamy-potato-29402 · Answer

depends on how it was set up

creamy-potato-29402 · Answer

it can be typically in a realistic corporate setting you d really lock down or delete it

hallowed-king-98702 · Answer

I m not setting up that s the thing It looks like pulumi or ECS is automatically setting up the VPC for me

creamy-potato-29402 · Answer

I mean it comes with your AWS account

lemon-spoon-91807 · Answer

Hi Andy Let s step back for a second slightly smiling face

creamy-potato-29402 · Answer

If you don t specify a VPC EC2 runs the instance inside the that default VPC

lemon-spoon-91807 · Answer

Could you clarify at a higher level what you re tryin to accomplish Knowing htat we can better help determine what the right approach is to go forward

creamy-potato-29402 · Answer

oh maybe cloud service does

hallowed-king-98702 · Answer

I m trying to use pulumi to launch a site and a web scrapper two services

hallowed-king-98702 · Answer

Trying to use cloud Service prevents me from making external requests hence the web scraper breaks since it can t talk to the world

lemon-spoon-91807 · Answer

Ok Removing pulumi for a moment what would you normally be doing with AWS

hallowed-king-98702 · Answer

I then attempted to spin up an EC2 server and use userData or userDataBase64 to upload some code

hallowed-king-98702 · Answer

I haven t used ECS directly But with EC2 I would spin up a server and just upload some code SFTP or some other mechanism which is what I m trying to avoid

lemon-spoon-91807 · Answer

Ok So this is more a question of how to use ECS then pulumi in particular For ECS the idea is generally that it s a container orchestrator

lemon-spoon-91807 · Answer

i e you provide what you want to run in Docker containers and you use ECS as the orchestration system that ensures that the right configuration of machiens are running your containers

hallowed-king-98702 · Answer

Shouldn t the containers be able to access external services make an HTTP request to google for example

lemon-spoon-91807 · Answer

it really depends on how you ve configured your network

lemon-spoon-91807 · Answer

so for example we have some really helpful APIs to help here in ` pulumi awsx`

lemon-spoon-91807 · Answer

note the `x` at the end of `awsx`

lemon-spoon-91807 · Answer

There you can easily make some services pointing at some containers using ECS as the orchestrator ECS then has two main strategies for managing instances you can use Fargate or EC2

hallowed-king-98702 · Answer

So you re saying use the ecs api there

lemon-spoon-91807 · Answer

EC2 gives a lot more control but is a lot more work on your part Fargate takes away from control but makes it a lot easier to just have things work

hallowed-king-98702 · Answer

What about ` pulumi cloud aws` cloud Service then My understanding was that it s an abstraction above the lower level stuff

lemon-spoon-91807 · Answer

It is But it will also hide a lot of stuff

hallowed-king-98702 · Answer

So by default it looks to create a restrictive VPC Is that correct And this prevents external access

lemon-spoon-91807 · Answer

no by default i beleive it is using your account s default VPC

lemon-spoon-91807 · Answer

unless you ve specified otherwise a nd it will try to use the public subnets of that VPC by defualt if you have any

lemon-spoon-91807 · Answer

but i m not sure what the configuration of your VPC is

lemon-spoon-91807 · Answer

in pulumi awsx you have a lot more control here including specifying an entirely independent VPC to run things in

hallowed-king-98702 · Answer

How do I attach a new VPC to a cloud Service Or is that not possible

lemon-spoon-91807 · Answer

pulumi cloud was more an attempt to show how you could create cross cloud attachments

lemon-spoon-91807 · Answer

but when you really need to be able to configure the individual cloud stuff packages like ` pulumi awsx` are hte recommended way to go

lemon-spoon-91807 · Answer

gt How do I attach a new VPC to a cloud Service Or is that not possible

lemon-spoon-91807 · Answer

For that i would recommend using awsx let me try to find an example

hallowed-king-98702 · Answer

so ` pulumi cloud aws` is fairly basic and takes a lot of secure defaults I m assuming hence no custom VPC s

lemon-spoon-91807 · Answer

Here are some basic examples of making your own VPCs Here are examples of how you could create an ECS Service sitting on top of Fargate

lemon-spoon-91807 · Answer

I would put it more like this cloud aws is not an area of high investment slightly smiling face

hallowed-king-98702 · Answer

lol Gotcha

hallowed-king-98702 · Answer

now what about spinning up a raw EC2 server and uploading a binary and running it It seems userData isn t a way to go

lemon-spoon-91807 · Answer

that isn t something i m personally familiar with

lemon-spoon-91807 · Answer

my question would first start with how would you normally do this with AWS s apis

hallowed-king-98702 · Answer

typically spin up a server and then SSH into it sweat smile

hallowed-king-98702 · Answer

Then git clone or download a release curl or similar

lemon-spoon-91807 · Answer

so you could do all that if you wanted slightly smiling face

lemon-spoon-91807 · Answer

it s more of a really low level way of doing things

lemon-spoon-91807 · Answer

i think more modern practices would be recommending you dockerize things

lemon-spoon-91807 · Answer

that said you could cetainly use pulumi to create your EC2 server

lemon-spoon-91807 · Answer

and then still ssh into it

hallowed-king-98702 · Answer

ya I ve managed to create the server just not upload data I guess then use sftp separate lib to upload

lemon-spoon-91807 · Answer

that should still likely work

hallowed-king-98702 · Answer

But pulumi doesn t provide any wrappers or conveniences in order to do so

lemon-spoon-91807 · Answer

for that no But if you dockerize things then definitely yes

lemon-spoon-91807 · Answer

For example we ll shell out to docker build everything for you upload your container to ECR AWS s container registry and point your services at those containers

lemon-spoon-91807 · Answer

see for example

lemon-spoon-91807 · Answer

lemon-spoon-91807 · Answer

Here we show an example of creating a Service based off of some container that is built locally from the ` app` folder

lemon-spoon-91807 · Answer

the Service will sit on fargate to manage instances automatically and ECS will ensure you always have at least to isntances running

lemon-spoon-91807 · Answer

and the Service will be exposed through an NLB to the internet through port 80

lemon-spoon-91807 · Answer

Does that make sense

lemon-spoon-91807 · Answer

The benefit of the Docker containerized approach is taht it s very clear what you re running on these instances as opposed to whatever random changes were made when ssh ed in this keeps things highly repeatable and managable

lemon-spoon-91807 · Answer

it s also much easier to test as you literally can just use the same containers locally that are in AWS

hallowed-king-98702 · Answer

ya I ve used docker before so I m familiar with it just not ECS directly I ll try playing around with awsx and avoid ` pulumi could aws`

hallowed-king-98702 · Answer

I have one other thing to note I ve tried using pulumi to build docker images and it works great locally but fails in github actions it looks like pulumi uses a blocked docker command can t remember off the top of my head My current solution is to have a separate dockerize action which uploads to ECR and then use that image in pulumi

lemon-spoon-91807 · Answer

gt but fails in github actions it looks like pulumi uses a blocked docker command can t remember off the top of my head

hallowed-king-98702 · Answer

Is there a plan to address this issue in the future

lemon-spoon-91807 · Answer

hrmm it works successfully for us in our CI I can pull in someone else to help you with that

lemon-spoon-91807 · Answer

taggign < bitter oil 46081>

lemon-spoon-91807 · Answer

Matt could you possibly help out Andy with <https pulumi community slack com team UGR3PG0FR> when you have some time

hallowed-king-98702 · Answer

thanks for the help < lemon spoon 91807> Appreciate it

bitter-oil-46081 · Answer

< lemon spoon 91807> The problem is that the execution environment that github actions run in disallow certain docker commands that ` pulumi docker` uses Specifically `docker image inspect` is blocked in this environment so fails I think we need to rework ` pulumi docker` to do something else when this command doesn t work

hallowed-king-98702 · Answer

ya that s the command

lemon-spoon-91807 · Answer

I apologize I completely misunderstood I thought this was about running `pulumi` in a CI scenario not specifically the github actions workflow

lemon-spoon-91807 · Answer

Matt thinks we have a bug on this If not we ll open one

hallowed-king-98702 · Answer

no worries

bitter-oil-46081 · Answer

I moved a bug we had in cloud that andy opened sorry for the delay andy to pulumi docker <https github com pulumi pulumi docker issues 84>

lemon-spoon-91807 · Answer

Can t guarantee we ll necessarily have a solution though I d have to understand the restrictions of github actions and what if any recommendations there are for doing this sort of thing

hallowed-king-98702 · Answer

Not a problem I m sure you guys are busy

hallowed-king-98702 · Answer

Thanks for the help < bitter oil 46081> and < lemon spoon 91807> Just want to say that I m really enjoying pulumi

bitter-oil-46081 · Answer

+1

lemon-spoon-91807 · Answer

That s great to hear