https://pulumi.com logo
#general
Title
# general
h

hallowed-king-98702

03/07/2019, 10:50 PM
Hey is there a way to upload some code to an ec2 instance? I’m using userData but it’s throwing a gRPC error specifying that my file is larger than max
c

creamy-potato-29402

03/07/2019, 10:51 PM
hmm cc @stocky-spoon-28903?
I think no?
h

hallowed-king-98702

03/07/2019, 10:55 PM
What’s the typical approach towards spinning up a basic server with an app? I’ve tried ECS but as noted yesterday, I can’t connect to external services.
c

creamy-potato-29402

03/07/2019, 11:00 PM
uh, are you doing this from inside a security group and vpc?
h

hallowed-king-98702

03/07/2019, 11:01 PM
The ecs? I’m just creating a cloud.Service and not using ECS directly.
c

creamy-potato-29402

03/07/2019, 11:02 PM
Hmm, @lemon-spoon-91807 knwos more about cloud.Service, but if you can’t connect to external services that sounds like you are launching into a VPC that has restrictive permissions.
oh when you say you CAN’T you mean that you’re not allowed to, or you are but something is stopping you.
l

lemon-spoon-91807

03/07/2019, 11:03 PM
could you clarify what you mean by: "upload some code to an ec2 instance? " 🙂
h

hallowed-king-98702

03/07/2019, 11:04 PM
That’s what I figured but I’m not specifying the VPC or anything - just letting pulumi do it’s thing. And by can’t I mean I get a timeout when trying to GET a basic webpage (which works locally)
c

creamy-potato-29402

03/07/2019, 11:04 PM
you’d deploy into a default vpc
in that case
you always deploy into a vpc.
h

hallowed-king-98702

03/07/2019, 11:04 PM
@lemon-spoon-91807 I’m trying to upload a go app to an EC2 instance and getting
8 RESOURCE_EXHAUSTED: grpc: received message larger than max (8762430 vs. 4194304)
Is the default VPC fairly open or?
c

creamy-potato-29402

03/07/2019, 11:05 PM
depends on how it was set up
it can be, typically in a realistic corporate setting you’d really lock down or delete it.
h

hallowed-king-98702

03/07/2019, 11:06 PM
I’m not setting up, that’s the thing. It looks like pulumi (or ECS) is automatically setting up the VPC for me.
c

creamy-potato-29402

03/07/2019, 11:06 PM
I mean, it comes with your AWS account.
l

lemon-spoon-91807

03/07/2019, 11:06 PM
Hi Andy. Let's step back for a second 🙂
c

creamy-potato-29402

03/07/2019, 11:07 PM
If you don’t specify a VPC, EC2 runs the instance inside the that default VPC
l

lemon-spoon-91807

03/07/2019, 11:07 PM
Could you clarify, at a higher level, what you're tryin to accomplish? Knowing htat, we can better help determine what the right approach is to go forward.
c

creamy-potato-29402

03/07/2019, 11:07 PM
oh maybe cloud.service does
h

hallowed-king-98702

03/07/2019, 11:07 PM
I’m trying to use pulumi to launch a site and a web scrapper (two services).
Trying to use cloud.Service prevents me from making external requests (hence the web scraper breaks since it can’t talk to the world)
l

lemon-spoon-91807

03/07/2019, 11:08 PM
Ok. Removing 'pulumi' for a moment, what would you normally be doing with AWS?
h

hallowed-king-98702

03/07/2019, 11:08 PM
I then attempted to spin up an EC2 server and use userData or userDataBase64 to upload some code
I haven’t used ECS directly. But with EC2 I would spin up a server and just upload some code (SFTP or some other mechanism) which is what I’m trying to avoid.
l

lemon-spoon-91807

03/07/2019, 11:10 PM
Ok. So this is more a question of how to use ECS, then pulumi in particular. For ECS, the idea is generally that it's a 'container orchestrator'
i.e. you provide what you want to run in Docker containers, and you use ECS as the orchestration system that ensures that the right configuration of machiens are running your containers.
h

hallowed-king-98702

03/07/2019, 11:11 PM
Shouldn’t the containers be able to access external services (make an HTTP request to google for example)?
l

lemon-spoon-91807

03/07/2019, 11:11 PM
it really depends on how you've configured your network.
so, for example, we have some really helpful APIs to help here in
@pulumi/awsx
(note: the
x
at the end of
awsx
)
There, you can easily make some services, pointing at some containers, using ECS as the orchestrator. ECS then has two main strategies for managing instances you can use. 'Fargate', or 'EC2'
h

hallowed-king-98702

03/07/2019, 11:13 PM
So you’re saying use the ecs api there
l

lemon-spoon-91807

03/07/2019, 11:13 PM
'EC2' gives a lot more control, but is a lot more work on your part. 'Fargate' takes away from control, but makes it a lot easier to just have things work.
h

hallowed-king-98702

03/07/2019, 11:14 PM
What about
@pulumi/cloud-aws
cloud.Service then? My understanding was that it’s an abstraction above the lower level stuff.
l

lemon-spoon-91807

03/07/2019, 11:14 PM
It is. But it will also hide a lot of stuff.
h

hallowed-king-98702

03/07/2019, 11:14 PM
So by default it looks to create a restrictive VPC. Is that correct? And this prevents external access??
l

lemon-spoon-91807

03/07/2019, 11:15 PM
no. by default i beleive it is using your account's default-VPC
unless you've specified otherwise. a nd it will try to use the public subnets of that VPC by defualt if you have any
but i'm not sure what the configuration of your VPC is.
in pulumi/awsx you have a lot more control here. including specifying an entirely independent VPC to run things in.
h

hallowed-king-98702

03/07/2019, 11:16 PM
How do I attach a new VPC to a cloud.Service? Or is that not possible?
l

lemon-spoon-91807

03/07/2019, 11:17 PM
pulumi/cloud was more an attempt to show how you could create cross-cloud attachments.
but when you really need to be able to configure the individual cloud stuff, packages like
@pulumi/awsx
are hte recommended way to go.
How do I attach a new VPC to a cloud.Service? Or is that not possible?
For that, i would recommend using awsx. let me try to find an example
h

hallowed-king-98702

03/07/2019, 11:18 PM
so
@pulumi/cloud-aws
is fairly basic and takes a lot of (secure) defaults I’m assuming (hence no custom VPC’s)?
l

lemon-spoon-91807

03/07/2019, 11:19 PM
Here are some basic examples of making your own VPCs: https://github.com/pulumi/pulumi-awsx/blob/master/nodejs/awsx/examples/vpc/index.ts Here are examples of how you could create an ECS Service sitting on top of Fargate: https://github.com/pulumi/pulumi-awsx/blob/master/nodejs/awsx/examples/fargate/fargate.ts
I would put it more like this: cloud-aws is not an area of high investment 🙂
h

hallowed-king-98702

03/07/2019, 11:19 PM
lol. Gotcha
now what about spinning up a raw EC2 server and uploading a binary and running it? It seems userData isn’t a way to go.
l

lemon-spoon-91807

03/07/2019, 11:21 PM
that isn't something i'm personally familiar with
my question would first start with: how would you normally do this with AWS's apis?
h

hallowed-king-98702

03/07/2019, 11:22 PM
typically spin up a server and then SSH into it 😅
Then git clone, or download a release (curl) or similar
l

lemon-spoon-91807

03/07/2019, 11:22 PM
so... you could do all that if you wanted 🙂
it's more of a really low level way of doing things
i think more modern practices would be recommending you dockerize things
that said, you could cetainly use pulumi to create your EC2 server...
and then still ssh into it
h

hallowed-king-98702

03/07/2019, 11:25 PM
ya. I’ve managed to create the server, just not upload data. I guess then use sftp (separate lib) to upload
l

lemon-spoon-91807

03/07/2019, 11:25 PM
that should still likely work.
h

hallowed-king-98702

03/07/2019, 11:26 PM
But pulumi doesn’t provide any wrappers or conveniences in order to do so?
l

lemon-spoon-91807

03/07/2019, 11:26 PM
for that? no. But if you dockerize things, then definitely yes.
For example, we'll shell out to docker, build everything for you, upload your container to ECR (AWS's container registry), and point your services at those containers
see, for example:
Here, we show an example of creating a Service based off of some container that is built locally (from the
./app
folder)
the Service will sit on fargate to manage instances automatically, and ECS will ensure you always have at least to isntances running.
and the Service will be exposed through an NLB to the internet through port 80.
Does that make sense?
The benefit of the Docker/containerized approach is taht it's very clear what you're running on these instances. as opposed to whatever random changes were made when ssh'ed in. this keeps things highly repeatable and managable
it's also much easier to test as you literally can just use the same containers locally that are in AWS.
h

hallowed-king-98702

03/07/2019, 11:34 PM
ya. I’ve used docker before so I’m familiar with it (just not ECS directly). I’ll try playing around with awsx and avoid
@pulumi/could-aws
I have one other thing to note. I’ve tried using pulumi to build docker images and it works great locally, but fails in github actions (it looks like pulumi uses a blocked docker command, can’t remember off the top of my head). My current solution is to have a separate dockerize action which uploads to ECR and then use that image in pulumi.
l

lemon-spoon-91807

03/07/2019, 11:36 PM
but fails in github actions (it looks like pulumi uses a blocked docker command, can’t remember off the top of my head)
h

hallowed-king-98702

03/07/2019, 11:36 PM
Is there a plan to address this issue in the future?
l

lemon-spoon-91807

03/07/2019, 11:36 PM
hrmm... it works successfully for us in our CI. I can pull in someone else to help you with that
taggign @bitter-oil-46081
Matt, could you possibly help out Andy with https://pulumi-community.slack.com/team/UGR3PG0FR when you have some time?
h

hallowed-king-98702

03/07/2019, 11:38 PM
thanks for the help @lemon-spoon-91807. Appreciate it!
b

bitter-oil-46081

03/07/2019, 11:45 PM
@lemon-spoon-91807 The problem is that the execution environment that github actions run in disallow certain docker commands that
@pulumi/docker
uses. Specifically,
docker image inspect
is blocked in this environment, so https://github.com/pulumi/pulumi-docker/blob/ecddfc468c7baf63e61afbc8dc57ea06247daf13/sdk/nodejs/docker.ts#L360-L363 fails. I think we need to rework
@pulumi/docker
to do something else when this command doesn't work.
h

hallowed-king-98702

03/07/2019, 11:46 PM
ya, that’s the command!
l

lemon-spoon-91807

03/07/2019, 11:46 PM
I apologize. I completely misunderstood. I thought this was about running
pulumi
in a CI scenario, not specifically the 'github actions' workflow.
Matt thinks we have a bug on this. If not, we'll open one.
h

hallowed-king-98702

03/07/2019, 11:47 PM
no worries
b

bitter-oil-46081

03/07/2019, 11:47 PM
I moved a bug we had in cloud (that andy opened, sorry for the delay, andy) to pulumi/docker: https://github.com/pulumi/pulumi-docker/issues/84
l

lemon-spoon-91807

03/07/2019, 11:47 PM
Can't guarantee we'll necessarily have a solution though. I'd have to understand the restrictions of 'github actions' and what (if any) recommendations there are for doing this sort of thing.
h

hallowed-king-98702

03/07/2019, 11:48 PM
Not a problem. I’m sure you guys are busy
Thanks for the help @bitter-oil-46081 and @lemon-spoon-91807. Just want to say that I’m really enjoying pulumi
b

bitter-oil-46081

03/07/2019, 11:53 PM
👍
l

lemon-spoon-91807

03/07/2019, 11:53 PM
That's great to hear!
17 Views