https://pulumi.com logo
Powered by
#google-cloud
Title
# google-cloud
a

able-camera-57198

11/05/2021, 1:22 PM
Copy code
var computeService = new Service(
            "compute",
            options: new() {
                Parent = this,
            },
            args: new ServiceArgs
            {
                ServiceName = "<http://compute.googleapis.com|compute.googleapis.com>",
            }
        );

        var defaultNetwork = Output.Create(GetNetwork.InvokeAsync(new()
        {
            Network = "default",
        }));
Is there any way for me to have the call to 
GetNetwork
depend on 
computeService
? Right now GCP is giving me an error because I'm trying to call 
GetNetwork
prior to the API for it being enabled...
p

prehistoric-activity-61023

11/05/2021, 1:38 PM
Where does this 
GetNetwork
come from?
a

able-camera-57198

11/05/2021, 1:38 PM
p

prehistoric-activity-61023

11/05/2021, 1:39 PM
oh, that’s the GoogleNative plugin
a

able-camera-57198

11/05/2021, 1:39 PM
There's an equivalent in the GCP plugin.
It would fail just the same.
p

prehistoric-activity-61023

11/05/2021, 1:39 PM
in the “old” google plugin, 
get
function has 
opts
where you should be able to set 
dependsOn
a

able-camera-57198

11/05/2021, 1:39 PM
Ahh, well maybe I can switch to that.
p

prehistoric-activity-61023

11/05/2021, 1:40 PM
I’ve never used 
dependsOn
while getting existing resources but I do use that while creating some of them
and this is especially needed when you first create a service in GCP and then create some resources that require it
(so kinda similar to your case)
a

able-camera-57198

11/05/2021, 1:41 PM
Yeah, I have to make sure APIs are enabled before trying to get stuff.
p

prehistoric-activity-61023

11/05/2021, 1:42 PM
If the 
get
behavior is no different than the one implemented while creating new resources that should do the trick. If no, I guess it’d be worth reporting this as a bug.
Not sure if GoogleNative plugin has the same set of features. I haven’t touched it yet myself (and to be honest, I have no motivation as the standard one works just great).
a

able-camera-57198

11/05/2021, 1:42 PM
Looks like 
dependsOn
isn't an option there either.
Just those three.
p

prehistoric-activity-61023

11/05/2021, 1:43 PM
🤔
is that c#?
a

able-camera-57198

11/05/2021, 1:43 PM
yep
And I guess this is what I was hoping would be possible. (part in red)
p

prehistoric-activity-61023

11/05/2021, 1:48 PM
a

able-camera-57198

11/05/2021, 1:48 PM
This is classic.
You can see the namespace there. 
Pulumi.Gcp.Compute
. That's the classic provider.
p

prehistoric-activity-61023

11/05/2021, 1:49 PM
I don’t have access to any C# env right now but from the docs, the signature seems to be different. Something like:
Copy code
public static Network Get(string name, Input<string> id, NetworkState? state, CustomResourceOptions? opts = null)
a

able-camera-57198

11/05/2021, 1:49 PM
p

prehistoric-activity-61023

11/05/2021, 1:49 PM
the last argument, 
CustomResourceOptions
seems to have `DependsOn`: https://www.pulumi.com/docs/reference/pkg/dotnet/Pulumi/Pulumi.CustomResourceOptions.html
a

able-camera-57198

11/05/2021, 1:49 PM
You're thinking about creating a network.
I'm using the 
GetNetwork
function
So the function used to look up an existing network.
p

prehistoric-activity-61023

11/05/2021, 1:50 PM
Hah, I see. It seems kinda… limited and redundant?
a

able-camera-57198

11/05/2021, 1:50 PM
Not at all.
p

prehistoric-activity-61023

11/05/2021, 1:51 PM
Can’t you use the one I referenced above? I’m using this approach from python.
a

able-camera-57198

11/05/2021, 1:51 PM
I don't want to manage this resource. I just want to be able to reference it. But in order to do so, I have to be able to look it up by name.
There's additional non-guessable properties that I need from it.
p

prehistoric-activity-61023

11/05/2021, 1:52 PM
if you read it like that, you won’t manage it
in order to manage it, you’d have to import it to the pulumi state
a

able-camera-57198

11/05/2021, 1:52 PM
Exactly. Which I don't want to do.
I don't want to manage it.
If I use what you're suggesting, I have to add it to state.
p

prehistoric-activity-61023

11/05/2021, 1:52 PM
nope (unless I gave you a wrong link - let me check)
look up != import
it won’t add it to your state, it’s gonna be read every time you run 
pulumi up/preview
and you’re gonna get full-fledged pulumi resource of the given type
a

able-camera-57198

11/05/2021, 1:54 PM
Oh interesting.... This is a link to the resource, but this 
Get
is a method on the type?
p

prehistoric-activity-61023

11/05/2021, 1:54 PM
according to the docs it should be a static function
a

able-camera-57198

11/05/2021, 1:54 PM
Yeah. Hah, amazing. I never even knew this existed, wow.
p

prehistoric-activity-61023

11/05/2021, 1:55 PM
so I guess… try to find type 
Network
within 
Pulumi.Gcp.Compute
and see whether it has a static method called 
Get
a

able-camera-57198

11/05/2021, 1:55 PM
Yup, it's there 🙂
For some reason I thought you were suggesting to use the resource and I couldn't figure out how that would work 😆
p

prehistoric-activity-61023

11/05/2021, 1:57 PM
although, right now I’m not sure it’s gonna work as expected… let me quickly test sth
a

able-camera-57198

11/05/2021, 1:58 PM
Yeah, doesn't appear to work still.
It seems to still be trying to resolve despite having a dependency on networking and compute.
p

prehistoric-activity-61023

11/05/2021, 2:03 PM
is that the classic or native plugin?
a

able-camera-57198

11/05/2021, 2:04 PM
All classic.
Ooh, hang on, I think my IDE grabbed from the native namespace.
Nice catch, gimme a sec.
p

prehistoric-activity-61023

11/05/2021, 2:05 PM
I’ve seen that in one of the screenshots above 😉
a

able-camera-57198

11/05/2021, 2:05 PM
Error 403: Compute Engine API has not been used in project ########### before or it is disabled.
Still, same error in the end. It doesn't appear to be honouring the 
DependsOn
.
This seems like it's happening during the planning phase. So it goes to read the resource anyway, even though the API is really a hard requirement before anything can be read.
At this rate, I feel like I should just set up a different stack that represents just enabling APIs and call it separately. GCP is a bit unique in this.
Although this is probably enough to be bug-worthy?
p

prehistoric-activity-61023

11/05/2021, 2:08 PM
I think so. I’m trying to reproduce something similar on my end but I guess I’m trying to be too quick 😅
so repro steps for this would be: • create a project with default VPC • make sure Compute API is not enabled • in one stack try to: ◦ enable Compute API ◦ retrieve (already existing) default network
Can you (for tests) manually enable the API (or run the code first without the part responsible for network) and make sure that this import actually works then?
that way we’ll be able to narrow down the case to: “`get` methods of resources does not respect 
dependsOn
attribute” or sth like that
a

able-camera-57198

11/05/2021, 2:12 PM
Sure.
Just enabling now and then I'll re-run to see if I get past the initial planning phase.
p

prehistoric-activity-61023

11/05/2021, 2:12 PM
exactly
a

able-camera-57198

11/05/2021, 2:13 PM
Yup, got past it now that the API is enabled.
p

prehistoric-activity-61023

11/05/2021, 2:14 PM
Although I have to admit that Service part of GCP is kinda… demanding. In my projects, I have two separate Pulumi projects called: • gcp-project-bootstrap • gcp-project In the first one I take care about: • creating the project itself (without default VPC) • enabling required services • creating VPC • setting up some top-level IAM things The second one takes care about more specific parts like CloudSQL, GKE etc.
I’m afraid “fixing” 
get
functions might be tricky. They (most probably) are evaluated during plan phase because other changes might depend on it. If you want it to depend on some resource creation, we have chicked-and-egg problem.
a

able-camera-57198

11/05/2021, 2:16 PM
Yup. I think the short term solution is for me to just set up a stack that represents going and flipping al the switches for the services I know that I'll end up using.
Longer term, I guess the GCP (and native?) providers could examine dependencies for instances of API-services and when that happens, defer any attempts to populate data?
I feel like this really is just a thorn with the way google APIs work and how they have to be enabled.
Most cloud providers have all APIs on at all times.
p

prehistoric-activity-61023

11/05/2021, 2:18 PM
that’s true
anyway, create a ticket for that
a

able-camera-57198

11/05/2021, 2:19 PM
Absolutely.
Just about to submit it.
p

prehistoric-activity-61023

11/05/2021, 2:19 PM
I’ll try to look into it once I have some free time.
for the sake of completeness, can you elaborate why you actually needed that?
a

able-camera-57198

11/05/2021, 2:20 PM
I'll add that to the ticket.
p

prehistoric-activity-61023

11/05/2021, 2:21 PM
I personally think it’s good to know the motivation/backstory so the fix can be appropriately implemented or rejected (in case this is something that can be actually resolved in a different way but we didn’t think about it now).
a

able-camera-57198

11/05/2021, 2:21 PM
For sure. Yeah, it's really just about being able to reference that 
"default"
network that GCP creates for all projects once compute is enabled.
It comes with a lot of useful stuff pre-configured and represents a much more turnkey experience.
p

prehistoric-activity-61023

11/05/2021, 2:22 PM
from my experience, it’s better to create a VPC yourself so you have full control over it
the same goes for example with using default svc account for GCE nodes (from my point of view, default permissions are way to high)
a

able-camera-57198

11/05/2021, 2:23 PM
Is there anywhere that defines the differences?
I feel like there were a lot of subnets defined in the default network which I just didn't want to have to go defining myself.
I'm okay with getting onboard with conventional wisdom there for sure. But I'd love to understand the differences at that point 😉
p

prehistoric-activity-61023

11/05/2021, 2:25 PM
I don’t think it will be easy to list all the differences. I just didn’t like the fact that GCP created the VPC all by itself without me knowing anything about it. The other problem (that you actually just described) is getting reference to the default network (that’s not managed by you).
regarding “subnetworks problem”, if you take a look at VPC resource in pulumi, you’ll see an option `auto_create_subnetworks`:
Copy code
def __init__(__self__,
                 resource_name: str,
                 opts: Optional[pulumi.ResourceOptions] = None,
                 auto_create_subnetworks: Optional[pulumi.Input[bool]] = None,
                 delete_default_routes_on_create: Optional[pulumi.Input[bool]] = None,
                 description: Optional[pulumi.Input[str]] = None,
                 mtu: Optional[pulumi.Input[int]] = None,
                 name: Optional[pulumi.Input[str]] = None,
                 project: Optional[pulumi.Input[str]] = None,
                 routing_mode: Optional[pulumi.Input[str]] = None,
                 __props__=None):
so one option to overcome your problems would be to create the project from the scratch using pulumi
• you create a project with 
auto_create_network=false
• you create a vpc with most of the default values (so you don’t need to create subnetwork manually if you don’t want to) • you have a managed vpc network (so no problems with getting it)
I know it’s hell of a difference. I’m just giving you an alternative idea how to tackle it 🙂. If you can recreate the project/network, I’d go that way.
a

able-camera-57198

11/05/2021, 2:29 PM
Oh this is greenfield 🙂
We're moving from Azure to GCP, so I wanna get it right.
p

prehistoric-activity-61023

11/05/2021, 2:31 PM
are you going to use private subnets?
like private GKE cluster (node do not have public IP) and CloudSQL with only private network access?
a

able-camera-57198

11/05/2021, 2:32 PM
Yeah, I have one service in Cloud Run that will need VPC access because it can't do sockets for Cloud SQL 😞
(third party self-host service, looking to kick it out of our architecture in the future...)
I've already prototyped out a terraform configuration that works though, so I make a simple subnet for that.
p

prehistoric-activity-61023

11/05/2021, 2:33 PM
so just in case to save you some time, let me copy’n’paste one part of my component (it’s in python but you should simply translate it to C#):
Copy code
#
        # Configure NAT router
        #
        self.router = gcp.compute.Router(
            name,
            network=self.network.name,
            project=project,
            region=region,
            opts=ResourceOptions(parent=self.network),
        )
        self.nat = gcp.compute.RouterNat(
            name,
            router=self.router.name,
            nat_ip_allocate_option="AUTO_ONLY",
            source_subnetwork_ip_ranges_to_nat="ALL_SUBNETWORKS_ALL_IP_RANGES",
            log_config=gcp.compute.RouterNatLogConfigArgs(
                enable=True, filter="ERRORS_ONLY"
            ),
            project=project,
            region=region,
            opts=ResourceOptions(parent=self.network),
        )

        #
        # Create private VPC peering connection
        #
        # Some Google managed services (such as Cloud SQL) require private services access for each VPC network
        # that is used for private IP connections. This is a one-time procedure but it must be performed before
        # attempting to create dependent services.
        #
        # See: <https://cloud.google.com/sql/docs/mysql/configure-private-services-access>
        #
        ip_address_range = gcp.compute.GlobalAddress(
            self._name,
            purpose="VPC_PEERING",
            address_type="INTERNAL",
            prefix_length=16,
            network=self.network.name,
            project=project,
            opts=ResourceOptions(parent=self.network),
        )
        self.private_connection = gcp.servicenetworking.Connection(
            self._name,
            network=self.network.id,
            service="<http://servicenetworking.googleapis.com|servicenetworking.googleapis.com>",
            reserved_peering_ranges=[ip_address_range.name],
            opts=ResourceOptions(parent=self.network),
        )
a

able-camera-57198

11/05/2021, 2:34 PM
Ah yeah, I have this all already in 
.tf
which I'm going to be transposing over.
I think every GCP deployment template project ends up with this boilerplate 😆
p

prehistoric-activity-61023

11/05/2021, 2:35 PM
can’t post the whole Vpc component resource but just the heads-up: • private GKE cluster => you most probably will need RouterNat • private CloudSQL access => you’re gonna need VPC peering connection (people have problems with that because if you configure that using Console Web UI, most of these things are created under the hood without your knowledge)
a

able-camera-57198

11/05/2021, 2:35 PM
Yeah, I went through the whole song and dance in terraform already. That was a week of digging 😞
p

prehistoric-activity-61023

11/05/2021, 2:36 PM
yeah, that’s why I left an explicit comment in my repo so others can understand why it’s necessary there
a

able-camera-57198

11/05/2021, 2:36 PM
It looks super arbitrary, yeah.
But it's totally not.
I think the one challenge waiting for me is a static outbound IP for a third party we're using. But I'll slog through that once I've got things moved over to Pulumi.
I was due to restructure my 
.tf
templates to be less monolith-y, so moving to Pulumi seemed like a smart move.
p

prehistoric-activity-61023

11/05/2021, 2:37 PM
nowadays it’s not that bad (static outbound IP)
a

able-camera-57198

11/05/2021, 2:37 PM
Oh, nice, well that's reassuring 🙂
p

prehistoric-activity-61023

11/05/2021, 2:37 PM
although it might depends on… what’s gonna use this IP 😄
a

able-camera-57198

11/05/2021, 2:37 PM
Cloud Run 😅
p

prehistoric-activity-61023

11/05/2021, 2:38 PM
that might be… fun 😄
a

able-camera-57198

11/05/2021, 2:38 PM
Even if it is, I actually find using a deployment templating system the best way to wrangle it.
p

prehistoric-activity-61023

11/05/2021, 2:38 PM
anyway, I remember creating a GCE instance with some proxy in order to achieve a static outbound IP
right now, you have access to NAT Gateway as AFAIK this is what you’re looking for
you might be saved 😉
a

able-camera-57198

11/05/2021, 2:40 PM
🙏🏻
p

prehistoric-activity-61023

11/05/2021, 2:40 PM
my suggestion though, check these things first - you might e.g. need to manually create the subnetwork or something
in worst case, some resources in GCP have immutable attributes…
I think you can’t migrate non-private GKE cluster to a private one.
a

able-camera-57198

11/05/2021, 2:42 PM
Oh yeah, I know... There's lots of needles to thread.
There's also a bunch of IAM stuff I have to do for cloud run. Tricky things that the webui totally hides like you mentioned.
And the loadbalancers... Oh man. That was a huge slog too. Very similar, just piecing things together.
p

prehistoric-activity-61023

11/05/2021, 2:43 PM
Anyway, if you have a working TF example, you’re good to go. Most of the TF code can be easily translated to pulumi.
a

able-camera-57198

11/05/2021, 2:43 PM
It totally is! 😄
I'm loving it so much, 'cause I'm basically just sprinkling C# on it all. It's a dream. And so easy to work with now with the strong typing in play.
p

prehistoric-activity-61023

11/05/2021, 2:43 PM
my advice - if you want to do something specific using IaC on GCP, search for terraform recipes and translate them to pulumi 😄
a

able-camera-57198

11/05/2021, 2:44 PM
✔️
p

prehistoric-activity-61023

11/05/2021, 2:44 PM
it’s still quite hard to find some great resources referencing pulumi
(if you want to do more advanced stuff that is)
Anyway, good luck my friend and have a nice day!
a

able-camera-57198

11/05/2021, 2:45 PM
Thank you! You have a great Friday too 🙂
Powered by