https://pulumi.com logo
Join Slack
Powered by
Is it intended behavior that reading the metadata ...
# kubernetes
c
Is it intended behavior that reading the metadata of a kubernetes resource attempts to immediately perform a read on the resource, even if the resource doesn't exist yet? concrete example:
Copy code
var myObj = new SomeKubernetesObject( "my-k8s-object", snip );
var myOtherObject = new SomeOtherResource( "my-other-object", new SomeOtherResourceArgs {
  ObjectName = myObj.Metadata.Apply( m => m.Name ),
} );
This results in an error along the lines of:
Copy code
Preview failed: resource 'my-k8s-object-00af0ebd' does not exist
if "my-k8s-object" doesn't exist yet at the time of 
pulumi up
pulumi knows that the object doesn't exist yet, so why is it trying to perform a Read when Metadata is read instead of blocking on the create?
In case it's relevant, the k8s object I'm creating is a CRD, using a library generated by crd2pulumi, with C# as the language
if I "taint" the output by doing a union with 
myObj.Id
, then the error doesn't occur
so pulumi knows what it should be doing in general, just not when reading the Metadata field
I don't know if this happens with official k8s objects, I haven't created a minimal repro yet
I wanted to see if this was 'known' behavior first
this issue should happen way more often if it's all k8s objects, so it must be something going wrong with the CRD resource
b
It shouldn't do that until it's been reported that (in your example) 
myObj
has been created. it might be because you're using something generated by crd2pulumi but it's still a resource... I'm just checking that there's a more general bug in the kubernetes provider, but do you have a simple CRD that you can target with crd2pulumi so we can reproduce this in an issue?
I've tried to reproduce this (code here) but I'm not able to. Are you able to share a reproduction that I work with?
c
yeah no problem, I will put together a minimal repro using the exact crd that caused the problem
@brave-planet-10645 here's a repro: https://github.com/Madgvox/crd-bug-repro So, an important bit I inadvertently left out of my explanation: this is occurring because I'm trying to do an immediate read of the resource elsewhere.
The problem is that Metadata is returning immediately with the name and namespace filled out. I guess since they're known values at up time. I don't know what Pulumi would do if a controller modified the labels or annotations upon create. Does it drop them? Or is the unknown-ness of Metadata field-specific?
But the effect is that anything that tries to read it will fail because the object doesn't exist yet
This happens with non-crd objects too
There seems to be some tension here in that I understand why one would want to resolve fields that are known immediately because otherwise the preview would have a bunch of 'unknowns' that should intuitively be known But that becomes an issue when you need to act on those resources during preview/up time.
I'm guessing the dependency model covers this for resource creation, but not for Reads for some reason?
maybe that's the real bug... the Resource.Get function takes an Input as its id, but doesn't seem to create a dependency on the upstream resources properly?
I'm at the edges of my understanding of the pulumi dependency model so I could be way off base here
is it the expectation that Metadata and Spec represent the state of the fields as passed into the resource or the state of the fields after resource creation?
s
1. Tested reading kubernetes resource metadata from another kubernetes resource. It looks fine with or without CRD library. Ref: Test Read Kubernetes metadata resource repo 2. Test your repo, Issue is with creating service where spec.ports are mandatory parameters.
Copy code
Mandatory: Only spec.ports[].port

Highly Recommended:

spec.selector - To route traffic to specific pods
spec.ports[].targetPort - To specify the pod's port
spec.type - To control how the service is exposed
Please find the screenshot for error detail.
c
I don't understand what you're trying to say here. I'm not getting an error related to ports, I'm getting an error trying to read a resource that doesn't exist. I'm also not sure what linking to a repository where the error doesn't happen accomplishes when I have a sample repository where the error does happen. Your sample repo is in typescript, I'm using C#. I'm also performing a SecretStore.Get call, and your test repo is not.
s
Let's go step by step. 1. Please help me with screenshot of the error. Can you please mention the steps in your repo to replicate? Ran 
pulumi up
command for now after replicating the repo. 2. Service Creation missing port as mandatory param compared with exemplary service creation
Copy code
# Create service with mandatory + recommended parameters
service = kubernetes.core.v1.Service(
    "nginx-service",
    spec=kubernetes.core.v1.ServiceSpecArgs(
        selector=app_labels,           # Match the deployment labels
        ports=[kubernetes.core.v1.ServicePortArgs(
            port=80,                   # MANDATORY: External port
            target_port=80,            # Pod's port
            protocol="TCP",
        )],
        type="LoadBalancer",          # ClusterIP, NodePort, or LoadBalancer
    ),
3. C# example is already posted above. 4. Sure, will address/test with SecretStore.Get call after we fix the service creation call.
c
You need a kubernetes cluster to target. Running 
pulumi preview
results in the error. I didn't build the repro with applying in mind because the error is happening during preview. So I'm not surprised the service is invalid, but that's not relevant because I'm not actually creating it. You don't need to apply the stack to see the error. The fact you're getting to the apply stage means you're not getting the error which is weird. I'm not able to apply the stack at all.
s
1. To me 
pulumi preview
looks good. 2. Using minikube to start the local cluster. 3. Can see crd-test ns running in cluster after pulumi up command but failed while creating service.
Copy code
crd-bug-repro % kubectl get po -A
NAMESPACE     NAME                                                READY   STATUS    RESTARTS        AGE
crd-test      external-secrets-568b5747b6-9txsk                   1/1     Running   0               36s
crd-test      external-secrets-cert-controller-79cbcf455f-8q85d   1/1     Running   0               36s
crd-test      external-secrets-webhook-7d79c46458-w8jbd           1/1     Running   0               36s
kube-system   coredns-668d6bf9bc-72n5k                            1/1     Running   2 (15h ago)     47d
kube-system   etcd-minikube                                       1/1     Running   2 (15h ago)     47d
kube-system   kube-apiserver-minikube                             1/1     Running   2 (15h ago)     47d
kube-system   kube-controller-manager-minikube                    1/1     Running   2 (15h ago)     47d
kube-system   kube-proxy-trctx                                    1/1     Running   2 (15h ago)     47d
kube-system   kube-scheduler-minikube                             1/1     Running   2 (15h ago)     47d
kube-system   storage-provisioner                                 1/1     Running   7 (6m43s ago)   47d
c
did you modify the source? the broken bit is commented out in the git repo
Copy code
// uncomment the below lines to get error: Preview failed: resource 'crd-test/secret-store-10e7357d' does not exist
Copy code
var brokenRead = secretStore.Metadata.Apply( s => 
        SecretStore.Get( "secret-store-read", $"{s.Namespace}/{s.Name}", new CustomResourceOptions { Provider = k8sProvider } )
            .Spec
            .Apply( s => s.Provider.Fake.Data )
    );
this is the broken read
cmd output
the broken bit is commented out in the git repo
forgot I did this, mb
s
Copy code
crd-bug-repro % git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
c
No I mean, the broken code is in the repo, it's just commented out. If you uncomment the lines I showed it should show the broken behavior
s
Got it. Understand it better now. Please let me get back to you.
👍 1
Able to recreate the error scenario
Key Points to Fix the Error 1. The service must exist before you can read it. 2. Use the correct resource ID format- It should be namespace/serviceName 3. Don't read resources you are creating in the same program - Try to reference them directly 4. If you must read, ensure Dependencies are set - Use DependsOn or reference outputs. Example of DependsOn is already in the thread.
b
@chilly-sunset-85353 So the 
metadata.name
and the 
metadata.namespace
values are generated by the Pulumi engine. That is why these are known straight away. They don't come from the cloud service. The way to do what you're trying to do is to wrap the 
namespace/name
ID value in 
Output.Format
which will wrap an 
Output<T>
around these values and it should all work properly. So something like this:
Copy code
var brokenRead = SecretStore.Get("secret-store-get", Output.Format($"{secretStore.Metadata.Apply(x => {
        return string.Format($"{x.Namespace}/{x.Name}");
    } )}"));
will work. Don't forget the 
return
as otherwise you won't get anything Having said that, when I run that and add it to the return statement to see what the values are in the CLI:
Copy code
var brokenRead = SecretStore.Get("secret-store-get", Output.Format($"{secretStore.Metadata.Apply(x => {
        return string.Format($"{x.Namespace}/{x.Name}");
    } )}"));

    return new Dictionary<string, object?>
    {
        ["br"] = brokenRead.Spec.Apply(x => x.Provider.Fake.Data)
    };
I'm not getting much useful information back, so not sure what information you're trying to access, but this should get you over the hump. Ok, I've adjusted the Dictionary above and can see the data now. So whichever way you choose you can get the data out
Ah I see now... Actually there's a much simpler option that you can use to ensure that the resource exists and that is to use the 
DependsOn
resource option:
Copy code
var brokenRead = secretStore.Metadata.Apply( s => 
        SecretStore.Get( "secret-store-read", $"{s.Namespace}/{s.Name}", new CustomResourceOptions { Provider = k8sProvider, DependsOn = secretStore } )
            .Spec
            .Apply( s => s.Provider.Fake.Data )
    );
That is identical to the code you sent over, except I've added 
DependsOn = secretStore
And now I can see the data that you've added to the 
secretStore
resource as well
c
Yeah, I was also able to create a dependency by touching secretStore.Id. It's six of one, half a dozen of another for creating the dependency. I don't see how Output.Format alone would prevent the error, though. I tried it both ways and still got the error. It seems functionally identical. But that aside, isn't there a logistical problem with having Metadata being assumed during plan time? As I mentioned previously, there are mutation hooks that can modify metadata for created resources. I don't think it's safe to assume that the metadata you pass into an object is the same metadata you get out of it. In c# at least, it doesn't seem like there's a way to express having only part of the metadata known at plan time.
9 Views
Open in Slack
PreviousNext
Powered by