https://pulumi.com logo
Powered by
#getting-started
Title
# getting-started
b

bright-horse-50102

04/22/2022, 3:44 AM
Hi! I have a statefulset:
Copy code
const labels = { appClass: 'app' };

export const appSvc = new k8s.core.v1.Service('app-svc', {
  metadata: {
    labels,
    name: 'app-svc',
  },
  spec: {
    type: 'ClusterIP',
    clusterIP: 'None',
    selector: labels,
  },
}, {
  provider,
});

export const interactionsApp = new k8s.apps.v1.StatefulSet('app', {
  metadata: {
    labels,
  },
  spec: {
    serviceName: 'app-svc',
    replicas: 3,
    selector: {
      matchLabels: labels,
    },
    template: {
      metadata: {
        labels,
      },
      spec: {
        imagePullSecrets: [
          { name: registryCredentials.metadata.name },
        ],
        containers: [
          { /* ... */ },
        ],
      },
    },
  },
}, {
  provider,
  dependsOn: [
    registryCredentials,
    databaseUser,
    redisInstance,
  ],
});
I hardcode the service's 
metadata.name
to 
'app-svc'
and the statefulset's 
spec.serviceName
to 
'app-svc'
as well. While this works, it feels wrong because now the resource name is not managed by pulumi. If I don't specify 
metadata.name
in the service definition, and pass 
appSvc.metadata.name
to the statefulset 
spec.serviceName
, the statefulset is never created.. it waits for the 
appSvc
to become ready, but the 
appSvc
will never become ready because the statefulset hasn't been made yet:
Copy code
kubernetes:core/v1:Service (app-svc):
  error: 2 errors occurred:
      * resource default/app-svc-4kc42v1c was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'app-svc-4kc42v1c' timed out waiting to be Ready
      * Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
how do I solve this issue? Is hardcoding the service name a good idea?
l

little-cartoon-10569

04/22/2022, 3:54 AM
Hardcoding the name is fine, so long as you understand the implications around delete/replace and the lifecycle of resources. I don't understand why interactionsApp isn't created if appSvc.metadata.name isn't set. Is that property automatically set when you don't provide a value? If it is, then as I understand it, Pulumi should be able to figure out dependencies and it should work... maybe the other error message is important, and the Service won't finish creating unless you define a pod?
b

bright-horse-50102

04/22/2022, 4:00 AM
The typings indicate that the property is required
I'm currently testing a more isolated environment, the other error was related to a service depending on the application
Specifically, I'm testing the example on the documentation here https://www.pulumi.com/registry/packages/kubernetes/api-docs/apps/v1/statefulset/#create-a-statefulset-with-auto-naming. I'm waiting for it to time out to be sure, but it looks like the exact same thing is going to happen: the statefulset isn't being created because it's waiting on the service to be ready, but the service isn't ready because it can't find pods matching it's selector, and it can't find pods matching it's selector because the statefulset isn't being created. 🔃
Yeah, it timed out with the example from the documentation
Copy code
Type                           Name                    Status                  Info
     pulumi:pulumi:Stack            _redacted_              **failed**              1 error
 +   └─ kubernetes:core/v1:Service  nginx-service           **creating failed**     1 error

Diagnostics:
  kubernetes:core/v1:Service (nginx-service):
    error: 2 errors occurred:
        * resource default/nginx-service-nm3byu6a was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'nginx-service-nm3byu6a' timed out waiting to be Ready
        * Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
 
  pulumi:pulumi:Stack (_redacted_):
    error: update failed
Is this the documentation being wrong or Pulumi being wrong?
l

little-cartoon-10569

04/22/2022, 4:18 AM
I'm afraid I don't know. Could it have to do with 
labels
? You're specifying it as an object, but the examples all have it as an array of strings....
b

bright-horse-50102

04/22/2022, 4:23 AM
I destroyed my stack and ran the example code from the documentation above, so my 
labels
isn't a thing anymore
Considering it's just example code I'm pretty certain either the documentation example is wrong, or something regressed in Pulumi (I assume the example worked at some point)
l

little-cartoon-10569

04/22/2022, 4:24 AM
Do any pods get created after the 10 minute timeout?
b

bright-horse-50102

04/22/2022, 4:25 AM
Nope. The statefulset itself just doesn't get created (
kubectl get statefulset
returns an empty list). I also don't see a 
creating...
in the pulumi cli output for the statefulset; it starts by trying to create the service first, it'll say the service is waiting for 
n
pods matching the selector, and this times out because those pods never get spawned (because the statefulset isn't being made)
If I hardcode the service name, everything works perfectly
l

little-cartoon-10569

04/22/2022, 4:27 AM
Then I'm at a loss. Maybe cross-post to #kubernetes?
b

bright-horse-50102

04/22/2022, 4:27 AM
Will do, thanks
👍 1
b

billowy-army-68599

04/22/2022, 8:11 AM
@bright-horse-50102 it's quite rare to use the servicename like this, is there a reason you're doing it?
b

bright-horse-50102

04/22/2022, 8:13 AM
Use it like what? Hardcoding it? My end goal here is to simply make a StatefulSet following best practices, I have no other specific requirements
b

billowy-army-68599

04/22/2022, 8:17 AM
I would use the example here: https://www.pulumi.com/registry/packages/kubernetes/api-docs/apps/v1/statefulset/ note this section:
Copy code
spec: {
        selector: {
            matchLabels: {
                app: "nginx",
            },
        },
The reason you're having a hard time using 
serviceName: 'app-svc',
is because pulumi has logic in it to ensure the service has healthy pods behind it. by setting the name this way, you're creating a circular dependency. remove the 
serviceName
property and use the matchlabels property instead
b

bright-horse-50102

04/22/2022, 8:20 AM
I've used that exact example, message with details here: https://pulumi-community.slack.com/archives/C01PF3E1B8V/p1650600176789779?thread_ts=1650599095.566589&cid=C01PF3E1B8V The line below the snippet you've sent includes 
serviceName
anyway (right below the selector). It's included in the example. Am I missing something? 😅
b

billowy-army-68599

04/22/2022, 8:21 AM
🤦‍♂️ forgive me, you're right
what image are you running, I'll take a look in a minute
b

bright-horse-50102

04/22/2022, 8:22 AM
I'm running that exact example, so the 
<http://k8s.gcr.io/nginx-slim:0.8|k8s.gcr.io/nginx-slim:0.8>
image. Copied the entire codeblock and didn't change anything else
b

billowy-army-68599

04/22/2022, 8:23 AM
okay, give me 10 mins
👍 1
b

bright-horse-50102

04/22/2022, 8:24 AM
I did change one thing actually, I renamed the service from 
nginxService
to 
nginx-service
on line 4 because I was getting an error about service names having to be valid DNS names (i.e. lowercase). No changes other than that
here's what I ran specifically
Copy code
import * as kubernetes from '@pulumi/kubernetes';

const nginxService = new kubernetes.core.v1.Service('nginx-service', {
  metadata: {
    labels: {
      app: 'nginx',
    },
  },
  spec: {
    ports: [ {
      port: 80,
      name: 'web',
    } ],
    clusterIP: 'None',
    selector: {
      app: 'nginx',
    },
  },
});

const wwwStatefulSet = new kubernetes.apps.v1.StatefulSet('wwwStatefulSet', {
  spec: {
    selector: {
      matchLabels: {
        app: 'nginx',
      },
    },
    serviceName: nginxService.metadata.name,
    replicas: 3,
    template: {
      metadata: {
        labels: {
          app: 'nginx',
        },
      },
      spec: {
        terminationGracePeriodSeconds: 10,
        containers: [ {
          name: 'nginx',
          image: '<http://k8s.gcr.io/nginx-slim:0.8|k8s.gcr.io/nginx-slim:0.8>',
          ports: [ {
            containerPort: 80,
            name: 'web',
          } ],
          volumeMounts: [ {
            name: 'www',
            mountPath: '/usr/share/nginx/html',
          } ],
        } ],
      },
    },
    volumeClaimTemplates: [ {
      metadata: {
        name: 'www',
      },
      spec: {
        accessModes: [ 'ReadWriteOnce' ],
        storageClassName: 'my-storage-class',
        resources: {
          requests: {
            storage: '1Gi',
          },
        },
      },
    } ],
  },
});
b

billowy-army-68599

04/22/2022, 8:43 AM
@bright-horse-50102 I am shocked this has never come up before, but it's actually a bug: https://github.com/pulumi/pulumi-kubernetes/issues/1974 I added a workaround at the bottom, you need to add an annotation to skip the await logic
b

bright-horse-50102

04/22/2022, 8:44 AM
I see 😅 Glad to have helped then! The workaround makes sense, thanks.
b

billowy-army-68599

04/22/2022, 8:45 AM
fyi: if you don't have a volume controller that'll work for the storageclass, it'll also faul
b

bright-horse-50102

04/22/2022, 8:46 AM
Gotcha, luckily I'm not dealing with volumes for my scenario
b

billowy-army-68599

04/22/2022, 8:46 AM
? why use a statefulset then, out of interest?
b

bright-horse-50102

04/22/2022, 8:49 AM
Our application is a service that consumes data from a third party platform (Discord - a chat application comparable to Slack). Discord applications have the concept of "shards", these shards are assigned a static range of Discord groups that they receive events for. Each of these shards have to identify with Discord's websocket with a numeric "shard ID", which the stable unique ordinal of a statefulset pod is perfect for
b

billowy-army-68599

04/22/2022, 8:50 AM
oh sweet, makes sense!
😄 1
b

bright-horse-50102

04/22/2022, 8:51 AM
Also, I suppose the name of the service being camelCase is a separate issue?
b

billowy-army-68599

04/22/2022, 8:52 AM
yep I'm fixing that now
👍 1
4 Views
Powered by