Hi all, does anyone have any tips for connecting K...
# google-cloud
Hi all, does anyone have any tips for connecting K8s -> CloudSQL? I've got a job running a simple ping golang app using the in-process CloudSQLProxy connector (https://github.com/GoogleCloudPlatform/cloud-sql-go-connector), but keep getting a`failed to connect to `host=localhost user=my-service-account@my-domain.iam database=my_db_name`: dial error (timeout: context deadline exceeded)` error. Note: I'm not sure why the log message has
as "localhost", as the host given in the connection string is `projectregiondb_instance_name`_._ I can see in the following in the DB logs when it attempts to connect:
Copy code
authorizationInfo: [
  0: {
    granted: true
    permission: "cloudsql.instances.connect"
    resource: "projects/my-project/instances/my-db-instance-8898595"
    resourceAttributes: {
    name: "projects/my-project/instances/my-db-instance-8898595"
    service: "<http://sqladmin.googleapis.com|sqladmin.googleapis.com>"
    type: "<http://sqladmin.googleapis.com/Instance|sqladmin.googleapis.com/Instance>"
methodName: "cloudsql.instances.connect"
So can see it's correctly reaching out for the right db etc, but that's the only message in the logs. The ping code is basically doing:
Copy code
driverName := "cloudsql-postgres"
cleanup, err := pgxv4.RegisterDriver(driverName, cloudsqlconn.WithIAMAuthN())

connStr := fmt.Sprintf("host=%v user=%v dbname=%v sslmode=disable", host, user, name)
db, err := sql.Open(driverName, connStr)

ctx, cancelfunc := context.WithTimeout(context.Background(), 30*time.Second)
pingErr := db.PingContext(ctx) // times out here

if pingErr != nil {
  log.Fatalf("server: Got error: %v\n", pingErr) // logs error here due to timeout
and my k8s IAM setup is basically:
Copy code
const saJob = new gcp.serviceaccount.Account("sa-job", {
    accountId: "sa-for-job",
    displayName: "My Service Account for the ping Job",

  const iamDBEditor = new gcp.projects.IAMBinding("iam-db-editor", {
    project: gcpProject,
    role: "roles/cloudsql.editor",
    members: [pulumi.interpolate`serviceAccount:${saJob.email}`],
  }, {
    parent: saJob

  const iamDBUser = new gcp.projects.IAMBinding("iam-db-user", {
    project: gcpProject,
    role: "roles/cloudsql.instanceUser",
    members: [pulumi.interpolate`serviceAccount:${saJob.email}`],
  }, {
    parent: saJob

  const k8sSAJob = new k8s.core.v1.ServiceAccount("k8s-sa-job", {
      metadata: {
        namespace: namespaceName,
        annotations: {
          "<http://iam.gke.io/gcp-service-account|iam.gke.io/gcp-service-account>": saJob.email,
      provider: clusterProvider,

  const iamK8sBinding = new gcp.serviceaccount.IAMBinding("iam-k8s-db-editor", {
    serviceAccountId: saJob.name,
    role: "roles/iam.workloadIdentityUser",
    members: [pulumi.interpolate`serviceAccount:${gcpProject}.svc.id.goog[${namespaceName}/${k8sSAJob.metadata.name}]`],

  /* Create a user with the configured IAM credentials (linked to the service account).
     For more info:
       * <https://cloud.google.com/sql/docs/postgres/iam-logins>
       * <https://cloud.google.com/sql/docs/postgres/add-manage-iam-users#creating-a-database-user>
  const dbUserJobAPIServerMigration = new gcp.sql.User("api_server_migrator", {
    instance: db.instance.name,
    name: saJob.email.apply((v) =>
      v.replace(".<http://gserviceaccount.com|gserviceaccount.com>", "")
    project: gcpProject,
I've also tested locally connecting to CloudSQL with pretty much the same golang code except using a
file instead of `cloudsqlconn.WithIAMAuthN()`along with a different db user and the ping code works as expected. So, I'm thinking the problem might be something related to perhaps kubernetes and IAM auth, but I'm honestly just a bit stuck and hence the reach out. Apologies for the long post, I'm hoping someone's been through this pain before and can offer a tip or two on troubleshooting or what might be going on. Thanks for your help!
It's worth testing with the credentials file in k8s, so you know your networking is working. I think the 'connect' log you're seeing is actually the api for fetching a certificate, not an actual db connection. https://cloud.google.com/sql/docs/mysql/admin-api/rest#rest-resource:-v1.connect
Good news is, that means your workload identity is working :)
Thanks Anthony, I'll give that a go :)
Update: I updated the Job code, hardcoding JSON to verify that what works locally isn't working in GKE (which it didn't). I then stripped back the cluster configuration to the bare essentials and removed the cluster network / subnet setup. Once I did this, the job connected to CloudSQL 🙂 So, I guess I'll have to do some reading up on networks / cluster configs. For now I'm happy that I can talk to the DB!
Glad you managed to narrow it down. I've got to do all this for my clusters soon. Absolutely dreading it, all the network configs are currently commented out 🤣 Hoping this will help: https://cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview