https://pulumi.com logo
#general
Title
# general
m

millions-furniture-75402

03/25/2024, 1:00 PM
Has anyone here successfully declared a Pinecone Index in gcp-starter pod using Pulumi? I'm able to do it with the pinecone client, and through the GUI no problem, but the provider has been an uphill battle.
e.g. this works:
Copy code
import os

from dotenv import load_dotenv
from pinecone import Pinecone, PodSpec  # pylint: disable=import-error

load_dotenv()

pc = Pinecone(api_key=os.environ.get("PINECONE_API_KEY"))

pc.create_index(
    name="my-index", dimension=1536, metric="cosine", spec=PodSpec(environment="us-west-1-gcp", pod_type="p1.x1", pods=1)
)
But many iterations of this, an I'm getting 404s:
Copy code
import pinecone_pulumi as pinecone

# Create a Pinecone Index with a "starter" pod configuration
pinecone_index = pinecone.PineconeIndex("my-index",
    name="my-index",
    spec=pinecone.PineconeSpecArgs(
        pod=pinecone.PineconePodSpecArgs(
            pods=1,
            shards=1,
            pod_type="p1.x1",
            replicas=1,
            environment=pinecone.ServerlessSpecCloud.GCP,
        )
    ),
    metric="cosine",
    dimension=1536,
)
m

many-telephone-49025

03/25/2024, 1:54 PM
@millions-furniture-75402 can you please run
pulumi up -d
I need to work on “detailed” message output 😅
m

millions-furniture-75402

03/25/2024, 2:06 PM
debug: Pinecone index creation response: {"error": {"code":"NOT_FOUND","message":"Resource gcp not found"},"status":404}
m

many-telephone-49025

03/25/2024, 2:08 PM
You are mixing
environment=pinecone.ServerlessSpecCloud.GCP,
with a pod based index
m

millions-furniture-75402

03/25/2024, 2:17 PM
Ahh, good looking out. I changed that to "gcp-starter" and "us-west-1-gcp" and get https://pastebin.com/0LNLLvTe
m

many-telephone-49025

03/25/2024, 2:36 PM
also run with -d for some more infos
Especially the
Pinecone index creation response: …
is important for me
m

millions-furniture-75402

03/25/2024, 3:45 PM
I've been experimenting with the
pinecone-client
directly The region should be
us-central1-gcp
but I can't overcome this issue: "Increase you quota or upgrade to create more indexes." If I list indexes (via library or web application), I don't see any. I've tried deleting and recreating the project.
m

many-telephone-49025

03/25/2024, 3:46 PM
do you have a index?
m

millions-furniture-75402

03/25/2024, 3:46 PM
no
m

many-telephone-49025

03/25/2024, 3:46 PM
in a different project?
have a look in the UI there is a dropdown
m

millions-furniture-75402

03/25/2024, 3:46 PM
Not that I can see. Maybe a bug in their service.
m

many-telephone-49025

03/25/2024, 3:47 PM
otherwise have a look to increase quota
m

millions-furniture-75402

03/25/2024, 3:47 PM
I was hoping to do with with the free plan for demonstration purposes.
m

many-telephone-49025

03/25/2024, 3:47 PM
maybe create a new account
m

millions-furniture-75402

03/25/2024, 3:47 PM
At this point, it might be easier to host my own vector db 😅
m

many-telephone-49025

03/25/2024, 3:47 PM
temporary
until it’s sorted out
m

millions-furniture-75402

03/25/2024, 3:48 PM
Will experiment more later. I can get by without this declaration for the moment.
Oh... interesting... deleting the default project might have been the bug, I cannot set my pod limit above 0 now.
m

many-telephone-49025

03/25/2024, 3:49 PM
interesting
m

millions-furniture-75402

03/25/2024, 3:50 PM
Will reach out to their support if I find the bug is on their end.
appreciate your help
m

many-telephone-49025

03/25/2024, 3:50 PM
np, as pinecone is not us i am only here to rubberduck debug with you
m

millions-furniture-75402

03/25/2024, 3:56 PM
ah ha, so that wasn't quite the error, but it did give me the hint.
pod_type
needs to be
starter
and
environment
needs to
gcp-starter
This isn't even documented in Pinecone docs, let alone the Pulumi provider. I was able to infer this by manually creating one in the console. A real missed opportunity for Pinecone.
Further bad news, I can create successfully with the script using the client directly, but those values don't work with the Pulumi declaration.
m

many-telephone-49025

03/25/2024, 3:57 PM
what is the error with -d ?
m

millions-furniture-75402

03/25/2024, 3:58 PM
ValueError: '' is not a valid ServerlessSpecCloud
seems to be the issue I can put together an issue for their GitHub repository later this week.
m

many-telephone-49025

03/25/2024, 3:58 PM
but starter is not a serverless
hmm strange
m

millions-furniture-75402

03/25/2024, 3:59 PM
Right
m

many-telephone-49025

03/25/2024, 3:59 PM
can you c&p the full code here?
i have a look and play around with, could be an easy fix in the provider!
m

millions-furniture-75402

03/25/2024, 4:04 PM
In trying to provide you with a minimum reproducible example, I found the issue, back to "pods". I had
pod=1
, which ultimately led to the issue. Here's the final "this works with the free tier" example.
Copy code
pinecone_index = pinecone.PineconeIndex(
    "my-index",
    name="my-index",
    spec=pinecone.PineconeSpecArgs(
        pod=pinecone.PineconePodSpecArgs(
            environment="gcp-starter",
            pod_type="starter",
            replicas=1,
        )
    ),
    metric="cosine",
    dimension=1536,
)
Again... I spoke to soon, refresh fails with that. Still some bug.
m

many-telephone-49025

03/25/2024, 4:06 PM
and -d
try to run with -d always
m

millions-furniture-75402

03/25/2024, 4:07 PM
It was the ServerlessSpec issue again, sorry currently working on a different computer than I'm talking to you on.
m

many-telephone-49025

03/25/2024, 4:08 PM
kk, let me check! I think that and the snippet should be fine to try
could be a change in the API which is not reflected in the provider
@millions-furniture-75402 for me it runs perfect through. But I used TS, if its matters
Copy code
const myPineconeIndex = new pinecone.PineconeIndex("myPineconeIndex", {
    name: "example-index",
    metric: pinecone.IndexMetric.Cosine,
    spec: {
        pod: {
            podType: "starter",
            replicas: 1,
            pods: 1,
            environment: "gcp-starter",
        }
    },
});
export const output = {
    value: myPineconeIndex.host,
};
m

millions-furniture-75402

03/25/2024, 5:52 PM
I've been using Python. I have not tried it with typescript.
m

many-telephone-49025

03/25/2024, 5:52 PM
let me check! one moment
i think I have it
i will release a new version with better error logging
You should now see an message like this:
error: failed to create Pinecone index: example-index3 with error: Projects only support a single Starter index.
This happens, when you edit an existing resource which has to be replaced to make the change (like remove pod property) You can use https://www.pulumi.com/docs/concepts/options/deletebeforereplace/ to have the index deleted before recreation! new version is
0.4.1
m

millions-furniture-75402

03/25/2024, 7:33 PM
I'll update you when I confirm.
m

many-telephone-49025

03/25/2024, 7:37 PM
Yep, keep an eye on the message now, in case you alter an existing index. I tried to create and delete a classsic, serverless and starter index and all worked smooth. Edit not.
m

millions-furniture-75402

03/25/2024, 7:52 PM
It "works" but has a failure the,
ValueError '' is not a valid ServerlessSpecCloud
. It might have to do with the remarshaling of outputs. I tried recreating my venv, and have the same failure. Looking at the enums in the library now.
m

many-telephone-49025

03/25/2024, 7:53 PM
i need the full stack
plus -d
i am out to grab food and try in pyton
cant get my head around what the issue is
m

millions-furniture-75402

03/25/2024, 7:54 PM
give me a moment
sent
m

many-telephone-49025

03/25/2024, 9:25 PM
ok i have it in python too the error!
nice find!
Congratulation @millions-furniture-75402, you found a potential bug in the provider! https://github.com/pulumi/pulumi-go-provider/issues/204 Where can I send you swag? 😄 I should definitely use more python! 😄
new version
v0.4.2