Hi guys - I am unable to create a Cosmos DB account lately. The issue seems to be that Pulumi is a bit too aggressive in its polling for resource creation status, or that it should handle HTTP 503 responses more gracefully in the context of such polling, or both. Here’s what I’m consistently getting:

Diagnostics:
  azure:cosmosdb:Account (orgflow-stacks-dev-cdb):
    error: Error creating CosmosDB Account "orgflow-stacks-dev-cdb" (Resource Group "orgflow-stacks-dev-rg"): Error waiting for the CosmosDB Account "orgflow-stacks-dev-cdb" (Resource Group "orgflow-stacks-dev-rg") to finish creating/updating: Code="ServiceUnavailable" Message="Sorry, we are currently experiencing high demand in this region, and cannot fulfill your request at this time. We work continuously to bring more and more capacity online, and encourage you to try again shortly. Please do not hesitate to contact us via Azure support at any time or for any reason using this link <http://aka.ms/azuresupport>.\r\nActivityId: a74b0d98-cc3c-44ed-b5b9-587168e7b8c2"

To be clear: the Cosmos DB account is eventually created on the Azure side, although it doesn’t seem to be complete (I suspect this might be because Pulumi gave up waiting on some part and therefore didn’t proceed to subsequent parts). As a result, we’re left with an incomplete Cosmos DB account that Pulumi doesn’t know about and doesn’t know what to do with in the next attempt - and as we know,

pulumi refresh

does not help in this situation. I suspect that the fix should be: • Handle 503 responses in the resource creation status polling more gracefully and retry • Do it with a back-off strategy to poll less frequently if these responses are coming out • Possibly also start with less frequent polling from the get-go

This surprises me. A lot of people reported getting “we are experiencing high remand” issues, but then their resources would not be created.

@tall-librarian-49374 sorry for the late reply, apparently I had notifications silenced for this workspace for some reason… We are only deploying to a single region. We tried a few different ones, but same result. As to the question whether the region is actually provisioned, I think it is not. The account is created as a resource but a lot of the portal menu options are greyed out, and some parts of the resource blade seem to get errors loading.

Yeah, we see the same errors in our CI now. Looks like Cosmos DB is super restricted now.

Creating the Cosmos DB manually through the portal, however, seems to always work fine.

So you are going through this steps https://www.pulumi.com/blog/adopting-existing-cloud-resources-into-pulumi/ ?

Yes, I went through those steps. @tall-librarian-49374 Pulumi complains during the

import

command, at the points when it starts importing the Cosmos DB account resource. It will stay on “importing…” for a few secs, but soon fail with the error message stating that the properties of the existing resource don’t match the inputs.

Could you open an issue on pulumi-azure describing all of these so that we could track?

Possibly, I’ve done that for some other stuff.

yes, there are many import scenarios that we need to work through. yours sounds good to me