https://pulumi.com logo
Title
p

powerful-football-81694

04/11/2020, 10:46 AM
Hi guys - I am unable to create a Cosmos DB account lately. The issue seems to be that Pulumi is a bit too aggressive in its polling for resource creation status, or that it should handle HTTP 503 responses more gracefully in the context of such polling, or both. Here’s what I’m consistently getting:
Diagnostics:
  azure:cosmosdb:Account (orgflow-stacks-dev-cdb):
    error: Error creating CosmosDB Account "orgflow-stacks-dev-cdb" (Resource Group "orgflow-stacks-dev-rg"): Error waiting for the CosmosDB Account "orgflow-stacks-dev-cdb" (Resource Group "orgflow-stacks-dev-rg") to finish creating/updating: Code="ServiceUnavailable" Message="Sorry, we are currently experiencing high demand in this region, and cannot fulfill your request at this time. We work continuously to bring more and more capacity online, and encourage you to try again shortly. Please do not hesitate to contact us via Azure support at any time or for any reason using this link <http://aka.ms/azuresupport>.\r\nActivityId: a74b0d98-cc3c-44ed-b5b9-587168e7b8c2"
To be clear: the Cosmos DB account is eventually created on the Azure side, although it doesn’t seem to be complete (I suspect this might be because Pulumi gave up waiting on some part and therefore didn’t proceed to subsequent parts). As a result, we’re left with an incomplete Cosmos DB account that Pulumi doesn’t know about and doesn’t know what to do with in the next attempt - and as we know,
pulumi refresh
does not help in this situation. I suspect that the fix should be: • Handle 503 responses in the resource creation status polling more gracefully and retry • Do it with a back-off strategy to poll less frequently if these responses are coming out • Possibly also start with less frequent polling from the get-go
t

tall-librarian-49374

04/11/2020, 2:28 PM
This surprises me. A lot of people reported getting “we are experiencing high remand” issues, but then their resources would not be created.
Are you creating the account in multiple regions? If so, are all of these regions actually provisioned, or just a subset of them?
p

powerful-football-81694

04/14/2020, 11:19 AM
@tall-librarian-49374 sorry for the late reply, apparently I had notifications silenced for this workspace for some reason… We are only deploying to a single region. We tried a few different ones, but same result. As to the question whether the region is actually provisioned, I think it is not. The account is created as a resource but a lot of the portal menu options are greyed out, and some parts of the resource blade seem to get errors loading.
t

tall-librarian-49374

04/14/2020, 11:20 AM
Yeah, we see the same errors in our CI now. Looks like Cosmos DB is super restricted now.
p

powerful-football-81694

04/14/2020, 12:55 PM
Creating the Cosmos DB manually through the portal, however, seems to always work fine.
@tall-librarian-49374 since creating through the portal (and also by ARM deployment) work fine consistently, I can only conclude something needs to change on the Pulumi side. We are currently blocked by this, and we cannot use the workaround to import the resource into Pulumi after creating it manually; no matter how hard we try to match the inputs exactly to what the existing resource has, Pulumi complains that inputs don’t match existing resource properties (but does not state what differences it detects).
t

tall-librarian-49374

04/15/2020, 6:30 AM
At which point does Pulumi complain?
p

powerful-football-81694

04/15/2020, 11:20 AM
Yes, I went through those steps. @tall-librarian-49374 Pulumi complains during the
import
command, at the points when it starts importing the Cosmos DB account resource. It will stay on “importing…” for a few secs, but soon fail with the error message stating that the properties of the existing resource don’t match the inputs.
t

tall-librarian-49374

04/15/2020, 11:22 AM
Could you open an issue on pulumi-azure describing all of these so that we could track?
FWIW, you could work around this by embedding an ARM template into your Pulumi program for now, if that works
p

powerful-football-81694

04/15/2020, 12:45 PM
Possibly, I’ve done that for some other stuff.
You know what would be super useful @tall-librarian-49374? If there was a
pulumi up --import
argument that, given the existence of a resource with the same name (at the cloud provider level), imports it into the state unconditionally, with whatever properties it has, before proceeding to do an
up
as usual.
That would work so beautifully in our situation where we always specify explicit resource names with the stack name embedded in them.
(Yes, I’ll open two issues, one for the Cosmos 503 issue, and one for the import issue)
t

tall-librarian-49374

04/15/2020, 1:59 PM
yes, there are many import scenarios that we need to work through. yours sounds good to me
p

powerful-football-81694

04/16/2020, 8:14 PM
@brief-machine-10802