Any folks having a similar problem with the Fastly...
# general
m
Any folks having a similar problem with the Fastly provider? https://github.com/pulumi/pulumi-fastly/issues/119
@broad-dog-22463 Figured i’d ping you on this since you are one of the committers on this repo and hopefully would be familiar with the issue
a
@billowy-army-68599 or @white-balloon-205 any input on ^? It seems that the Fastly provider is currently unusable due to this bug. We're happy to provide some contribution to get it resolved if you can point us in the right direction.
w
I believe @stocky-restaurant-98004 is looking into this. He had some questions about getting a repro that captures the current state of the issue here.
a
We've got an issue in our repository that details our experience of the bug. https://github.com/mitodl/ol-infrastructure/issues/652
s
I'm gonna need a minimal example to try to reproduce. Can one of you fine folks comment on the issue with a minimal example? FWIW, we also released a couple of upstream bug fixes that may fix the issue, but that's far from definite.
m
Hey @stocky-restaurant-98004! Sorry for the late response but i was out last week. I added a minimal example to the issue and tested the new version of the provider which appears to still throw the same issue - https://github.com/mitodl/ol-infrastructure/issues/652
Morning @broad-dog-22463 ! Just trying to see whether u had a chance to reproduce the issue reported above.
@stocky-restaurant-98004 Hate to be a pest about this but is there a plan to address the issue cause if there isn’t, i can just drop using this provider and do it directly in terraform. Would just like to know.
b
Hi @mysterious-area-77666, I am looking at this issue - I am trying to identify what we can do in Pulumi to fix it
m
Awesome! Thanks for the update and really appreciate it. Please let me know if i can help in any way
a
@broad-dog-22463 any chance you've had the opportunity to make progress on the Fastly provider?
b
Sorry @astonishing-quill-88807 - I got pulled to an urgent issue this week for the AWS provider. I promise to get back to this now that I have fixed the AWS provider
a
No problem, thanks for the update!
m
@broad-dog-22463 Anything new on this?
a
@broad-dog-22463 do you have any pointers of where in the code we might look in order to help tackle this problem? We're happy to get our hands dirty if it will help keep things moving. Thanks for all of your help!
s
So here's the possible sources of this issue, in order of likelihood: 1. An upstream bug in the TF provider. (I think I checked the Issues and did not see anything for this a while back, but it's probably worth checking again.) If it can be reproduced in TF, we know it's not on the Pulumi side, since we're relying on the TF provider to make the correct API call (although there can occasionally be an issue that can be resolved with a mapping change in the Pulumi provider). 2. We could need an additional mapping of something in provider/resources.go. 3. It could be an issue with the Fastly API itself (less likely). Nothing we can do in this case (but still worth tracking). Hope this helps.
a
Thanks, I'll take a look and see if there's anything we can figure out.
@stocky-restaurant-98004 @broad-dog-22463 I dug through the code a bit and added a comment to the issue on the pulumi-fastly provider with what I've found. I'm not sure what specific changes will be needed but I'm happy to help in any way I can.
@stocky-restaurant-98004 @broad-dog-22463 can you give any indication as to whether this bug has been prioritized at all, or if there is a loose ETA on digging into it? We are currently completely blocked on using Pulumi to manage our Fastly deployments. In the absence of guidance on whether/when this might be looked at we'll likely need to fall back to using the TF provider directly.
s
To clarify, can you confirm the minimal steps to reproduce are as follows: 1. Create a Fastly Service VCL with a domain that includes a comment (I figure a blank comment may also trigger the bug since it's an optional field per the docs) 2.
pulumi up
3. Update the domain comment 4.
pulumi up
again 5. 💥
Also, I'm sorry this is taking so long to resolve. This is not a scenario in a bridged provider that I'm familiar with.
a
To my knowledge yes, that is the minimal reproduction, and I understand the time pressures. Thank you for taking the time to respond and give this attention. This comment in our own tracking issue has a minimal set of Python code to reproduce the bug. https://github.com/mitodl/ol-infrastructure/issues/652#issuecomment-1075220987
s
Working on this now. I've confirmed I can update a comment on a ServiceVCL in Terraform (provider v1.1.4).
And it's reproducible in TypeScript with the corresponding provider version
Interesting... it seems to work if you run
pulumi up
a second time.
Still looking into this - hang in there!
I have a possible fix. Need to make a bug fix release of the Fastly provider and run the same test and see if it resolves the issue.
a
Awesome! Thank you for taking the time to dig into it
s
No problem. I just put a release out of the provider (4.0.2). Still need to test and make sure the issue is resolved. Will do tonight or tomorrow AM.
Ugh. Looks like it's still not resolved.
To give you an idea of just how tricky this issue was, here's the comment that summarized this issue, written by one of our most senior and tenured ICs. You'd basically have to be an expert in both Pulumi and Terraform to fix this. (Publishing the fix now.)
Copy code
// FORK: The Fastly provider takes advantage of TF's behavior in a somewhat novel way that causes a bug when the
	// same provider is bridged in Pulumi. Therefore, we need to add the following block of code to ensure consistent
	// behavior in the bridged provider.
	//
	// Detailed explanation:
	//
	// The TF provider interface looks like this:
	//   Plan(prior state, config) -> planned state
	//   Apply(prior state, planned state, config) -> new state
	//
	// In the Plan() stage, diffs include any CustomizeDiff funcs in the schema. However, Terraform's behavior (at
	// least after v2.10.1 of the plugin SDK) is to strip diff customizers when performing an Apply.
	//
	// The Pulumi interface looks like this:
	//   Diff(prior state, config) -> diff result
	//   Update(prior state, config) -> new state
	//
	// In both cases, Pulumi does not strip diff customizers.
	//
	// For this resource, the effect of the difference in behavior is that during a call to Apply, the cloned_version
	// attribute is not treated as computed, even though in the Plan it is treated as computed. This difference is
	// because the cloned_version attribute is marked as computed by CustomizeDiff, which Pulumi runs during Update but
	// TF does not run during Apply. The Pulumi provider reads the cloned_version field in its implementation of Update
	// for that resource, so in TF it gets back the last actual value, but with Pulumi it gets back the zero value (0)
	// because the new value is not known.
The release is out. Please confirm that the issue is resolved for you.
m
Just tried it out and it WORKS! Thanks a lot for looking into this and figuring it out. Will surely help us with some of our Fastly config drift that we’re starting to see since we’ve been handling it manually
woohoo 1
s
Sorry this took so long! If it's any comfort, it took so long because it was a very, very difficult bug to fix diagnose, let alone fix.
a
Not a problem, we totally understand how these kinds of thing can snowball and we greatly appreciate the effort put into finding the fix! 😄
s
Hope the rest of your Pulumi experience is buttery smooth!
a
It's been great so far 🙂