https://pulumi.com logo
Join Slack
Powered by
Hi, I'm trying to build a new pulumi provider usin...
# package-authoring
f
Hi, I'm trying to build a new pulumi provider using the boilerplate [repo](https://github.com/pulumi/pulumi-tf-provider-boilerplate), when I execute 
make tfgen
I get errors like
Copy code
warning: unable to convert HCL example for Pulumi entity '#/functions/xenorchestra:index/getHosts:getHosts': 6 errors occurred:
	* :17,3-6: Argument or block definition required; An argument or block definition is required here.
	* :17,3-6: Argument or block definition required; An argument or block definition is required here.
	* :17,3-6: Argument or block definition required; An argument or block definition is required here.
	* :17,3-6: Argument or block definition required; An argument or block definition is required here.
	* :17,3-6: Argument or block definition required; An argument or block definition is required here.
	* :17,3-6: Argument or block definition required; An argument or block definition is required here.
where do I find the HCL that pulumi can't convert?
b
@fierce-rainbow-21947 Hard to tell because there are at least two TF providers out there for Xen Orchestra. Which one you're wrapping?
f
I'm working on the one from vates
going through the steps of the boilerplate readme and seeing these warning resulting in <100% example conversion
b
After 
make provider
you'll find a statistic about how much of the documentation an examples have been successfully converted. From my experience everything above 90% or 95% I'd consider a success. But let me have a quick look at the TF documentation if I can spot any issues that could be resolved quickly.
I quickly wrapped the provider with my Cookiecutter Template, which uses slights newer Pulumi GO components, to exclude any issues with your setup, but sadly aside your mentioned errors, there are plenty of others like: 
warning: error: Failure in parsing resource name: xenorchestra_vm, subsection: ## Schema
. So the overall percentage of converted documentation doesn't look pretty good.
Copy code
Provider:     xenorchestra
Success rate: 71.43% (90/126)

Converted 76.19% of csharp examples (16/21)
Converted 76.19% of go examples (16/21)
Converted 61.90% of java examples (13/21)
Converted 76.19% of python examples (16/21)
Converted 76.19% of typescript examples (16/21)
Converted 61.90% of yaml examples (13/21)
I'd like to loop in @enough-garden-22763 and @ancient-policeman-24615 into this, because from my point of view the docs, which are created by 
hashicorp/terraform-plugin-docs
look okay to me.
@fierce-rainbow-21947 Okay THIS looks much better 😁
Copy code
Provider:     xenorchestra
Success rate: 86.11% (93/108)

Converted 88.89% of csharp examples (16/18)
Converted 88.89% of go examples (16/18)
Converted 88.89% of java examples (16/18)
Converted 88.89% of python examples (16/18)
Converted 88.89% of typescript examples (16/18)
Converted 72.22% of yaml examples (13/18)
Please upgrade:
Copy code
<http://github.com/pulumi/pulumi-terraform-bridge/v3|github.com/pulumi/pulumi-terraform-bridge/v3> v3.80.0
<http://github.com/pulumi/pulumi/sdk/v3|github.com/pulumi/pulumi/sdk/v3> v3.113.0
And
Copy code
<http://github.com/hashicorp/terraform-plugin-sdk/v2|github.com/hashicorp/terraform-plugin-sdk/v2> => <http://github.com/pulumi/terraform-plugin-sdk/v2|github.com/pulumi/terraform-plugin-sdk/v2> v2.0.0-20240229143312-4f60ee4e2975
Aside expected error messages like 
unknown function 'tomap';
and 
Failed to generate YAML program: *model.UnaryOpExpression; Unimplemented! Needed for  -1
I still see 
3,11-12: Invalid character; This character is not used within the language., and 5 other diagnostic(s)
and 
:17,3-6: Argument or block definition required; An argument or block definition is required here.
which I can't explain. @ancient-policeman-24615 any idea on this?
Copy code
warning: unable to convert HCL example for Pulumi entity '#/resources/xenorchestra:index/vm:Vm'. The example will be dropped from any generated docs or SDKs: 1 error occurred:
        * [csharp, go, java, python, typescript, yaml] :3,11-12: Invalid character; This character is not used within the language., and 5 other diagnostic(s)

warning: unable to convert HCL example for Pulumi entity '#/functions/xenorchestra:index/getHosts:getHosts'. The example will be dropped from any generated docs or SDKs: 1 error occurred:
        * [csharp, go, java, python, typescript, yaml] :17,3-6: Argument or block definition required; An argument or block definition is required here.
Respective docs should be: 
#/resources/xenorchestra:index/vm:Vm
https://github.com/vatesfr/terraform-provider-xenorchestra/blob/master/docs/resources/vm.md 
'#/functions/xenorchestra:index/getHosts:getHosts'
https://github.com/vatesfr/terraform-provider-xenorchestra/blob/master/docs/data-sources/hosts.md
a
I’m guessing it’s because they both have 
…
blocks in their example HCL.
b
Is that illegal or does this has to be added/patched in the Pulumi code to handle such HCL?
e
Hi all, the way to work with these failures is running 
COVERAGE_OUTPUT_DIR="$PWD/cov" tfgen
then extracting the failing test case from 
./cov/byExample.json
. These can be then reported to https://github.com/pulumi/pulumi-converter-terraform - I've in fact have been doing some of this as we're rolling out an update to internal providers to start using an improved Terraform converter hosted at this repository.
https://github.com/pulumi/ci-mgmt/issues/889 to be more precise tracks the progress of the rollout
We'd like to make the new converter the default way to render TF examples (right now it is still flagged off) following the rollout. It is better but it is not however perfect.
b
@enough-garden-22763 THIS is really a great tip! Thanks! Wasn't aware of this. Definitely now in my tool belt. Is this already documented and I missed it? Or was it, up to now, tribal knowledge of Pulumi 😁
e
We're a small team so appreciate any help we can get, bug reports, upvotes or contributions 🙏
make tfgen
points to COVERAGE_OUTPUT_DIR variable but admittedly this flow with finding example failures is a bit arcane
b
@enough-garden-22763 I wrapped the provider myself for doing some analysis on the mentioned issues by @fierce-rainbow-21947. I can provide the 
json
file you mentioned if you like.
e
The JSON will have the original HCL/TF code and the error, ideal flow for us is if you could (1) verify it is happening under PULUMI_CONVERT=1 env var; (2) take that and making an issue in https://github.com/pulumi/pulumi-converter-terraform
We likely don't have bandwidth to fix issues on the old stack (pre PULUMI_CONVERT=1) and will be deprecating that soon-ish.
b
@enough-garden-22763 so I've the following files created by 
COVERAGE_OUTPUT_DIR="$PWD/cov" make tfgen
Copy code
-rw------- 1 user user 85281 Apr 19 17:44 byExample.json
-rw------- 1 user user  4440 Apr 19 17:44 byLanguage.json
-rw------- 1 user user   319 Apr 19 17:44 shortSummary.txt
-rw------- 1 user user  1017 Apr 19 17:44 summary.json
-rw------- 1 user user 25970 Apr 19 17:44 summary.md
As I understood I should open a issue in https://github.com/pulumi/pulumi-converter-terraform. Right? Were to put the files? Creating GIST in my account? Or what's the preferred way?
e
IS there a particular example that you'd like fixed cov/byExamples.json should have an entry for it
b
Okay, understood. Have to sort out the "valid" warning like YAML function not implemented and the 
tomap
HCL function. I think there are 3 problems left. Opening 3 separate issues?
e
Yeah thats' reasonable.
b
Okay, on my way 🫡
e
TYVM!
b
@enough-garden-22763 Okay, the warnings are actually real problems with the documentation. Code blocks tagged as 
hcl
or 
terraform
which are actually no Terraform (HCL) at all. Or using 
...
instead of completely specifying at least all required properties for a resource. I think I gonna close the issue I opened because I jumped to quickly on my conclusions about what is happening.
e
Well, interesting
We could attempt some data correction here, I already have a quick regex somewhere in the bridge that tries to guess "does this look like terraform"
But if you only have 2-3 of these patching your sources is probably easiest
or ignoring the warning
what we really want to do internally is correlate all warnings from the 60-odd providers in the warehouse and then go after the frequent ones... hopefully the sample is big enough to represent ecosystem issues as well
b
@fierce-rainbow-21947 after analyzing the documentation of the Terraform provider please correct the following issues there to get a better result of document conversion by TFGEN 1. 
<https://raw.githubusercontent.com/vatesfr/terraform-provider-xenorchestra/master/docs/data-sources/hosts.md>
complete the resource declaration for 
xenorchestra_vm
. 
...
are not valid here. 2. Ensure that code blocks which are tagged with 
hcl
or 
terraform
contain correct Terraform (HCL) code. 
<https://raw.githubusercontent.com/vatesfr/terraform-provider-xenorchestra/master/docs/resources/vm.md>
The 
#cloud-config
is not valid TF code. Please tag the code block with another type. I think that should drive the TFGEN success rate to 
>90%
.
> But if you only have 2-3 of these patching your sources is probably easiest @enough-garden-22763 
vatesfr
controls the Terraform and the bridged Pulumi provider. So it should be easy for them to get the issues in the TF provider docs straight. 😁
Oh I've to say the 
cov
files were really, really valuable to pin down the issues! GREAT FEATURE!!!
e
Great hidden gem.
f
oh wow, thanks for all the input! I'll have to open issues in the terraform provider then, so those docs can be fixed! 🙂
so just a quick update, we managed to convert 94% (16/17* of the examples, now 🙂 The providers in the languages build and we can start testing 👍 we'll probably also get try to get the package published on the pulumi registry \o/
b
@fierce-rainbow-21947 good to hear 👍🏼
3 Views
Open in Slack
PreviousNext
Powered by