I m attempting to bridge a terraform provider that uses the Pulumi Community #package-authoring

I'm attempting to bridge a terraform provider that...

prehistoric-salesclerk-5929

07/11/2024, 3:50 PM

I'm attempting to bridge a terraform provider that uses the terraform-plugin-sdk v1 and running into a few issues on the

make tfgen

step. Specifically, its throwing this error which looks like it may not be able to parse the examples correctly? Anyone know what I may be doing wrong here?

Copy code

➜  pulumi-jetstream git:(main) make tfgen   
[ -x /opt/homebrew/bin/pulumi ] || curl -fsSL <https://get.pulumi.com> | sh
pulumi plugin install resource random 4.3.1
cd provider && go build -o /Users/terryyanko/triplewhale/pulumi-jetstream/bin/pulumi-tfgen-jetstream -ldflags "-X <http://github.com/Triple-Whale/pulumi-jetstream/provider/pkg/version.Version=0.0.1-alpha.1720709561+f0ffa6ba|github.com/Triple-Whale/pulumi-jetstream/provider/pkg/version.Version=0.0.1-alpha.1720709561+f0ffa6ba>" <http://github.com/Triple-Whale/pulumi-jetstream/provider/cmd/pulumi-tfgen-jetstream|github.com/Triple-Whale/pulumi-jetstream/provider/cmd/pulumi-tfgen-jetstream>
/Users/terryyanko/triplewhale/pulumi-jetstream/bin/pulumi-tfgen-jetstream schema --out provider/cmd/pulumi-resource-jetstream
warning: Found <elided> in description for [jetstream_consumer], but was able to preserve the examples. The description proper will be dropped in the Pulumi provider.
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.reformatExamples({0x14000999260|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.reformatExamples({0x14000999260>, 0x2, 0x2})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/docs.go:619 +0xc18
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*tfMarkdownParser).parse(0x14000776638|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*tfMarkdownParser).parse(0x14000776638>, {0x140009a0000, 0x283, 0x284})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/docs.go:580 +0x1e0
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.parseTFMarkdown(0x0|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.parseTFMarkdown(0x0>?, {0x102703668?, 0x140001cefc0?}, {0x10194ef24?, 0x10?}, {0x140009a0000?, 0x14000776998?, 0x100394fbc?}, {0x14000d0bef0?, 0x104054108?}, ...)
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/docs.go:493 +0xfc
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.getDocsForResource(0x140007b6388|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.getDocsForResource(0x140007b6388>, {0x1026f3ce0, 0x140002ff340}, {0x10194ef24, 0x9}, {0x101962a37, 0x12}, {0x102703668, 0x140001cefc0})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/docs.go:314 +0x18c
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).gatherResource(0x140007b6388|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).gatherResource(0x140007b6388>, {0x101962a37, 0x12}, {0x10270ef60, 0x14000c07a40}, 0x140001cefc0, 0x0)
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/generate.go:1210 +0x374
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).gatherResources(0x140007b6388)|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).gatherResources(0x140007b6388)>
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/generate.go:1154 +0x274
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).gatherPackage(0x140007b6388)|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).gatherPackage(0x140007b6388)>
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/generate.go:1012 +0x1cc
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).Generate(0x140007b6388)|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.(*Generator).Generate(0x140007b6388)>
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/generate.go:870 +0x54
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.Main.func1({{0x10194eeb8|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.Main.func1({{0x10194eeb8>, 0x9}, {0x102106500, 0x1f}, {0x16fa832a2, 0x6}, {{0x10271dfe0, 0x14000a98600}, {0x10194eeb8, 0x9}, ...}, ...})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/main.go:48 +0x50
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.newTFGenCmd.func1(0x0|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.newTFGenCmd.func1(0x0>?, {0x14000736a20, 0x1, 0x0?})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/main.go:164 +0x450
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.newTFGenCmd.RunFunc.func4(0x0|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.newTFGenCmd.RunFunc.func4(0x0>?, {0x14000736a20?, 0x0?, 0x0?})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.113.0/go/common/util/cmdutil/exit.go:99 +0x28
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.newTFGenCmd.RunFunc.RunResultFunc.func6(0x1400097a608|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.newTFGenCmd.RunFunc.RunResultFunc.func6(0x1400097a608>, {0x14000736a20, 0x1, 0x3})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.113.0/go/common/util/cmdutil/exit.go:115 +0x34
<http://github.com/spf13/cobra.(*Command).execute(0x1400097a608|github.com/spf13/cobra.(*Command).execute(0x1400097a608>, {0x140001dc010, 0x3, 0x3})
        /Users/terryyanko/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0x828
<http://github.com/spf13/cobra.(*Command).ExecuteC(0x1400097a608)|github.com/spf13/cobra.(*Command).ExecuteC(0x1400097a608)>
        /Users/terryyanko/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
        /Users/terryyanko/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.MainWithCustomGenerate({_|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.MainWithCustomGenerate({_>, _}, {_, _}, {{0x10271dfe0, 0x14000a98600}, {0x10194eeb8, 0x9}, {0x0, 0x0}, ...}, ...)
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/main.go:61 +0x54
<http://github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.Main({_|github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen.Main({_>, _}, {_, _}, {{0x10271dfe0, 0x14000a98600}, {0x10194eeb8, 0x9}, {0x0, 0x0}, ...})
        /Users/terryyanko/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.80.0/pkg/tfgen/main.go:40 +0x64
main.main()
        /Users/terryyanko/triplewhale/pulumi-jetstream/provider/cmd/pulumi-tfgen-jetstream/main.go:25 +0x8c
make: *** [tfgen] Error 2

big-architect-71258

07/11/2024, 4:16 PM

The code at the location of the panic assumes that there is always (!) at least ONE section header

Copy code

matches := exampleHeaderRegexp.FindStringSubmatch(s[0])

Copy code

func reformatExamples(sections [][]string) [][]string {
	canonicalExampleUsageSectionIndex := -1
	var exampleUsageSection []string
	var exampleSectionIndices []int
	for i, s := range sections {
		matches := exampleHeaderRegexp.FindStringSubmatch(s[0])
		if len(matches) == 0 {
			continue
		}

Seems the documentation of the provider does not adhere to requirements that even Hashicorp stipulates for correct documentation for a resource or data source. You can open an issue in the Pulumi Terraform Bridge repository to get the code adjusted for such non compliant documentation. Or get the docs fixed by the maintainer of the Terraform provider. But because they state the project (provider) is deprecated I doubt they'll change the docs as they work for their purpose.

prehistoric-salesclerk-5929

07/11/2024, 4:24 PM

ahh, gotcha. Actually I forked that project as well and am using it as my source repo so it's NP to fix the docs. Thank you again!

prehistoric-salesclerk-5929

07/11/2024, 4:27 PM

Is this the doc you were referring to related to the provider documentation requirements from HC? https://developer.hashicorp.com/terraform/registry/providers/docs

big-architect-71258

07/11/2024, 4:30 PM

Exactly

big-architect-71258

07/11/2024, 4:31 PM

Typically you generate the docs from the code, not by hand. But I suspect that for v1 TF providers this is not possible.

big-architect-71258

07/11/2024, 4:35 PM

I you want or have to correct the docs manually, the section header that is expected must match the following regex

Copy code

var exampleHeaderRegexp = regexp.MustCompile(`(?i)^(## Example Usage\s*)(?:(?:(?:for|of|[\pP]+)\s*)?(.*?)\s*)?$`)

and ensure that all sections use the heading level 2 Markdown tag

##

Copy code

sections := splitGroupLines(markdown, "## ")

👍 1

big-architect-71258

07/11/2024, 4:42 PM

The https://github.com/Triple-Whale/terraform-provider-jetstream/blob/main/docs/resources/jetstream_kv_entry.md?plain=1 is wrongly formatted.

👍 1

prehistoric-salesclerk-5929

07/11/2024, 5:24 PM

I just went ahead and regenerated the entire docs dir via

tfplugindocs

and that appears to have worked, thank you!

🔥 1

prehistoric-salesclerk-5929

07/11/2024, 5:27 PM

might not be a bad idea at minimum improve the error handling around the terraform generation with tfgen to account for the possibility that the source doc formatting isn't correct. Either that or maybe just throw a warning instead.

👍🏻 1

big-architect-71258

07/11/2024, 5:28 PM

Yes that's true. That's why I came up with asking you to open up an issue in the Pulumi Terraform Bridge repository.

👍 1

prehistoric-salesclerk-5929

07/11/2024, 5:30 PM

No problem! I'll go ahead and create the issue

🔥 1

big-architect-71258

07/11/2024, 5:30 PM

Thank you. Much appreciated!

prehistoric-salesclerk-5929

07/11/2024, 5:54 PM

Sure thing! https://github.com/pulumi/pulumi-terraform-bridge/issues/2176

🙏 1

prehistoric-salesclerk-5929

07/11/2024, 8:37 PM

Also opened up a PR with some minor changes to bring the docs up to date.

👍🏻 1

big-architect-71258

07/11/2024, 9:08 PM

Because people often struggle with the boilerplate repos I created a Cookiecutter template, which automatically wraps a Terraform Provider into a Pulumi provider and does the heavy lifting often by 100%. https://github.com/tmeckel/pulumi-tf-provider-cookiecutter

👍 1

prehistoric-salesclerk-5929

07/12/2024, 1:36 PM

Ahh nice! Wish I would have found that first haha

big-architect-71258

07/12/2024, 1:38 PM

Well I advertise it constantly, even inside Pulumi. But it didn't make it into the official documentation.

prehistoric-salesclerk-5929

07/12/2024, 1:38 PM

in fairness I didn't look too hard, my co-worker just pointed me directly to the template 😆

big-architect-71258

07/12/2024, 1:38 PM

You mean the boilerplate?

prehistoric-salesclerk-5929

07/12/2024, 1:39 PM

yes sorry

🤗 1

big-architect-71258

07/12/2024, 1:39 PM

No reason to be sorry.

3 Views

Open in Slack

Previous Next