I have question regarding testing. So I see that c...
# contribute
b
I have question regarding testing. So I see that currently conformance tests are recommended, but I have question how I can create my own custom test (that I don't want in actual Pulumi repository)? From what I've seen the tests depend on this
pulumi-test-language
binary that is compiled from upstream and I don't see a way of injecting custom tests.
e
If you've got a good general test we'd probably be ok just adding it to conformance. If you've got tests that are specific to your codebase could you do them via a small integration test instead? Given the way the conformance test runner is structured I'm not sure how we could add support for custom tests to it.
b
I'm currently looking at it - I think I've managed to extract the runner
I will test it a bit and submit PR
https://github.com/pulumi/pulumi/pull/22649/changes - AI did some weird stuff related to comments and token. I'll fix it - but the main idea is still there.
e
oh I see I don't think you need to pull it out into it's own package. It's perfectly fine to have a go package which is a library and an executable at the same time. Just lift the bits you need to write your own test runner to be public.
b
yeah - I guess that's true. That also helps me with another PR I have in mind
Copy code
// runTestingHost boots up a new instance of the language conformance test runner, `pulumi-test-language`, as well as a
// fake Pulumi engine for collecting logs. It returns the address of the fake engine and a connection to the test runner
// that can be used to manage a test suite run.
func runTestingHost(t *testing.T) (string, testingrpc.LanguageTestClient) {
	// We can't just go run the pulumi-test-language package because of
	// <https://github.com/golang/go/issues/39172>, so we build it to a temp file then run that.
	binary := t.TempDir() + "/pulumi-test-language"
	cmd := exec.Command("go", "build", "-o", binary, "<http://github.com/pulumi/pulumi/pkg/v3/testing/pulumi-test-language|github.com/pulumi/pulumi/pkg/v3/testing/pulumi-test-language>") //nolint:gosec,lll
	output, err := cmd.CombinedOutput()
	t.Logf("build output: %s", output)
	require.NoError(t, err)

	cmd = exec.Command(binary)
	stdout, err := cmd.StdoutPipe()
	require.NoError(t, err)
	stderr, err := cmd.StderrPipe()
	require.NoError(t, err)
	stderrReader := bufio.NewReader(stderr)

	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		for {
			text, err := stderrReader.ReadString('\n')
			if err != nil {
				wg.Done()
				return
			}
			t.Logf("engine: %s", text)
		}
	}()

	err = cmd.Start()
	require.NoError(t, err)

	stdoutBytes, err := io.ReadAll(stdout)
	require.NoError(t, err)

	address := string(stdoutBytes)

	conn, err := grpc.NewClient(
		address,
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithUnaryInterceptor(rpcutil.OpenTracingClientInterceptor()),
		grpc.WithStreamInterceptor(rpcutil.OpenTracingStreamClientInterceptor()),
		rpcutil.GrpcChannelOptions(),
	)
	require.NoError(t, err)

	client := testingrpc.NewLanguageTestClient(conn)

	t.Cleanup(func() {
		assert.NoError(t, cmd.Process.Kill())
		wg.Wait()
		// We expect this to error because we just killed it.
		contract.IgnoreError(cmd.Wait())
	})

	return address, client
}
Making this a function. I think it's copied into every single language_test
but that's for another time
e
yes, wasn't sure where to put that but just putting on the package is probably fine
b
Yeah, I cannot do that
I had this issue before - package main is a big magical I think and does not allow itself to be imported
e
oh right yeh, normally main would be in a subpackage like cmd
I think that might be fine to do, we can break things on this boundary, its just internal testing
b
The issue in that would be that the MR would be huge, because it would also move the test data - and would require changes also in
pulumi-java
. Well - anything that actually runs these tests.
I'll think about it - for now I'll fix the formatting issues in current PR
https://github.com/pulumi/pulumi/pull/22649 - I've moved it into
runner
inside language-test