https://pulumi.com logo
Join Slack
Powered by
[solved] Testing question: Hi all, I'm trying to l...
# general
f
[solved] Testing question: Hi all, I'm trying to learn to test Pulumi code. I wrote this pulumi program in Python to create and configure a GitHub repo. I wrapped the program in a CLI tool using the automation API. Is there a way I can mock the API calls to pulumi while running my CLI tool in some sort of testing mode, and then make assertions about what pulumi says would be created?
Copy code
import pulumi_github as github
from my_module import GithubRepo  # a dataclass with repo configuration

def _create_repo(repo_config: GithubRepo) -> github.Repository:
    repo = github.Repository(
        resource_name=repo_config.name,
        name=repo_config.name,
        archive_on_destroy=True,
        description=repo_config.description,
        visibility="private",
    )

    if repo_config.actions_variables:
        create_actions_variables(actions_variables=repo_config.actions_variables, repo_name=repo.name)

    if repo_config.actions_secrets:
        create_actions_secrets(action_secrets=repo_config.actions_secrets, repo_name=repo.name)

    for env in repo_config.environments:
        create_environment(env, repo_name=repo.name)

    create_repository_teams(admin_teams=repo_config.admin_teams, repo_name=repo.name)

    return repo
Here's how we're using the pulumi automation API
Copy code
def create_repo_pulumi_program():
    _create_repo(repo_config=...)

stack = auto.create_or_select_stack(
    project_name=project_name,
    stack_name=stack_name,
    program=create_repo_pulumi_program,
)

stack.set_config(key="github:owner", value=auto.ConfigValue(repo.owner, secret=False))

stack.up(on_output=print)
l
Yes, absolutely. Have you looked at https://www.pulumi.com/docs/using-pulumi/testing/unit/?
Note that unit testing is for units (e.g. your GithubRepo class) not programs (they're integrated, not testable units). Also, the assertions are on the inputs, not the outputs; you're testing your code, not Pulumi's/AWS'/GitHub's...
You can also use integration testing (there's a page next to the unit test page), but then there's no faking of the API calls.
f
Oh nice. So for a test of 
_create_repo()
as a unit, are you saying I could modify my 
_create_repo()
function to do something like this
Copy code
def _create_repo(repo_config: GithubRepoConfig) -> Tuple[
    github.Repository,
    List[github.RepositorySecret],
    List[github.ActionsEnvironment],
    List[github.ActionsEnvironmentSecret],
    List[github.ActionsEnvironmentVariable],
]:
    ...
And then basically run assertions on the returned object representations? So I imagine I'd be making statements like
Copy code
repo, repo_secrets, repo_envs, repo_env_secrets, repo_env_vars = _create_repo(repo_config)

assert repo_config.repo_name == repo.repo_name
assert len(repo_config.environments) == len(repo_envs)
assert repo_config.environments[0].name == repo_envs[0].name
...
Is this kind of what you're saying?
l
No. Your class is a great thing to test, assuming it's a unit. If this code is also a unit, then your class is a vehicle for inject test data into it. You tests can assert on the outputs of the resources that Pulumi creates, but those values are just the inputs: there is no real provider called, so anything that the provider does won't happen.
The point I wanted to make is that programs are not testable units. You can't wrap a full program in unit test code.
Some parts of Pulumi won't work inside the test runner, or aren't worth getting to work. Things like Pulumi config, getStack(), isDryRun(), etc.
All of those things need to be used in your program but outside of your testable units.
f
Ah, okay trying again. If I were to turn this "program" into a "class" (a pulumi component) called 
OpinionatedGithubRepo
, at that point would it make sense to attribute test that. But in the greater context of a program, integration tests are the only way? I can totally see the value of integration tests that set up real infrastructure. They're just so slow and complex to set up.
With AWS CDK, you can basically "synth" your whole stack into a JSON representation and also an object representation and ask things like "are there 3 buckets in the stack? what are their names?"
And get more granular about resources outputs, etc. I like that model because the question that seems to answer is: if I were to run this deployment and didn't have network/permissions issues what would the final stack look like?
Ahhhh, here we go! Solved:
Copy code
@pulumi.runtime.test
def test_repo_properties():
    """Test that running `stack.preview` doesn't crash."""
    repo = _create_repo(GithubRepo(
        name="testrepo",
        owner="testowner",
        description="test description",
        # environments=["Sandbox", "test-dev", "test-prod"],
        # admin_teams=["testteam"],
    ))

    def check_repo_props(args):
        repo_name, repo_desc = args
        assert repo_name == "testrepo"
        assert repo_desc == "test description"

    return pulumi.Output.all(repo.name, repo.description).apply(check_repo_props)
l
Re:
If I were to turn this "program" into a "class" (a pulumi component) called OpinionatedGithubRepo , at that point would it make sense to attribute test that.
Yes, absolutely 🙂
Re:
I can totally see the value of integration tests that set up real infrastructure. They're just so slow and complex to set up.
Pulumi has just launched Review Stacks which makes this much easier. https://www.pulumi.com/blog/review-stacks/ https://www.pulumi.com/docs/pulumi-cloud/deployments/review-stacks/
2 Views
Open in Slack
PreviousNext
Powered by