👋 @bored-oyster-3147 another community member is interested in contributing to the C# Automation API effort. Meet @tall-needle-56640 👋 Are there any areas where you feel like there are opportunities to collaborate or parallelize? Perhaps looking into inline programs and implementation of the gRPC language server as I know you said you might not get to that part of the implementation in your first pass.

Hi @tall-needle-56640. You can definitely start the language server implementation if you want. You can see what I've done so far here. After that is done it's just a matter of wiring it up with the inline entry point in

LocalWorkspace

. @lemon-agent-27707 I think I'm still awaiting a response on some questions I had in my most recent comment on that PR. But I haven't returned to this yet since before thanksgiving as I have been busy. I'm hoping to get back to this this week, either tomorrow or friday. I will be working on the deserialization of project settings when I return, both YAML & JSON. I have been putting it off since it is a little more complex without the duck typing the other platforms have, so I'll need to write some custom converters I think. Shouldn't be too bad though but should be fairly easy for @tall-needle-56640 to work without colliding. How do we want to deal with merging? Is my PR into

pulumi/auto/dotnet

fine as-is?

If we are both raising PRs against the same branch, and are not working on the same feature, merges shouldn't really be a problem.

The bullet about adding

InlineProgram

support here

Ah, my apologies! I missed that while I was out last week. I'll take a look at get a response to you today. @tall-needle-56640 the language server is a gRPC service that needs to be implemented in C# to enable inline programs. The typescript reference is here: https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/x/automation/server.ts If an inline program exists in the workspace, then

up

and

preview

spin up this server and invert control with the engine https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/x/automation/stack.ts#L147-L169

@bored-oyster-3147 Or, if I'm understanding correctly, I could create a pull request to your branch when it's ready.

that would work too.

Is there a C# equivalent to

runtime.runInPulumiStack

? Or does that still need to be developed?

The equivalent is

Deployment.RunAsync

OK Thanks. Now I need to set the config. You mentioned C# having a global variable problem there, but I don't see that. The real problem, I think, is that the config is parsed from env. vars. and cannot otherwise be set. So my solution would be to add an instance method

SetAllConfig(IDictionary)

, so that it can be changed for each deployment instance. Something like:

public void SetAllConfig(IDictionary<string, string> config)
{
    AllConfig = config.ToImmutableDictionary();
}

What do you think?

I believe something similar was done for nodejs, this sounds like a reasonable approach to me. cc @tall-librarian-49374 who has greater familiarity with the C# runtime

I've done a lot of back and forth between reading NodeJS and Go to try to figure out this config stuff. Go's Run method accepts a custom-defined context parameter. In C#, I get

ServiceCallContext

which is generic. I couldn't find any auto-generated contexts either. Do I need to create my own? NodeJS's Run method sets the config through

runtime.setAllConfig

, but all that does is set a global config variable. But I can't figure out where it is actually used in the context of pulumi.runInPulumiStack. I need direction of which approach to choose and where the config is inserted into the stack/deployment/whatever.

So both in the case of node, and Go the config get to the RPC lang server via the request. In traditional CLI driven pulumi, the language runtime server (pulumi-language-go, pulumi-language-nodejs) sets an environment variable PULUMI_CONFIG so that when the user code executes the SDKs can parse that environment variable to read the config. CLI invokes language runtime (a go program) -> language runtime sets env vars -> language runtime runs user code -> user code parses env var and stores config in the inline program case it's a little different We set up language-native runtime server written in dotnet with a pointer to the user code we want to run -> CLI makes an RPC run request to that server (req contains the config) -> RPC server needs to set the runtime config before invoking the user code. This is the point where config is parsed in the traditional CLI driven flow: https://github.com/pulumi/pulumi/blob/394c91d7f6ab7a4096f4454827690a460f665433/sdk%2Fdotnet%2FPulumi%2FDeployment%2FDeployment_Config.cs#L19 If you look at the nodejs impl, you'll see that it does the same thing with a global: https://github.com/pulumi/pulumi/blob/7d171917ead72d73531927b01e236523a1f7b276/sdk%2Fnodejs%2Fruntime%2Fconfig.ts#L20 The nodejs impl was extended to allow overwriting that global config instead of just parsing it from an env var: https://github.com/pulumi/pulumi/blob/7d171917ead72d73531927b01e236523a1f7b276/sdk%2Fnodejs%2Fruntime%2Fconfig.ts#L32 We'll probably need to extend the internal instance of deployment to allow setting all config. It looks like there is already a method there for setting a single value.

Hmmm... I'm thinking about config wrong. What I really need to know is how to inject RunInfo/Options.

You may need to refactor deployment. Right now all these values are set in a private constructor: https://github.com/pulumi/pulumi/blob/3d2e31289aba2065853d06024a3da9adfc013202/sdk%2Fdotnet%2FPulumi%2FDeployment%2FDeployment.cs#L70 Perhaps you can create a new constructor that accepts a set of overrides? Might be able to take some inspiration from how we mock things out for unit testing: https://github.com/pulumi/pulumi/blob/0bdf73341b83a1437ce061bc62418e80e365ea6c/sdk%2Fdotnet%2FPulumi%2FDeployment%2FDeployment_Run.cs#L183 Looks like this is the code that actually creates the deployment in the current CLI workflow: https://github.com/pulumi/pulumi/blob/0bdf73341b83a1437ce061bc62418e80e365ea6c/sdk%2Fdotnet%2FPulumi%2FDeployment%2FDeployment_Run.cs#L217 Feels like some of the context is coming back, I believe @tall-librarian-49374 suggested creating another implementation of Deployment for the inline program use case. Yes looking back at the C# issue there are notes in here about that: https://github.com/pulumi/pulumi/issues/5596

One option here is to create another implementation of 

Deployment

 that is used for 

InlinePrograms

 that has the 

Up

 

Preview

 etc functions exposed. You should only ever be using one form of deployment in a program, either 

Deployment.runAsync

 or 

Deployment.Up

 so this would probably work out fine.

do we have an example of what a stack settings file looks like as JSON?

https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/tests/automation/data/json/Pulumi.dev.json https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/tests/automation/data/json/Pulumi.json

perfect, thank you!

Current nodejs and go implementations don't support reading/writing structured config (

pulumi config set --path

). It just works with KVPs and requires the user to marshal/unmarshal.

hmmmm ok. What's KVPs? & can you clarify what you mean by marshal/unmarshal? I have seen that referenced in the code but am not familiar with the term. Do I even need to represent a config object in .NET then? Like is this boolean here necessary?

key value pairs. The typescript implementation tries to keep this fairly simple: https://github.com/pulumi/pulumi/blob/9f4461db375cc2cb2785cb8bf4d4ee5cbf8e8e87/sdk%2Fnodejs%2Fx%2Fautomation%2FstackSettings.ts#L20 The idea behind stack settings config is not to be an interface for manipulation, but for importing/exporting.

that makes sense to me, but in .NET the problem with:

export type StackSettingsConfigValue = string | StackSettingsSecureConfigValue | any;

is that

any

case at the end. I would like the .NET implementation to have parity with the TS implementation, but I need some way to represent that final dynamic type case that isn't too ugly.

Hm, I wonder if it would be appropriate to treat it as opaque data. For instance go has a json.RawMessage type: https://golang.org/pkg/encoding/json/#RawMessage

that could work, the

System.Text.Json

equivalent is

JsonElement

I think, so if it is an object we could just deserialize it as that and let the consumer figure it out. It means that our

StackSettingsConfigValue

would now be aware of whether the underlying config was JSON or YAML, because it would also need whatever the

Yaml.Dotnet

equivalent of

JsonElement

is. So our config value type would be something like:

{
    string? Value,
    JsonElement? ValueJson,
    YamlElement? ValueYaml,
    bool IsSecure,
    bool IsObject,
}

I don't think we should leak json/yaml into the type. Is there something one level removed?

I think that

Dictionary<string, object>

might work for JSON deserialization. Will need to experiment with that. I'll let you know

I'm walking through this example and the automation code. And I don't ever see the language server used in any meaningful way. I am very confused.

The presence of

program

in the arguments (

InlineProgramArguments

) causes the stack and the workspace to operate in "inline" program mode: https://github.com/pulumi/automation-api-examples/blob/main/nodejs/inlineProgram-ts/index.ts#L69 When

stack.up

is called, we check to see a program was specified (rather than relying on a pulumi project on disk): https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/x/automation/stack.ts#L147 When we do have a program (a function pointer), we start the language server: https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/x/automation/stack.ts#L152

Right but in that same block

languageServer.Run

is not being used. Is something else calling it? If so, from where?

The port is passed to the pulumi CLI to establish an RPC connection: https://github.com/pulumi/pulumi/blob/master/sdk/nodejs/x/automation/stack.ts#L168

Oh OK. I get it now. Sorry for all the questions, I'm learning many things at the same time. lol But I appreciate the help (and patience).

You are learning the guts of the pulumi architecture! I think it's awesome 😛artypus-8bit:

👍

I have some thoughts / questions if that's OK: 1. I think the new

Deployment

partial should reside up by the other

Deployment

partials if we're going to keep it a partial. Rather than in the Automation directory. So maybe

X/Deployment/Deployment_InlineProgram.cs

? 2. I think the LanguageServer file should be called

LanguageServer.cs

rather than

Server.cs

3. Why is

LanguageServer<T>

generic? It doesn't look like we're using the type argument anywhere. those are largely styling thoughts I had on first glance.. I will need to figure out the functionality to remark on anything else

Cool. 1. I wanted to get some more feedback before deciding on a location. 2. It's named Server.cs only because the other languages named it that. But I completely agree with you, LanguageServer.cs makes more sense. 3. Thanks. That was a relic from the nodejs code.

gotcha I didn't realize the others called it Server.cs - you were right to name it that then

@tall-needle-56640 Took a brief look and looks structurally in the right direction. There doesn't appear to be any concurrency control over Run() as there is in the go impl. I think in terms of giving a deeper review, it would be helpful to see the implementation of Stack.Up(), how the lifecycle of the grpc server is managed, and a sample program using an inline deployment to see everything end to end.

@lemon-agent-27707 Is the scenario that

Stack.Up()

is called concurrently?

I believe it's that the CLI makes duplicate requests to the Run() gRPC endpoint.

Hi @tall-needle-56640 I merged your language server implementation and I'm trying to plot out how to wire it up but I think I'm missing something. Did you have a working example? If we look here in this

Stack.Up

implementation they are starting a GRPC Server & the LanguageServer side-by-side. They then wire them together and handle the closing / shutdown of the server. It looks like though that our LanguageServer implementation is instantiating an instance of

Deployment

with your new

Deployment_Inline

constructor, and within that a new GRPC Server is instantiated as well. Is that intended? What am I missing?

This is my first gRPC project, but, from what I can tell, GrpcEngine (in C#) is not providing the same functionality as

grpc

(in TS). It seems that GrpcEngine has very limited functionaility, and doesn't have a way to add the LanguageServer to it, like in TS (i.e.

server.addService()

).

Does that workdir have a pulumi.yaml?

I see - I think I’ll need to try to get a deeper understanding of the gRPC/languageserver bits before I make this connection. Thanks for the info. It does look like the error you were getting was due to not having a project file or stack setup before selecting? Could try the CreateOrSelect method? But you are correct that the methods that would actually spin up the server don’t exist yet, that’s the final bit I gotta work on.

@lemon-agent-27707 @bored-oyster-3147 Ah... something altered my Pulumi.yaml file:

-name: PulumiPort
-runtime: dotnet
-description: A minimal Azure C# Pulumi program
+Name: asd
+Runtime:
+  Name: NodeJS
+  Options:
+Main:
+Description:
+Author:
+Website:
+License:
+Config:
+Template:
+Backend:

Hmmm yea I have an idea of what may be doing that. Serialization should be ignoring null values and the property name serialization needs a lowercase naming policy. But, we decided to defer serialization til later so I gotta pull all that out and I don’t think that will be happening anymore.

Merry Christmas!

well it shouldn't be overwriting an existing yaml file, it shouldn't be serializing null properties, & it shouldn't be serializing property names capitalized

@bored-oyster-3147 Did you see my PR with the gRPC server implementation?

The need for a web host for the GRPC bit brings in a AspNetCore reference at the bare minimum. So that leads back to questions about maybe splitting automation API into a separate class library? Will need thoughts from pulumi team on that

@bored-oyster-3147 Sounds good!

Heads up that programs are only run during up/preview, so the gRPC server won't be needed for refresh/destroy.

@lemon-agent-27707 noted, I think I remember that being the case. @tall-needle-56640 I will be fleshing out the stack actions some more tomorrow so we will be colliding there. But things that remain from the original PR that you can maybe look at are: • Workspace cancel/export/import implementation • RemoteProgram implementation