https://pulumi.com logo
Powered by
b

big-architect-71258

07/25/2023, 3:15 PM
I have a question about the 
ExtraFunctions
field in in the 
ProviderInfo
struct inside TF Bridge. How is it supposed to be used? Backgroud: I wanna add a 
getClientToken
function to the Azure Classic provider. Just like the function which is available in the Azure Native provider. I think I created the declaration correctly because 
tfgen
creates the `schema.json`file with the added function definition without an error.
Copy code
ExtraFunctions: map[string]pschema.FunctionSpec{
			"azure:core/getClientToken:getClientToken": {
				Description: "Use this function to get an Azure authentication token for the current login context.",
				Inputs: &pschema.ObjectTypeSpec{
					Type:        "string",
					Description: "Optional authentication endpoint. Defaults to the endpoint of Azure Resource Manager.",
				},
				Outputs: &pschema.ObjectTypeSpec{
					Type:        "string",
					Description: "OAuth token for Azure Management API and SDK authentication.",
				},
			},
		},
Copy code
"azure:core/getClientToken:getClientToken": {
            "description": "Use this function to get an Azure authentication token for the current login context.",
            "inputs": {
                "description": "Optional authentication endpoint. Defaults to the endpoint of Azure Resource Manager.",
                "type": "string"
            },
            "outputs": {
                "description": "OAuth token for Azure Management API and SDK authentication.",
                "type": "string"
            }
        },
But now I'm stuck, because I don't know where to put the function implementation. Can someone help me here? I found this PR of @microscopic-pilot-97530 that sheds some light on this. But neither I know how to create a multiplexed provider nor do I know how to create an "overlay" 😀 https://github.com/pulumi/pulumi-terraform-bridge/pull/625
m

microscopic-pilot-97530

07/25/2023, 3:19 PM
Our docker provider uses the multiplexing approach: https://github.com/pulumi/pulumi-docker/blob/master/provider/hybrid.go You’d likely want to do something along those lines
The implementation of your function would be in the provider’s 
Invoke
. You’d have an 
if
check to see if it is 
azure:core/getClientToken:getClientToken
, in which case it’d run your implementation. Otherwise, it’d forward the call to the bridged provider
b

big-architect-71258

07/25/2023, 4:43 PM
@microscopic-pilot-97530 Thanks for you quick and detailed reply! I'll see what I can achieve 🙂
@microscopic-pilot-97530 As I described I want to implement an additional function, merely a Data Source. Is it correct that I must implement the mentioned if statement in the 
Read(ctx context.Context, request *rpc.ReadRequest) (*rpc.ReadResponse, error)
method only?
m

microscopic-pilot-97530

07/25/2023, 5:23 PM
No, in 
Invoke(ctx context.Context, request *rpc.InvokeRequest) (*rpc.InvokeResponse, error)
b

big-architect-71258

07/25/2023, 5:26 PM
@microscopic-pilot-97530 Sorry 🙏🏼 ... you already wrote it! Overlooked it. One thing: I saw that TF Pride provides a 
MuxWith
function. Would this the preferred way to add additional stuff to a TF provider. I mean in contrast to the implementation in the Docker provider.
m

microscopic-pilot-97530

07/25/2023, 5:31 PM
I’m not familiar with 
MuxWith
. @ancient-policeman-24615 might be able to help answer your question. An alternative approach would be to fork the TF provider, add the new data source to it, and bridge your fork. But then you have to maintain the fork.
a

ancient-policeman-24615

07/25/2023, 5:43 PM
MuxWith
is about half of a way to extend a TF provider, but I don’t think that code path is fully implemented. https://github.com/pulumi/pulumi-terraform-bridge/blob/beb85b03de3e2b1a7c43957486e302fbb49d56fe/pkg/tfbridge/main.go#L83-L84 I believe that 
MuxWith
will allow you to serve another provider as an extension to a bridged provider… but it will not allow you to participate in schema generation for that provider. If I was going to pursue a TF bridge provider extension, I would start with 
MuxWith
. Right now there isn’t a blessed way to do this, so it will probably require either forking the bridge and adding tfgen support for 
<http://github.com/pulumi-terraform-bridge/x/muxer|github.com/pulumi-terraform-bridge/x/muxer>
or writing a custom 
Main
function for the schema generation (which amounts to the same thing).
Forking and maintaining the fork will be the easiest short term solution. If your interested in contributing to the bridge here, it would be appreciated.
b

big-architect-71258

07/25/2023, 6:19 PM
@ancient-policeman-24615 As I described above I used the 
ExtraFunctions
property of the 
ProviderInfo
struct to create an additional, not existing function in the Azure TF provider. 
tfgen
added this function to the 
schema.json
just fine. So I suspect that when I use 
MuxWith
with a provider implementation that detects and executes the new function, this should do the trick. Am I on the wrong track?
The 
muxer
has the following implementation of 
Invoke
Copy code
func (m *muxer) Invoke(ctx context.Context, req *rpc.InvokeRequest) (*rpc.InvokeResponse, error) {
	server := m.getFunction(req.GetTok())
	if server == nil {
		return nil, status.Errorf(codes.NotFound, "Invoke '%s' not found.", req.GetTok())
	}
	return server.Invoke(ctx, req)
}
What to my mind will eventually call functions from a provider implementation, if the Token points to the correct provider implementation. So I think I had to change the function URN from 
azure:core/getClientToken:getClientToken
to something different, so that the muxer of TF pridge is able to find the additional provider implemtation which then executes the new function. Right?
And yes, I've to change the implementation in 
main.go
of the Azure Classic provider, so that the 
MuxWith
is called with the additional provider implementation.
And according to the documentation in 
main.go
of 
x/muxer
the muxer will be able to create a schema for the muxed provider.
Copy code
//   - GetSchema: When Mux is called, GetSchema is called once on each server. The muxer
//     merges each schema with earlier servers overriding later servers. The origin of each
//     resource and function in the presented schema is remembered and used to route later
//     resource and function requests.
a

ancient-policeman-24615

07/26/2023, 12:56 PM
If you use 
tfbridge.ProviderInfo.ExtraFunctions
it will populate the bridged provider’s schema, not the other provider’s schema. I think that will be sufficient if you manually create the mappings for the muxer.
b

big-architect-71258

07/26/2023, 3:09 PM
@ancient-policeman-24615 I saw that the Muxer tries to load it's configuration from the 
"muxer"
key from the 
bridge-metadata.json
file. When I compile the provider, the metadata does not contain that key (object) at the top level of the .json file. I even didn't find any code in TF Bridge. No wonder though because since now I only added the 
tfbridge.ProviderInfo.ExtraFunctions
and eventually your comment "create the mappings for the muxer" makes perfectly sense. But what's the best way to jump into the process which loads the Provider metadata. I'm thinking about to replace the current implementation in 
resource.go
which uses 
MetadataInfo: tfbridge.NewProviderMetadata(metadata),
with a custom function, that first loads the Metadata using 
tfbridge.NewProviderMetadata(metadata)
and then adds the required 
muxer
configuration. If you think that makes any sense, one question persists: I saw the 
muxer
dispatch table can be created using 
MergeSchemasAndComputeDispatchTable
. But the function requires an slice of 
[]schema.PackageSpec
. Is there a way to create a 
PackageSpec
for a bridged Provider?
@ancient-policeman-24615 Ahhh ... found something in 
x/muxer/main.go
😀 Is this the way to go?
Copy code
schemas := make([]schema.PackageSpec, len(servers))
		for i, s := range servers {
			resp, err := s.GetSchema(context.Background(), req)
			contract.AssertNoErrorf(err, "Server %d failed GetSchema", i)
			o := schema.PackageSpec{}
			err = json.Unmarshal([]byte(resp.GetSchema()), &o)
			contract.AssertNoErrorf(err, "Server %d schema failed to parse", i)
			schemas[i] = o
		}
If I go that way It makes no sense to go with 
tfbridge.ProviderInfo.ExtraFunctions
and 
MuxWith
because I need a real separate schema anyway. But will this break SDK generation?
I'd really like to use the 
muxer
but the "manual" approach @microscopic-pilot-97530 used in the Docker provider seems a better fit here.
a

ancient-policeman-24615

07/26/2023, 4:39 PM
The manual approach is definitely more tested. It would be great if you could open an issue with your use case in the bridge. I agree that this should be possible but we just haven’t done the leg work yet.
b

big-architect-71258

08/14/2023, 7:31 AM
@ancient-policeman-24615 So my progress so far: Updated TFGEN to create the 
muxer
Metadata which is required by the 
muxer
implementation when using 
MuxWith
. To accomplish this I added a new property 
MuxWith []pschema.PackageSpec
to 
type ProviderInfo struct
. The 
Generate
method of TFGEN 
Generator
then calls 
MergeSchemasAndComputeDispatchTable
to create the muxed schemas and the 
dispatchTable
for the 
muxer
attribute in the provider Metadata. Generation of 
schema.json
and 
bridge-metadata.json
work just fine and the expected data is found in each file respectively. Before I go on I wanna ask, if this is the way to go, or, because I saw that the implementation of the muxer for PF is completely different, if there is a another way to pursue here? @enough-garden-22763
@ancient-policeman-24615 With the following YAML program
Copy code
name: azure-test
runtime: yaml
description: A Pulumi YAML project to test the muxed Azure Classic Provider

variables:
  current:
    fn::invoke:
      function: azure:core:getClientConfig
      options:
        provider: ${provider}

  clientToken:
    fn::invoke:
      function: azure:core:getClientToken
      options:
        provider: ${provider}

outputs:
  currentClientId: ${current.clientId}
  accessToken: ${clientToken.token}

resources:
  provider:
    type: pulumi:providers:azure
    properties:
      skipProviderRegistration: true
The following output is generated:
Copy code
Updating (default):
     Type                       Name                Status              
 +   pulumi:pulumi:Stack        azure-test-default  created (3s)        
 +   └─ pulumi:providers:azure  provider            created (0.01s)     


Outputs:
    accessToken    : "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUz...xpNNA"
    currentClientId: "8d8e346e-fd94-4a3b-ac01-86dbcd543f3f"

Resources:
    + 2 created

Duration: 4s
Whereas the 
azure:core:getClientToken
function is muxed in with the other functions (like: 
azure:core:getClientConfig
) of the upstream TF provider.
@ancient-policeman-24615 because I wasn't really happy with the implementation via 
MuxWith
and the new property 
MuxWIth
in 
ProviderInfo
(it required two configurations steps at two different places) I replaced the property 
MuxWith []pschema.PackageSpec
with 
MuxWith []muxer.Provider
where 
muxer.Provider
is defined as an interface as follows:
Copy code
type Provider interface {
	GetSpec() (schema.PackageSpec, error)
	GetInstance(host *provider.HostClient) (pulumirpc.ResourceProviderServer, error)
}
With this approach I could remove the 
MuxWith
option on the 
func Main
in 
pkg/tfbridge/main.go
because in 
pkg/tfbridge/serve.go
it's now possible to create the muxed provider instances directly. 
tfgen
in 
pkg/tfgen/generate.go
has been changed to use 
GetSpec()
respectively. With those changes in place and some adjustments in 
x/muxer/mapping.go
(especially copying type definitions between schemas) it is now possible to either implemet a mxued provider using the bare `ProviderSpec`approach (via 
UnimplementedResourceProviderServer
) or even the 
pulumi-go-provider
. The changes to 
x/muxer/mapping.go
even supports the complete replacement of an upstream resource or function (data source). Please let me know if this would be a way to implement muxed providers for TF bridge and if I should add a PR (including the Azure Classic provider which I used as an example implementation) @enough-garden-22763
e

enough-garden-22763

08/22/2023, 2:07 PM
Hi @big-architect-71258 could you link some code here for us to have closer look? I think you're onto something really valuable here.
b

big-architect-71258

08/22/2023, 2:08 PM
@enough-garden-22763 let me check the status of my forks. I'll prepare respective branches for an evaluation.
e

enough-garden-22763

08/22/2023, 2:08 PM
At a very high level, we started 
x/muxer
from a desire to be able to mix-match all pulumi providers, however it's experimental yet (note the 
x
) and it's only really used in anger in the Bridge PF/SDkv2 use case. This is not a great state of affairs and if we could generalize this out that'd make for a better world.
I think perhaps one day we can even lift it out of the bridge repo into it's own project with stable API guarantees.
Checking up on current muxer API:
Copy code
package muxer // import "<http://github.com/pulumi/pulumi-terraform-bridge/x/muxer|github.com/pulumi/pulumi-terraform-bridge/x/muxer>"


CONSTANTS

const SchemaVersion int32 = 0
    The version expected to be specified by GetSchema


TYPES

type DispatchTable struct {
	// Has unexported fields.
}

func MergeSchemasAndComputeDispatchTable(schemas []schema.PackageSpec) (DispatchTable, schema.PackageSpec, error)

type Endpoint struct {
	Server func(*provider.HostClient) (rpc.ResourceProviderServer, error)
}

type GetMappingArgs interface {
	Fetch() []GetMappingResponse
}

type GetMappingResponse struct {
	Provider string
	Data     []byte
}

type Main struct {
	Servers []Endpoint

	// An optional pre-computed mapping of functions/resources to servers.
	DispatchTable DispatchTable

	// An optional pre-computed schema. If not provided, then the schema will be
	// derived from layering underlying server schemas.
	//
	// If set, DispatchTable must also be set.
	Schema string

	GetMappingHandler map[string]MultiMappingHandler
}
    Mux multiple rpc servers into a single server by routing based on request
    type and urn.

    Most rpc methods are resolved via a schema lookup based precomputed
    mapping created when the Muxer is initialized, with earlier servers getting
    priority.

    For example:

        Given server s1 serving resources r1, r2 and server s2 serving r1, r3, the muxer
        m1 := Mux(host, s1, s2) will dispatch r1 and r2 to s1. m1 will dispatch only r3 to
        s2.  If we swap the order of of creation: m2 := Mux(host, s2, s1) we see different
        prioritization. m2 will serve r1 and r3 to s2, only serving r2 to s1.

    Most methods are fully dispatch based:

      - Create, Read, Update, Delete, Check, Diff: The type is extracted from
        the URN associated with the request. The server who's schema provided
        the resource is routed the whole request.

      - Construct: The type token is passed directly. The server who's schema
        provided the resource is routed the whole request.

      - Invoke, StreamInvoke, Call: The type token is passed directly. The
        server who's schema provided the function is routed the whole request.

    Each provider specifies in it's schema what options it accepts as
    configuration. Config based endpoints filter the schema so each provider
    is only shown keys that it expects to see. It is possible for multiple
    subsidiary providers to accept the same key.

      - CheckConfig: Broadcast to each server. Any diffs between returned
        results errors.

      - DiffConfig: Broadcast to each server. Results are then merged with the
        most drastic action dominating.

      - Configure: Broadcast to each server for individual configuration.
        When computing the returned set of capabilities, each option is set to
        the AND of the subsidiary servers. This means that the Muxed server is
        only as capable as the least capable of its subsidiaries.

    A dispatch strategy doesn't make sense for methods related to the provider
    as a whole. The following methods are broadcast to all providers:

      - Cancel: Each server receives a cancel request.

    The remaining methods are treated specially by the Muxed server:

      - GetSchema: When Mux is called, GetSchema is called once on each server.
        The muxer merges each schema with earlier servers overriding later
        servers. The origin of each resource and function in the presented
        schema is remembered and used to route later resource and function
        requests.

      - Attach: `Attach` is never called on Muxed providers. Instead the host
        passed into `Mux` is replaced. If subsidiary servers where constructed
        with the same `host` as passed to `Mux`, then they will observe the new
        `host` spurred by `Attach`.

      - GetMapping: `GetMapping` dispatches on all underlerver Servers.
        If zero or 1 server responds with a non-empty data section, we call
        GetMappingHandler[Key] to merge the data sections, where Key is the key
        given in the GetMappingRequest.

func (m Main) Server(host *provider.HostClient, module, version string) (pulumirpc.ResourceProviderServer, error)

type MultiMappingHandler = func(GetMappingArgs) (GetMappingResponse, error)
There seems to be some built-in facility to mux schemas but it's not super obvious; the API is currently geared toward muxing the runtime providers. In a real case you have to mux both. You have to mux schemas at tfgen time and mux providers at runtime.
We can ponder a few alternatives here how to make this more obvious/usable. Note also
b

big-architect-71258

08/22/2023, 2:12 PM
@enough-garden-22763 that's exactly what my current code does: `tfgen`is able to create the muxed schema and at runtime the muxer will detect which code should run by which provider implementation via the prepared dispatch table in the provider metadata json in the 
muxer
property. Which in turn is very efficient because the token points to the index of the provider which eventually implements the token call.
e

enough-garden-22763

08/22/2023, 2:12 PM
Regarding the docker provider Justin mentions above, it's indeed another case of hand-rolled muxing. Ideally we'd unify that with our own muxer package at some point. These projects evolved in parallel and have not cross-polinated yet.
b

big-architect-71258

08/22/2023, 2:13 PM
@enough-garden-22763 The docker provider can easily migrated to my current approach what in the end will greatly reduce the amout fo code used in the provider.
e

enough-garden-22763

08/22/2023, 2:14 PM
Yeah! Sounds nice! Looking forward to having a close look. It may be a few days as I'm keeping busy.
Ah haha, yes, "easily", as far as industrial-scale software ever changes easily 🙂 There's a surprising number of corner cases that can go wrong in these and the test bar needs to be high.
Hopefully in the end state as you say we have less code and the code we have is much more trusted.
b

big-architect-71258

08/22/2023, 2:16 PM
@enough-garden-22763 I'll prepare the branches. When they're ready on GH I'll let you know. I'll be on vacation in Crete till 9/5 starting on 25/8 anyway, so now sweat 🙂
e

enough-garden-22763

08/22/2023, 2:16 PM
Perfect
We might do a mini-design session with Ian and interested parties internally on the API suggestion and consider a few alternatives. Thanks for pushing on this! great area to improve Pulumi.
b

big-architect-71258

08/22/2023, 2:21 PM
I ever wanted a way to easily add or even replace functionality from an upstream provider so that I don't have to deal with the upstream project at all. Regarding the Azure Classic provider, the team rejected a data source to create a Bearer token from the current because that would be out of the scope of the provider even if it would be very valuable. And with my changes to the muxer this was implemented to the Pulumi Azure Classic provider in minutes.
e

enough-garden-22763

08/22/2023, 2:27 PM
Yeah having an API for this would be easier to use than managing patches to upstream.
b

big-architect-71258

08/25/2023, 10:54 AM
@enough-garden-22763 @ancient-policeman-24615 as discussed I prepared two branches in my forks of the pulumi-terraform-bridge, pulumi-azure repositories. 1. The branch implement-muxer in pulumi-terraform-bridge repo contains my changes to 
tfgen
to generate the required 
muxer
entry in the provider metadata and changes to the `Serve`method of 
tfbridge
to initialize all configured providers. 2. The branch muxed-provider in the pulumi-azure repository is an example how the changed muxer implementation in TF bridge can be used to add a new function to an upstream TF provider and how to completely replace an existing function. I used a "raw" approach by implementing a provider using the 
UnimplementedResourceProviderServer
struct. Obviously the mux provider could be implemented via the pulumi-provider-go framework and by implementing the 
muxer.Provider
interface.
Let me know if you guys need additional stuff. I'm keen on hearing about your assessment of my code.
@enough-garden-22763 @ancient-policeman-24615 I went further with the provider muxing in the Azure Classic Provider and implemented the muxed provider using the 
pulumi-go-provider
module. Made the code much cleaner compared to the last approach using the 
UnimplementedResourceProviderServer
. However I had to add some code to 
pulumi-go-provider
so that the generated Pulumi Token for the two functions and some types could be statically set to a specific value. Otherwise I wasn't able to overwrite the 
azure:core/getResources:getResources
data source (function) from the upstream TF Provider. I prepared two new branches for a review: 1. https://github.com/pulumi/pulumi-go-provider/compare/main...tmeckel:pulumi-go-provider:feat/set-token contains my changes to 
pulumi-go-provider
to support setting the Pulumi token 2. https://github.com/pulumi/pulumi-azure/compare/master...tmeckel:pulumi-azure:feat/mux-provider-go shows the implementation of a muxed provider for a wrapped TF provider using the 
pulumi-go-provider
I'm keen on hearing from you guys. With the last iteration I'm pretty happy what I've achieved and it would be awesome if Pulumi TF Bridge would support muxing of providers in some way soon.
@enough-garden-22763 @ancient-policeman-24615 I addition to my statements from yesterday, I want to add that I had a look how the Azure classic is build and I saw that there are couple of diff files (
.patch
) get applied to the upstream TF provider during build. Especially the 0003-Add-new-resources.patch and 0006-Add-privatedns-parse.patch and potentially the 0005-Modify-resources.patch could be replaced by a muxed provider implementation. What will make those changes to the upstream provider much more reliable than applying source code patches to an upstream project. Maybe you guys could involve the maintainers of the Azure Classic provider into this discussion here.
e

enough-garden-22763

09/06/2023, 1:36 PM
Sorry for the delay here Thomas, trying to clear some time to look at this properly.
a

ancient-policeman-24615

09/26/2023, 9:38 PM
@big-architect-71258 I’m looking at the pulumi-go-provider branch right now.
@big-architect-71258 I made some changes to get tests passing and restrict the 
SetToken
interface to what we actually respect. I would love it if you could take a look. PR is https://github.com/pulumi/pulumi-go-provider/pull/129.
b

big-architect-71258

09/27/2023, 9:13 AM
@ancient-policeman-24615 had a look at your PR and left a comment
4 Views
Powered by