big-architect-71258
07/25/2023, 3:15 PMExtraFunctions
field in in the ProviderInfo
struct inside TF Bridge. How is it supposed to be used?
Backgroud: I wanna add a getClientToken
function to the Azure Classic provider. Just like the function which is available in the Azure Native provider.
I think I created the declaration correctly because tfgen
creates the `schema.json`file with the added function definition without an error.
ExtraFunctions: map[string]pschema.FunctionSpec{
"azure:core/getClientToken:getClientToken": {
Description: "Use this function to get an Azure authentication token for the current login context.",
Inputs: &pschema.ObjectTypeSpec{
Type: "string",
Description: "Optional authentication endpoint. Defaults to the endpoint of Azure Resource Manager.",
},
Outputs: &pschema.ObjectTypeSpec{
Type: "string",
Description: "OAuth token for Azure Management API and SDK authentication.",
},
},
},
"azure:core/getClientToken:getClientToken": {
"description": "Use this function to get an Azure authentication token for the current login context.",
"inputs": {
"description": "Optional authentication endpoint. Defaults to the endpoint of Azure Resource Manager.",
"type": "string"
},
"outputs": {
"description": "OAuth token for Azure Management API and SDK authentication.",
"type": "string"
}
},
But now I'm stuck, because I don't know where to put the function implementation. Can someone help me here?
I found this PR of @microscopic-pilot-97530 that sheds some light on this. But neither I know how to create a multiplexed provider nor do I know how to
create an "overlay" 😀
https://github.com/pulumi/pulumi-terraform-bridge/pull/625microscopic-pilot-97530
Invoke
.
You’d have an if
check to see if it is azure:core/getClientToken:getClientToken
, in which case it’d run your implementation.
Otherwise, it’d forward the call to the bridged providerbig-architect-71258
07/25/2023, 4:43 PMRead(ctx context.Context, request *rpc.ReadRequest) (*rpc.ReadResponse, error)
method only?microscopic-pilot-97530
Invoke(ctx context.Context, request *rpc.InvokeRequest) (*rpc.InvokeResponse, error)
big-architect-71258
07/25/2023, 5:26 PMMuxWith
function. Would this the preferred way to add additional stuff to a TF provider. I mean in contrast to the implementation in the Docker provider.microscopic-pilot-97530
MuxWith
. @ancient-policeman-24615 might be able to help answer your question.
An alternative approach would be to fork the TF provider, add the new data source to it, and bridge your fork. But then you have to maintain the fork.ancient-policeman-24615
07/25/2023, 5:43 PMMuxWith
is about half of a way to extend a TF provider, but I don’t think that code path is fully implemented.
https://github.com/pulumi/pulumi-terraform-bridge/blob/beb85b03de3e2b1a7c43957486e302fbb49d56fe/pkg/tfbridge/main.go#L83-L84
I believe that MuxWith
will allow you to serve another provider as an extension to a bridged provider… but it will not allow you to participate in schema generation for that provider. If I was going to pursue a TF bridge provider extension, I would start with MuxWith
. Right now there isn’t a blessed way to do this, so it will probably require either forking the bridge and adding tfgen support for <http://github.com/pulumi-terraform-bridge/x/muxer|github.com/pulumi-terraform-bridge/x/muxer>
or writing a custom Main
function for the schema generation (which amounts to the same thing).big-architect-71258
07/25/2023, 6:19 PMExtraFunctions
property of the ProviderInfo
struct to create an additional, not existing function in the Azure TF provider. tfgen
added this function to the schema.json
just fine. So I suspect that when I use MuxWith
with a provider implementation that detects and executes the new function, this should do the trick. Am I on the wrong track?muxer
has the following implementation of Invoke
func (m *muxer) Invoke(ctx context.Context, req *rpc.InvokeRequest) (*rpc.InvokeResponse, error) {
server := m.getFunction(req.GetTok())
if server == nil {
return nil, status.Errorf(codes.NotFound, "Invoke '%s' not found.", req.GetTok())
}
return server.Invoke(ctx, req)
}
What to my mind will eventually call functions from a provider implementation, if the Token points to the correct provider implementation. So I think I had to change the function URN from azure:core/getClientToken:getClientToken
to something different, so that the muxer of TF pridge is able to find the additional provider implemtation which then executes the new function. Right?main.go
of the Azure Classic provider, so that the MuxWith
is called with the additional provider implementation.main.go
of x/muxer
the muxer will be able to create a schema for the muxed provider.
// - GetSchema: When Mux is called, GetSchema is called once on each server. The muxer
// merges each schema with earlier servers overriding later servers. The origin of each
// resource and function in the presented schema is remembered and used to route later
// resource and function requests.
ancient-policeman-24615
07/26/2023, 12:56 PMtfbridge.ProviderInfo.ExtraFunctions
it will populate the bridged provider’s schema, not the other provider’s schema. I think that will be sufficient if you manually create the mappings for the muxer.big-architect-71258
07/26/2023, 3:09 PM"muxer"
key from the bridge-metadata.json
file. When I compile the provider, the metadata does not contain that key (object) at the top level of the .json file. I even didn't find any code in TF Bridge.
No wonder though because since now I only added the tfbridge.ProviderInfo.ExtraFunctions
and eventually your comment "create the mappings for the muxer" makes perfectly sense. But what's the best way to jump into the process which loads the Provider metadata. I'm thinking about to replace the current implementation in resource.go
which uses MetadataInfo: tfbridge.NewProviderMetadata(metadata),
with a custom function, that first loads the Metadata using tfbridge.NewProviderMetadata(metadata)
and then adds the required muxer
configuration.
If you think that makes any sense, one question persists: I saw the muxer
dispatch table can be created using MergeSchemasAndComputeDispatchTable
. But the function requires an slice of []schema.PackageSpec
. Is there a way to create a PackageSpec
for a bridged Provider?x/muxer/main.go
😀
Is this the way to go?
schemas := make([]schema.PackageSpec, len(servers))
for i, s := range servers {
resp, err := s.GetSchema(context.Background(), req)
contract.AssertNoErrorf(err, "Server %d failed GetSchema", i)
o := schema.PackageSpec{}
err = json.Unmarshal([]byte(resp.GetSchema()), &o)
contract.AssertNoErrorf(err, "Server %d schema failed to parse", i)
schemas[i] = o
}
tfbridge.ProviderInfo.ExtraFunctions
and MuxWith
because I need a real separate schema anyway. But will this break SDK generation?muxer
but the "manual" approach @microscopic-pilot-97530 used in the Docker provider seems a better fit here.ancient-policeman-24615
07/26/2023, 4:39 PMbig-architect-71258
08/14/2023, 7:31 AMmuxer
Metadata which is required by the muxer
implementation when using MuxWith
.
To accomplish this I added a new property MuxWith []pschema.PackageSpec
to type ProviderInfo struct
. The Generate
method of TFGEN Generator
then calls MergeSchemasAndComputeDispatchTable
to create the muxed schemas and the dispatchTable
for the muxer
attribute in the provider Metadata.
Generation of schema.json
and bridge-metadata.json
work just fine and the expected data is found in each file respectively.
Before I go on I wanna ask, if this is the way to go, or, because I saw that the implementation of the muxer for PF is completely different, if there is a another way to pursue here?
@enough-garden-22763name: azure-test
runtime: yaml
description: A Pulumi YAML project to test the muxed Azure Classic Provider
variables:
current:
fn::invoke:
function: azure:core:getClientConfig
options:
provider: ${provider}
clientToken:
fn::invoke:
function: azure:core:getClientToken
options:
provider: ${provider}
outputs:
currentClientId: ${current.clientId}
accessToken: ${clientToken.token}
resources:
provider:
type: pulumi:providers:azure
properties:
skipProviderRegistration: true
The following output is generated:
Updating (default):
Type Name Status
+ pulumi:pulumi:Stack azure-test-default created (3s)
+ └─ pulumi:providers:azure provider created (0.01s)
Outputs:
accessToken : "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUz...xpNNA"
currentClientId: "8d8e346e-fd94-4a3b-ac01-86dbcd543f3f"
Resources:
+ 2 created
Duration: 4s
Whereas the azure:core:getClientToken
function is muxed in with the other functions (like: azure:core:getClientConfig
) of the upstream TF provider.MuxWith
and the new property MuxWIth
in ProviderInfo
(it required two configurations steps at two different places) I replaced the property MuxWith []pschema.PackageSpec
with MuxWith []muxer.Provider
where muxer.Provider
is defined as an interface as follows:
type Provider interface {
GetSpec() (schema.PackageSpec, error)
GetInstance(host *provider.HostClient) (pulumirpc.ResourceProviderServer, error)
}
With this approach I could remove the MuxWith
option on the func Main
in pkg/tfbridge/main.go
because in pkg/tfbridge/serve.go
it's now possible to create the muxed provider instances directly. tfgen
in pkg/tfgen/generate.go
has been changed to use GetSpec()
respectively.
With those changes in place and some adjustments in x/muxer/mapping.go
(especially copying type definitions between schemas) it is now possible to either implemet a mxued provider using the bare `ProviderSpec`approach (via UnimplementedResourceProviderServer
) or even the pulumi-go-provider
. The changes to x/muxer/mapping.go
even supports the complete replacement of an upstream resource or function (data source).
Please let me know if this would be a way to implement muxed providers for TF bridge and if I should add a PR (including the Azure Classic provider which I used as an example implementation)
@enough-garden-22763enough-garden-22763
08/22/2023, 2:07 PMbig-architect-71258
08/22/2023, 2:08 PMenough-garden-22763
08/22/2023, 2:08 PMx/muxer
from a desire to be able to mix-match all pulumi providers, however it's experimental yet (note the x
) and it's only really used in anger in the Bridge PF/SDkv2 use case. This is not a great state of affairs and if we could generalize this out that'd make for a better world.package muxer // import "<http://github.com/pulumi/pulumi-terraform-bridge/x/muxer|github.com/pulumi/pulumi-terraform-bridge/x/muxer>"
CONSTANTS
const SchemaVersion int32 = 0
The version expected to be specified by GetSchema
TYPES
type DispatchTable struct {
// Has unexported fields.
}
func MergeSchemasAndComputeDispatchTable(schemas []schema.PackageSpec) (DispatchTable, schema.PackageSpec, error)
type Endpoint struct {
Server func(*provider.HostClient) (rpc.ResourceProviderServer, error)
}
type GetMappingArgs interface {
Fetch() []GetMappingResponse
}
type GetMappingResponse struct {
Provider string
Data []byte
}
type Main struct {
Servers []Endpoint
// An optional pre-computed mapping of functions/resources to servers.
DispatchTable DispatchTable
// An optional pre-computed schema. If not provided, then the schema will be
// derived from layering underlying server schemas.
//
// If set, DispatchTable must also be set.
Schema string
GetMappingHandler map[string]MultiMappingHandler
}
Mux multiple rpc servers into a single server by routing based on request
type and urn.
Most rpc methods are resolved via a schema lookup based precomputed
mapping created when the Muxer is initialized, with earlier servers getting
priority.
For example:
Given server s1 serving resources r1, r2 and server s2 serving r1, r3, the muxer
m1 := Mux(host, s1, s2) will dispatch r1 and r2 to s1. m1 will dispatch only r3 to
s2. If we swap the order of of creation: m2 := Mux(host, s2, s1) we see different
prioritization. m2 will serve r1 and r3 to s2, only serving r2 to s1.
Most methods are fully dispatch based:
- Create, Read, Update, Delete, Check, Diff: The type is extracted from
the URN associated with the request. The server who's schema provided
the resource is routed the whole request.
- Construct: The type token is passed directly. The server who's schema
provided the resource is routed the whole request.
- Invoke, StreamInvoke, Call: The type token is passed directly. The
server who's schema provided the function is routed the whole request.
Each provider specifies in it's schema what options it accepts as
configuration. Config based endpoints filter the schema so each provider
is only shown keys that it expects to see. It is possible for multiple
subsidiary providers to accept the same key.
- CheckConfig: Broadcast to each server. Any diffs between returned
results errors.
- DiffConfig: Broadcast to each server. Results are then merged with the
most drastic action dominating.
- Configure: Broadcast to each server for individual configuration.
When computing the returned set of capabilities, each option is set to
the AND of the subsidiary servers. This means that the Muxed server is
only as capable as the least capable of its subsidiaries.
A dispatch strategy doesn't make sense for methods related to the provider
as a whole. The following methods are broadcast to all providers:
- Cancel: Each server receives a cancel request.
The remaining methods are treated specially by the Muxed server:
- GetSchema: When Mux is called, GetSchema is called once on each server.
The muxer merges each schema with earlier servers overriding later
servers. The origin of each resource and function in the presented
schema is remembered and used to route later resource and function
requests.
- Attach: `Attach` is never called on Muxed providers. Instead the host
passed into `Mux` is replaced. If subsidiary servers where constructed
with the same `host` as passed to `Mux`, then they will observe the new
`host` spurred by `Attach`.
- GetMapping: `GetMapping` dispatches on all underlerver Servers.
If zero or 1 server responds with a non-empty data section, we call
GetMappingHandler[Key] to merge the data sections, where Key is the key
given in the GetMappingRequest.
func (m Main) Server(host *provider.HostClient, module, version string) (pulumirpc.ResourceProviderServer, error)
type MultiMappingHandler = func(GetMappingArgs) (GetMappingResponse, error)
big-architect-71258
08/22/2023, 2:12 PMmuxer
property. Which in turn is very efficient because the token points to the index of the provider which eventually implements the token call.enough-garden-22763
08/22/2023, 2:12 PMbig-architect-71258
08/22/2023, 2:13 PMenough-garden-22763
08/22/2023, 2:14 PMbig-architect-71258
08/22/2023, 2:16 PMenough-garden-22763
08/22/2023, 2:16 PMbig-architect-71258
08/22/2023, 2:21 PMenough-garden-22763
08/22/2023, 2:27 PMbig-architect-71258
08/25/2023, 10:54 AMtfgen
to generate the required muxer
entry in the provider metadata and changes to the `Serve`method of tfbridge
to initialize all configured providers.
2. The branch muxed-provider in the pulumi-azure repository is an example how the changed muxer implementation in TF bridge can be used to add a new function to an upstream TF provider and how to completely replace an existing function. I used a "raw" approach by implementing a provider using the UnimplementedResourceProviderServer
struct. Obviously the mux provider could be implemented via the pulumi-provider-go framework and by implementing the muxer.Provider
interface.pulumi-go-provider
module. Made the code much cleaner compared to the last approach using the UnimplementedResourceProviderServer
.
However I had to add some code to pulumi-go-provider
so that the generated Pulumi Token for the two functions and some types could be statically set to a specific value. Otherwise I wasn't able to overwrite the azure:core/getResources:getResources
data source (function) from the upstream TF Provider.
I prepared two new branches for a review:
1. https://github.com/pulumi/pulumi-go-provider/compare/main...tmeckel:pulumi-go-provider:feat/set-token contains my changes to pulumi-go-provider
to support setting the Pulumi token
2. https://github.com/pulumi/pulumi-azure/compare/master...tmeckel:pulumi-azure:feat/mux-provider-go shows the implementation of a muxed provider for a wrapped TF provider using the pulumi-go-provider
I'm keen on hearing from you guys. With the last iteration I'm pretty happy what I've achieved and it would be awesome if Pulumi TF Bridge would support muxing of providers in some way soon..patch
) get applied to the upstream TF provider during build. Especially the 0003-Add-new-resources.patch and 0006-Add-privatedns-parse.patch and potentially the 0005-Modify-resources.patch could be replaced by a muxed provider implementation. What will make those changes to the upstream provider much more reliable than applying source code patches to an upstream project.
Maybe you guys could involve the maintainers of the Azure Classic provider into this discussion here.enough-garden-22763
09/06/2023, 1:36 PMancient-policeman-24615
09/26/2023, 9:38 PMSetToken
interface to what we actually respect. I would love it if you could take a look.
PR is https://github.com/pulumi/pulumi-go-provider/pull/129.big-architect-71258
09/27/2023, 9:13 AM