https://pulumi.com logo
Join Slack
Powered by
What would it take to get the automation api into ...
# java
f
What would it take to get the automation api into the Java SDK?
m
Any interest in helping out to make it happen?
f
Potentially, thus the question 🙂
m
🙌 We’d love your help!
I’m happy to answer questions, discuss designs, review changes, etc.
Getting a core Automation API into the Java SDK should be relatively straightforward given it would be shelling out to the CLI and the other language implementations are available to use as reference.
q
Hi, one of the guys that helped put pulumi-java together here. Last time we had this discussion (when working on pulumi-java that is) the biggest hurdle my colleague (who wrote core sdk) has seen was the fact that there's global mutable state in use (the Deployment singleton).
@microscopic-pilot-97530 given that js/ts impl does seem to have the same limitation (I might be sorely mistaken here, my node.js skills are rusty) I wanted to ask if that's actually a problem? Specifically - what are the semantics of concurrent calls to automation API? In a scenario where, let's say, pulumi is embedded in a grpc server, how should a sdk behave when two concurrent grpc requests arrive and trigger two separate executions of automation API?
m
This is only a concern when using inline programs with Automation API. Local programs in a directory don’t have the problem, and can be written in any Pulumi language (i.e. the Pulumi program doesn’t have to be written in the same language as the automation API code is). Historically, Node.js has had problems with global state for inline programs, but it was addressed late last year by using Node’s `AsyncLocalContext`: https://github.com/pulumi/pulumi/pull/10568 This is similar to how the .NET SDK uses 
AsyncLocal<T>
for the 
Deployment
instance: https://github.com/pulumi/pulumi-dotnet/blob/12adc4120be1c06d3b4510c60667e1984ecbdf5a/sdk/Pulumi/Deployment/Deployment.cs#L44 Does Java have something equivalent to .NET’s 
AsyncLocal<T>
that could be used to store the 
Deployment
instance? https://learn.microsoft.com/en-us/dotnet/api/system.threading.asynclocal-1?view=net-7.0
q
No, that was the problem 😞
the closest related concept is ThreadLocal and from what I recall Anton and Paweł tried to figure out whether they could leverage it but given that Output is designed on top of CompletableFuture that can and often does execute on top of built-in thread pool there was a constant danger that a user would spawn a CF without providing an instance of Executor that knows how to propagate context via ThreadLocal and it would break the whole thing
m
BTW, I don’t think we necessarily need to block on this issue. A Java Automation API, even without initial support for inline programs would still be really useful. For inline programs, I wonder if we could make it work by having the callback for the program take a 
Deployment
instance as an arg to the callback. Then rather than accessing the singleton, the program would use the passed-in instance. (I’m not sure if that’s even possible if there are things within the SDK that will be trying to use the singleton).
q
yeah, it's totally possible for non-inline programs
there was some initial draft work done by Paweł to find out what could be done to propagate Deployment in scope but thanks to ForkJoinPool.commonPool() being a global singleton too there's not much that can be done in terms of contextual async abstractions, Anton was reluctant to go the golang-way of just forcing user to pass Context instance (an interface allowing internals access to deployment) to all resource constructors (I agree it's ugly and not idiomatic) and the ThreadLocal avenue dried up due to the problem mentioned above
the biggest issue I can think of is that there's a direct reference to the singleton as the last call performed in the constructor of Resource superclass, namely: 
Deployment.getInstance().readOrRegisterResource(...)
In the future this is going to be solved by ScopedValue from Project Loom, which does what AsyncLocal<T> from dotnet does but for virtual threads / coroutines.
What could be done is to provide pulumi's own intermediate execution wrapper that is capable of handling asynchronicity instead of CompletableFuture and rewrite internals in terms of that construct. There is a precedent - pulumi-go implements it's own sort-of-Promise. That's also what we do in pulumi scala sdk (because we don't have one async construct in our ecosystem, we have several widely used sort-of-Promises).
b
yes, what @quaint-spring-93350 wrote is true, there were platform specific problems with thread executors and implementation specific problems inherited from C#, but we’ve had 2 promising solutions for the first and work was underway for the second
I’d be more than happy to finish the refactoring, but I’d have figure out how to find enough time to do it, the time commitment would be non-trivial
Open in Slack
PreviousNext
Powered by