https://pulumi.com logo
Join Slack
Powered by
Hi guys, quick question. i have a multistep pulum...
# general
d
Hi guys, quick question. i have a multistep pulumi stack, where i initially create some aws kms keys and iam user / accesskey. in the 2nd. step i create some infrastructure where i need to inject this information as a string (since this is what the software recognizes). I'm unable to find a consistent method for extracting the initial resources information, as they are being created asynchronously is there a simple way in python - to await resources being created - before i fetch resource information to inject into the next phase of my stack creation ?
note, I've tried using the Output.all(<resource>).apply() method, to no avail
basically what i need in this case is simply.. importantresource= await aws.kms.Key(..) somerandomstring= importantresource.id + "stuff"
p
Output.all
should do the trick
in case of single output, you can also use 
apply
Copy code
somerandomstring = importantresource.id.apply(lambda _id: f"{_id}stuff")
or in case of such a simple concatenation, there’s a 
concat
function available:
Copy code
somerandomstring = pulumi.Output.concat(importantresource.id, "stuff")
d
tried all kinds of variants with that - and the string doesn't resolve fx: kmskeyid= Output.all(kmskey.id).apply(lambda l: f"troloo {l[0]}") or kmskeyid= Output.all(kmskey.id).apply() the string output when i print it after this supposedly "await/resolve" is fx: kms_key_id = "<pulumi.output.Output object at 0x7f7e87950040>"
p
you cannot transform Output instance to pure string class
as you said yourself, it needs to be awaited
d
yes
p
so the real question is: where do you want to use it later?
all pulumi resources should access 
pulumi.Input
and resolve this just fine
it’s possible that you’re tricking yourself by trying to print that value
d
i basically want to create it here-and-now, and then use the outputs immediatlly in my next lines of code.
p
could you paste these “next lines of code”?
If you use that value to create some additional resources, I assume it should work just fine. I’m afraid you cannot simply print the output value in most cases (cause it might not available at that particular moment).
However, pulumi engine will treat 
pulumi.Output[str]
differently than 
str
under the hood so it should (implicitly) create a dependency graph between the resources.
d
sure, heres a little example: kmskey = kms.Key("example", deletion_window_in_days=10, description="decent description") kmskeyid= Output.all(kmskey.id).apply() someconfig= ''' kms_key_id = {{0}} ''' someconfig.format(kmskeyid)
where the last step is part of a more complicated configfile
but yeah, as you said - i simply cannot "print" the value as it might not be available. I'm completely okay with putting in a WAITJUSTAMINUTE call 🙂 - to suspend the routine until the resource has been created and the variable has been filled with proper data
i DO need to "print" the value in this case
p
I am not aware of any trick to do “WAITJUSTAMINUTE call” 😐
d
otherwise i would have to create it as a seperate pulumi stack, and then load it back in with a stackreference.. which seems super complicated for what i need to do
p
I’ve never needed it myself to be honest.
d
yeah its a special case - i know
p
what kind of config is this 
someconfig
?
Is it like k8s configmap? Some other AWS/GCP resource?
d
yes a helm values config
p
I’d like to get to the bottom of this so I can advice whether you can or can’t make it work.
d
and specifically im injecting this in a raw text helm values field
p
helm values, I see
is this helm release created by pulumi as well?
d
yes
im attempting to do all of it in the same pulumi script 🙂
p
cool
so I guess there’s still hope but I’d need to get more context
d
create aws resources create iam stuff create helm deployment
p
that is: paste the snippet where you actually create a helm release and pass this value
I’m pretty sure if you stop trying to format that “manually” and always pass it either directly or manipulate it with 
apply/concat/all
, everything should be fine.
If you’re not able to do that, maybe there’s a bug in helm provider.
d
yeah i thought about this as well
and then i've been frantically trying to just resolve the promise/async call to the resource creation
because what i'm really asking for is just to get my resource result / output - in a synchronous fashion- so i can inject it in a helm values config later
p
TL;DR; as far as I know: • you cannot make a pause to wait until all the currently defined outputs are resolved (pulumi is still trying to be declarative tool and that would go against that philosophy) • if you try to manually manipulate output with 
format
or even string concatenation -> you’re gonna harm yourself 😉
the thing is that you don’t have to synchronize manually this values at all
you’re supposed to pass the Output variable as input to helm
d
it gets even more interesting though
p
it should be resolved automatically and additionally, thanks to that, it will create a dependency between the resources (so the helm is gonna wait for the kms cause it knows it’s used there)
d
the resource is actually evaluated properly in the first test i did
(its slightly garbage - so im just trying to figure out whats going on with pulumi and its string issues)
p
I’m afraid that the whole values are declared as Input
d
but if you look at the image, the "readinessprobe" values area - DOES evaluate properly
p
I don’t have much time now but what Id try is to create the whole values at once
let me draft you an idea what it would look like
d
i guess what youre suggesting is to do a concat around the whole values field ?
p
Copy code
values=Output.all([first_res.id, second_res.id]).apply(lambda res: {
  "server": ...
})
d
yeh
p
I’m not sure it can work recursively in that manner
d
well if thats how it works.. 😕
ill give it a go
p
I’ll take a look at my examples later and see whether it’s necessary or not
oh, I see sth
on the snippet you posted above, you said the first value works just fine, right?
d
yup
p
what’s that 
vaulthcl
function?
d
a string
p
I mean, it looks like it’s gonna operate on string value (and we already discussed that these values are not strings but awaitable outputs)
replace the line with 
vaulthcl.format
with proper 
Output.all
call
d
indeed, but thats also why i mashed the output.all into the format function 🙂
p
that’s not enough
d
yeah i get that 😄 - its not working
p
because the call to 
vaulthcl.format
got stringified version of the Output object and not the real value
rule of thumb: DO NOT ever, ever, ever manipulate Output[str] as normal strings
d
yeah, i've gathered as much
p
you can either pass it directly without any modifications
or if you need to create an array based on that -> use 
apply
or 
all
(in case you need to rely on more than one output)
if you need to simply concatenate, you can also use 
concat
(but it’s totally fine to use more generic 
apply
and 
all
)
d
ill drop the "outer" string, and attempt to use the output.concat function
p
can you paste the content of vaulthcl?
plus the wonderful line with three 
Output.all
from the screenshot
d
p
I’ll try to alter that
anything that can be copy’n’pasted sir? 😄
d
so needy! 😄
p
dude, I’m just trying to help - don’t expect me to copy the code from the screenshot 😜
d
this should work no ? Output.all(var1,var2,var3,var4).apply( lambda l: f'''ui = true listener "tcp" {{ address = "[:]8200" cluster_address = "[:]8201" }} storage "raft" {{ path = "/vault/data" }} service_registration "kubernetes" {{}} seal "awskms" {{ region = "{0}" kms_key_id = "{1}" access_key = "{2}" secret_key = "{3}" }} ''')
p
let me see
yeah, that’s something we should aim for
but I guess you’ll have to replace 
{0}
with proper reference to 
l
variable from lambda arguments
d
yeh
alright ill give this a go
cheers man
p
if that works, extract that to separate function like:
Copy code
def format_vault_hcl(kms_key_id: Output[str], ...) -> Output[str]:
   Output.all(kms_key_id, ...).apply(lambda args: ...)
so you won’t have such an ugly helm release definition 😉
d
yeah for sure itll need a iteration or two
p
anyway, once you get it work, I’m sure you’ll be able to make it look cleaner
d
will try 🙂
p
let me know if you manage to make it work 🙂
d
trying it out as we speak
it works 😄
thanks for clearing my headache
p
🙌
d
@prehistoric-activity-61023 you still around ? - i managed to finalize the initial part, but now i'm facing a somewhat related problem. got 5mins ?
the whole helm values area works great though
so if you do get a chance to look at it, its again stringrelated.. but this time i'm attempting to apply a custom YAML via pulumi. and this yaml requires info from previously created resources. specifically i'm trying to create a yaml for our existing cert-manager, since you dont have these resources - applying the RAW yaml would do. but again, getting the simple awaited output from a resource is a pain, and most likely involves wrapping it in some convoluted way. I ended up with this :
Copy code
# Creating vault namespace
namespace=Namespace("vault", opts=ResourceOptions(provider=k8s_provider))

certmgryaml=Output.all(namespace.id).apply(lambda l: ConfigGroup("selfsigned_crtmgr", yaml='''apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1> \
kind: Issuer \
metadata: \
  name: vault-selfsigned \
  namespace: {l[0]} \
spec: \
  selfSigned: {{}} \
--- \
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1> \
kind: Certificate \
metadata: \
  name: selfsigned-cert \
  namespace: {namespace.id} \
spec: \
  commonName: vault \
  usages: \
    - server auth \
  dnsNames: \
    - vault \
    - vault.{l[0]} \
    - vault.{l[0]}.svc \
    - vault.{l[0]}.svc.cluster.local \
  ipAddresses: \
    - 127.0.0.1 \
  secretName: vault-selfsigned-cert-tls \
  issuerRef: \
    name: vault-selfsigned''', opts=ResourceOptions(provider=k8s_provider)))
again, my problems would've been solved if pulumi simply offered a simple way to await resources to finish so i can pull the information i need :(
just getting abit frustrated - i really want to like pulumi as it offers some great features
p
well, I’d say that allowing you to explicitly await the resource breaks the implicit paradigm pulumi is trying to follow
IMO you shouldn’t mix imperative and declarative approach - usually it’s not a good mix 😉
anyway, back to your question - gimme me a minute to read it
personally, I’d “recreate” the k8s resource in pulumi as that way it’s more flexible
so instead of loading the YAML file, I’d create these custom resources using appropriate classes from the provider
anyway, your solution… should work? What kind of problems do you have now?
btw, aren’t you missing sth here?
Copy code
selfSigned: {{}}
and:
Copy code
namespace: {namespace.id}
should be replaced with:
Copy code
namespace: {l[0]}
try to apply what I just wrote and let me try to prepare another version
I’m gonna actually run my version to see whether it works as I expect it to.
If I were you, I’d create it like that:
Copy code
from pulumi_kubernetes.apiextensions import CustomResource
from pulumi_kubernetes.core.v1 import Namespace
from pulumi_kubernetes.meta.v1 import ObjectMetaArgs

namespace = Namespace(
    "vault",
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
    ),
)

issuer = CustomResource(
    "test-issuer",
    api_version="<http://cert-manager.io/v1|cert-manager.io/v1>",
    kind="Issuer",
    metadata=ObjectMetaArgs(
        name="vault-selfsigned",
        namespace=namespace.metadata.name,
    ),
    spec={
        "selfSigned": {}
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
    ),
)

certificate = CustomResource(
    "test-certificate",
    api_version="<http://cert-manager.io/v1|cert-manager.io/v1>",
    kind="Certificate",
    metadata=ObjectMetaArgs(
        name="selfsigned-cert",
        namespace=namespace.metadata.name,
    ),
    spec={
        "commonName": "vault",
        "usages": [
            "server auth",
        ],
        "dnsNames": [
            "vault",
            pulumi.Output.concat("vault.", namespace.metadata.name),
            pulumi.Output.concat("vault.", namespace.metadata.name, ".svc"),
            pulumi.Output.concat("vault.", namespace.metadata.name, ".svc.cluster.local"),
        ],
        "ipAddresses": [
            "127.0.0.1",
        ],
        "secretName": "vault-selfsigned-cert-tls",
        "issuerRef": {
            "name": "issuer",
        },
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=[issuer],
    ),
)
Summary: • because you create an python object, you can more easily use outputs from other resources • 
Issuer
is NOT gonna be created before the namespace because it uses the namespace object (
namespace.metadata.name
) - that creates an implicit dependency between these 2 resources • actually, I wasn’t able to repeat the trick with name in case of 
Certificate
and 
Issuer
and that’s why I used 
depends_on
and explicitly stated that they depend on each other (although, I guess it would be possible to create the simultaneously and it will eventually resolve itself)
Side note: if you create the namespace like this, it’s gonna be affected by pulumi autonaming scheme (so it will be named e.g. 
vault-0fumaikh
). If you want it to be named just 
vault
, pass the 
metadata.name
explicitly while creating it.
Now that I look at it, you could create the 
dnsNames
like this as well:
Copy code
namespace.metadata.name.apply(lambda name: [
  "vault",
  f"vault.{name}",
  f"vault.{name}.svc",
  f"vault.{name}.svc.cluster.local",
])
I hope all that is gonna help you or at least guide you in the right direction 🙂.
@dry-salesmen-32588
d
@prehistoric-activity-61023 thank you VERY much for the assistance, i hope to get time to work with your suggestions today or tomorrow. I didn't see that there was a CustomResource object to generate the config as a "pure" pulumi object, so i'm fine with that. i will give this a go 🙂
3 Views
Open in Slack
PreviousNext
Powered by