https://pulumi.com logo
Powered by
Title
d

dry-salesmen-32588

03/25/2022, 8:15 AM
Hi guys, quick question. i have a multistep pulumi stack, where i initially create some aws kms keys and iam user / accesskey. in the 2nd. step i create some infrastructure where i need to inject this information as a string (since this is what the software recognizes). I'm unable to find a consistent method for extracting the initial resources information, as they are being created asynchronously is there a simple way in python - to await resources being created - before i fetch resource information to inject into the next phase of my stack creation ?
note, I've tried using the Output.all(<resource>).apply() method, to no avail
basically what i need in this case is simply.. importantresource= await aws.kms.Key(..) somerandomstring= importantresource.id + "stuff"
p

prehistoric-activity-61023

03/25/2022, 8:44 AM
Output.all
should do the trick
in case of single output, you can also use 
apply
somerandomstring = importantresource.id.apply(lambda _id: f"{_id}stuff")
or in case of such a simple concatenation, there’s a 
concat
function available: 
somerandomstring = pulumi.Output.concat(importantresource.id, "stuff")
d

dry-salesmen-32588

03/25/2022, 8:46 AM
tried all kinds of variants with that - and the string doesn't resolve fx: kmskeyid= Output.all(kmskey.id).apply(lambda l: f"troloo {l[0]}") or kmskeyid= Output.all(kmskey.id).apply() the string output when i print it after this supposedly "await/resolve" is fx: kms_key_id = "<pulumi.output.Output object at 0x7f7e87950040>"
p

prehistoric-activity-61023

03/25/2022, 8:47 AM
you cannot transform Output instance to pure string class
as you said yourself, it needs to be awaited
d

dry-salesmen-32588

03/25/2022, 8:47 AM
yes
p

prehistoric-activity-61023

03/25/2022, 8:47 AM
so the real question is: where do you want to use it later?
all pulumi resources should access 
pulumi.Input
and resolve this just fine
it’s possible that you’re tricking yourself by trying to print that value
d

dry-salesmen-32588

03/25/2022, 8:47 AM
i basically want to create it here-and-now, and then use the outputs immediatlly in my next lines of code.
p

prehistoric-activity-61023

03/25/2022, 8:48 AM
could you paste these “next lines of code”?
If you use that value to create some additional resources, I assume it should work just fine. I’m afraid you cannot simply print the output value in most cases (cause it might not available at that particular moment).
However, pulumi engine will treat 
pulumi.Output[str]
differently than 
str
under the hood so it should (implicitly) create a dependency graph between the resources.
d

dry-salesmen-32588

03/25/2022, 8:50 AM
sure, heres a little example: kmskey = kms.Key("example", deletion_window_in_days=10, description="decent description") kmskeyid= Output.all(kmskey.id).apply() someconfig= ''' kms_key_id = {{0}} ''' someconfig.format(kmskeyid)
where the last step is part of a more complicated configfile
but yeah, as you said - i simply cannot "print" the value as it might not be available. I'm completely okay with putting in a WAITJUSTAMINUTE call 🙂 - to suspend the routine until the resource has been created and the variable has been filled with proper data
i DO need to "print" the value in this case
p

prehistoric-activity-61023

03/25/2022, 8:54 AM
I am not aware of any trick to do “WAITJUSTAMINUTE call” 😐
d

dry-salesmen-32588

03/25/2022, 8:54 AM
otherwise i would have to create it as a seperate pulumi stack, and then load it back in with a stackreference.. which seems super complicated for what i need to do
p

prehistoric-activity-61023

03/25/2022, 8:54 AM
I’ve never needed it myself to be honest.
d

dry-salesmen-32588

03/25/2022, 8:54 AM
yeah its a special case - i know
p

prehistoric-activity-61023

03/25/2022, 8:55 AM
what kind of config is this 
someconfig
?
Is it like k8s configmap? Some other AWS/GCP resource?
d

dry-salesmen-32588

03/25/2022, 8:55 AM
yes a helm values config
p

prehistoric-activity-61023

03/25/2022, 8:56 AM
I’d like to get to the bottom of this so I can advice whether you can or can’t make it work.
d

dry-salesmen-32588

03/25/2022, 8:56 AM
and specifically im injecting this in a raw text helm values field
p

prehistoric-activity-61023

03/25/2022, 8:56 AM
helm values, I see
is this helm release created by pulumi as well?
d

dry-salesmen-32588

03/25/2022, 8:56 AM
yes
im attempting to do all of it in the same pulumi script 🙂
p

prehistoric-activity-61023

03/25/2022, 8:56 AM
cool
so I guess there’s still hope but I’d need to get more context
d

dry-salesmen-32588

03/25/2022, 8:57 AM
create aws resources create iam stuff create helm deployment
p

prehistoric-activity-61023

03/25/2022, 8:57 AM
that is: paste the snippet where you actually create a helm release and pass this value
I’m pretty sure if you stop trying to format that “manually” and always pass it either directly or manipulate it with 
apply/concat/all
, everything should be fine.
If you’re not able to do that, maybe there’s a bug in helm provider.
d

dry-salesmen-32588

03/25/2022, 8:58 AM
yeah i thought about this as well
and then i've been frantically trying to just resolve the promise/async call to the resource creation
because what i'm really asking for is just to get my resource result / output - in a synchronous fashion- so i can inject it in a helm values config later
p

prehistoric-activity-61023

03/25/2022, 9:00 AM
TL;DR; as far as I know: • you cannot make a pause to wait until all the currently defined outputs are resolved (pulumi is still trying to be declarative tool and that would go against that philosophy) • if you try to manually manipulate output with 
format
or even string concatenation -> you’re gonna harm yourself 😉
the thing is that you don’t have to synchronize manually this values at all
you’re supposed to pass the Output variable as input to helm
d

dry-salesmen-32588

03/25/2022, 9:01 AM
it gets even more interesting though
p

prehistoric-activity-61023

03/25/2022, 9:01 AM
it should be resolved automatically and additionally, thanks to that, it will create a dependency between the resources (so the helm is gonna wait for the kms cause it knows it’s used there)
d

dry-salesmen-32588

03/25/2022, 9:02 AM
the resource is actually evaluated properly in the first test i did
(its slightly garbage - so im just trying to figure out whats going on with pulumi and its string issues)
p

prehistoric-activity-61023

03/25/2022, 9:02 AM
I’m afraid that the whole values are declared as Input
d

dry-salesmen-32588

03/25/2022, 9:02 AM
but if you look at the image, the "readinessprobe" values area - DOES evaluate properly
p

prehistoric-activity-61023

03/25/2022, 9:03 AM
I don’t have much time now but what Id try is to create the whole values at once
let me draft you an idea what it would look like
d

dry-salesmen-32588

03/25/2022, 9:04 AM
i guess what youre suggesting is to do a concat around the whole values field ?
p

prehistoric-activity-61023

03/25/2022, 9:04 AM
values=Output.all([first_res.id, second_res.id]).apply(lambda res: {
  "server": ...
})
d

dry-salesmen-32588

03/25/2022, 9:04 AM
yeh
p

prehistoric-activity-61023

03/25/2022, 9:04 AM
I’m not sure it can work recursively in that manner
d

dry-salesmen-32588

03/25/2022, 9:04 AM
well if thats how it works.. 😕
ill give it a go
p

prehistoric-activity-61023

03/25/2022, 9:06 AM
I’ll take a look at my examples later and see whether it’s necessary or not
oh, I see sth
on the snippet you posted above, you said the first value works just fine, right?
d

dry-salesmen-32588

03/25/2022, 9:06 AM
yup
p

prehistoric-activity-61023

03/25/2022, 9:06 AM
what’s that 
vaulthcl
function?
d

dry-salesmen-32588

03/25/2022, 9:06 AM
a string
p

prehistoric-activity-61023

03/25/2022, 9:07 AM
I mean, it looks like it’s gonna operate on string value (and we already discussed that these values are not strings but awaitable outputs)
replace the line with 
vaulthcl.format
with proper 
Output.all
call
d

dry-salesmen-32588

03/25/2022, 9:07 AM
indeed, but thats also why i mashed the output.all into the format function 🙂
p

prehistoric-activity-61023

03/25/2022, 9:07 AM
that’s not enough
d

dry-salesmen-32588

03/25/2022, 9:08 AM
yeah i get that 😄 - its not working
p

prehistoric-activity-61023

03/25/2022, 9:08 AM
because the call to 
vaulthcl.format
got stringified version of the Output object and not the real value
rule of thumb: DO NOT ever, ever, ever manipulate Output[str] as normal strings
d

dry-salesmen-32588

03/25/2022, 9:08 AM
yeah, i've gathered as much
p

prehistoric-activity-61023

03/25/2022, 9:08 AM
you can either pass it directly without any modifications
or if you need to create an array based on that -> use 
apply
or 
all
(in case you need to rely on more than one output)
if you need to simply concatenate, you can also use 
concat
(but it’s totally fine to use more generic 
apply
and 
all
)
d

dry-salesmen-32588

03/25/2022, 9:09 AM
ill drop the "outer" string, and attempt to use the output.concat function
p

prehistoric-activity-61023

03/25/2022, 9:10 AM
can you paste the content of vaulthcl?
plus the wonderful line with three 
Output.all
from the screenshot
d

dry-salesmen-32588

03/25/2022, 9:10 AM
p

prehistoric-activity-61023

03/25/2022, 9:10 AM
I’ll try to alter that
anything that can be copy’n’pasted sir? 😄
d

dry-salesmen-32588

03/25/2022, 9:11 AM
so needy! 😄
p

prehistoric-activity-61023

03/25/2022, 9:12 AM
dude, I’m just trying to help - don’t expect me to copy the code from the screenshot 😜
d

dry-salesmen-32588

03/25/2022, 9:13 AM
this should work no ? Output.all(var1,var2,var3,var4).apply( lambda l: f'''ui = true listener "tcp" {{ address = "[::]:8200" cluster_address = "[::]:8201" }} storage "raft" {{ path = "/vault/data" }} service_registration "kubernetes" {{}} seal "awskms" {{ region = "{0}" kms_key_id = "{1}" access_key = "{2}" secret_key = "{3}" }} ''')
p

prehistoric-activity-61023

03/25/2022, 9:13 AM
let me see
yeah, that’s something we should aim for
but I guess you’ll have to replace 
{0}
with proper reference to 
l
variable from lambda arguments
d

dry-salesmen-32588

03/25/2022, 9:13 AM
yeh
alright ill give this a go
cheers man
p

prehistoric-activity-61023

03/25/2022, 9:15 AM
if that works, extract that to separate function like: 
def format_vault_hcl(kms_key_id: Output[str], ...) -> Output[str]:
   Output.all(kms_key_id, ...).apply(lambda args: ...)
so you won’t have such an ugly helm release definition 😉
d

dry-salesmen-32588

03/25/2022, 9:15 AM
yeah for sure itll need a iteration or two
p

prehistoric-activity-61023

03/25/2022, 9:16 AM
anyway, once you get it work, I’m sure you’ll be able to make it look cleaner
d

dry-salesmen-32588

03/25/2022, 9:16 AM
will try 🙂
p

prehistoric-activity-61023

03/25/2022, 9:16 AM
let me know if you manage to make it work 🙂
d

dry-salesmen-32588

03/25/2022, 9:16 AM
trying it out as we speak
it works 😄
thanks for clearing my headache
p

prehistoric-activity-61023

03/25/2022, 9:26 AM
🙌
d

dry-salesmen-32588

03/25/2022, 2:13 PM
@prehistoric-activity-61023 you still around ? - i managed to finalize the initial part, but now i'm facing a somewhat related problem. got 5mins ?
the whole helm values area works great though
so if you do get a chance to look at it, its again stringrelated.. but this time i'm attempting to apply a custom YAML via pulumi. and this yaml requires info from previously created resources. specifically i'm trying to create a yaml for our existing cert-manager, since you dont have these resources - applying the RAW yaml would do. but again, getting the simple awaited output from a resource is a pain, and most likely involves wrapping it in some convoluted way. I ended up with this : 
# Creating vault namespace
namespace=Namespace("vault", opts=ResourceOptions(provider=k8s_provider))

certmgryaml=Output.all(namespace.id).apply(lambda l: ConfigGroup("selfsigned_crtmgr", yaml='''apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1> \
kind: Issuer \
metadata: \
  name: vault-selfsigned \
  namespace: {l[0]} \
spec: \
  selfSigned: {{}} \
--- \
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1> \
kind: Certificate \
metadata: \
  name: selfsigned-cert \
  namespace: {namespace.id} \
spec: \
  commonName: vault \
  usages: \
    - server auth \
  dnsNames: \
    - vault \
    - vault.{l[0]} \
    - vault.{l[0]}.svc \
    - vault.{l[0]}.svc.cluster.local \
  ipAddresses: \
    - 127.0.0.1 \
  secretName: vault-selfsigned-cert-tls \
  issuerRef: \
    name: vault-selfsigned''', opts=ResourceOptions(provider=k8s_provider)))
again, my problems would've been solved if pulumi simply offered a simple way to await resources to finish so i can pull the information i need :(
just getting abit frustrated - i really want to like pulumi as it offers some great features
p

prehistoric-activity-61023

03/25/2022, 2:59 PM
well, I’d say that allowing you to explicitly await the resource breaks the implicit paradigm pulumi is trying to follow
IMO you shouldn’t mix imperative and declarative approach - usually it’s not a good mix 😉
anyway, back to your question - gimme me a minute to read it
personally, I’d “recreate” the k8s resource in pulumi as that way it’s more flexible
so instead of loading the YAML file, I’d create these custom resources using appropriate classes from the provider
anyway, your solution… should work? What kind of problems do you have now?
btw, aren’t you missing sth here? 
selfSigned: {{}}
and: 
namespace: {namespace.id}
should be replaced with: 
namespace: {l[0]}
try to apply what I just wrote and let me try to prepare another version
I’m gonna actually run my version to see whether it works as I expect it to.
If I were you, I’d create it like that: 
from pulumi_kubernetes.apiextensions import CustomResource
from pulumi_kubernetes.core.v1 import Namespace
from pulumi_kubernetes.meta.v1 import ObjectMetaArgs

namespace = Namespace(
    "vault",
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
    ),
)

issuer = CustomResource(
    "test-issuer",
    api_version="<http://cert-manager.io/v1|cert-manager.io/v1>",
    kind="Issuer",
    metadata=ObjectMetaArgs(
        name="vault-selfsigned",
        namespace=namespace.metadata.name,
    ),
    spec={
        "selfSigned": {}
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
    ),
)

certificate = CustomResource(
    "test-certificate",
    api_version="<http://cert-manager.io/v1|cert-manager.io/v1>",
    kind="Certificate",
    metadata=ObjectMetaArgs(
        name="selfsigned-cert",
        namespace=namespace.metadata.name,
    ),
    spec={
        "commonName": "vault",
        "usages": [
            "server auth",
        ],
        "dnsNames": [
            "vault",
            pulumi.Output.concat("vault.", namespace.metadata.name),
            pulumi.Output.concat("vault.", namespace.metadata.name, ".svc"),
            pulumi.Output.concat("vault.", namespace.metadata.name, ".svc.cluster.local"),
        ],
        "ipAddresses": [
            "127.0.0.1",
        ],
        "secretName": "vault-selfsigned-cert-tls",
        "issuerRef": {
            "name": "issuer",
        },
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=[issuer],
    ),
)
Summary: • because you create an python object, you can more easily use outputs from other resources • 
Issuer
is NOT gonna be created before the namespace because it uses the namespace object (
namespace.metadata.name
) - that creates an implicit dependency between these 2 resources • actually, I wasn’t able to repeat the trick with name in case of 
Certificate
and 
Issuer
and that’s why I used 
depends_on
and explicitly stated that they depend on each other (although, I guess it would be possible to create the simultaneously and it will eventually resolve itself)
Side note: if you create the namespace like this, it’s gonna be affected by pulumi autonaming scheme (so it will be named e.g. 
vault-0fumaikh
). If you want it to be named just 
vault
, pass the 
metadata.name
explicitly while creating it.
Now that I look at it, you could create the 
dnsNames
like this as well: 
namespace.metadata.name.apply(lambda name: [
  "vault",
  f"vault.{name}",
  f"vault.{name}.svc",
  f"vault.{name}.svc.cluster.local",
])
I hope all that is gonna help you or at least guide you in the right direction 🙂.
@dry-salesmen-32588
d

dry-salesmen-32588

03/28/2022, 10:52 AM
@prehistoric-activity-61023 thank you VERY much for the assistance, i hope to get time to work with your suggestions today or tomorrow. I didn't see that there was a CustomResource object to generate the config as a "pure" pulumi object, so i'm fine with that. i will give this a go 🙂
3 Views
#generalJoin Slack
Powered by