https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
a
I can avoid this problem by manually adding a user profile to the aws_auth config file. But I am not sure of how to do this automatically via Pulumi? Or if there is a better solution?
w
You have to wait for the api server to be ready, then create aws-auth config map before creating the node group; otherwise eks will create the aws-auth config map first and then you have to modify it, which is harder since pulumi won't own it.
I've done this with dotnet/c#. What language are you using?
a
Hi @worried-city-86458, I'm using Python primarily and, at times, typescript. But if you wouldn't mind sharing your approach with dotnet/c#, maybe I can adapt it?
w
Okay, so after creating the cluster you need the kubeconfig, and you can use that as the sync point:
Copy code
var cluster = new Cluster($"{prefix}-cluster",
    new ClusterArgs
    {
        //EnabledClusterLogTypes = { "api", "audit", "authenticator", "controllerManager", "scheduler" },
        RoleArn = clusterRole.Arn,
        Tags = config.EnvTags,
        Version = config.KubeVersion,
        VpcConfig = new ClusterVpcConfigArgs { SubnetIds = subnetIds }
    },
    new CustomResourceOptions { DependsOn = clusterPolicies });

ClusterName = cluster.Name;
KubeConfig = cluster.GetKubeConfig();
var kubeProvider = new Provider($"{prefix}-kube", new ProviderArgs { KubeConfig = KubeConfig });
GetKubeConfig
is an extension method that waits for the api server via 
Output.Format
(see 
WaitForEndpoint
):
Copy code
public static class ClusterExtensions
    {
        public static Output<string> GetKubeConfig(this Cluster cluster) =>
            Output.Format(@$"{{
    ""apiVersion"": ""v1"",
    ""clusters"": [{{
        ""cluster"": {{
            ""server"": ""{cluster.WaitForEndpoint(TimeSpan.FromMinutes(5))}"",
            ""certificate-authority-data"": ""{cluster.CertificateAuthority.Apply(ca => ca.Data)}""
        }},
        ""name"": ""kubernetes"",
    }}],
    ""contexts"": [{{
        ""context"": {{
            ""cluster"": ""kubernetes"",
            ""user"": ""aws"",
        }},
        ""name"": ""aws"",
    }}],
    ""current-context"": ""aws"",
    ""kind"": ""Config"",
    ""users"": [{{
        ""name"": ""aws"",
        ""user"": {{
            ""exec"": {{
                ""apiVersion"": ""<http://client.authentication.k8s.io/v1alpha1|client.authentication.k8s.io/v1alpha1>"",
                ""command"": ""aws-iam-authenticator"",
                ""args"": [
                    ""token"",
                    ""-i"",
                    ""{cluster.Name}"",
                ],
            }},
        }},
    }}],
}}").Apply(Output.CreateSecret);

        private static Output<string> WaitForEndpoint(this Cluster cluster, TimeSpan timeout) =>
            cluster.Endpoint.Apply(async endpoint =>
            {
                if (!Deployment.Instance.IsDryRun)
                {
                    await new ApiServer(endpoint).WaitForHealthzAsync(timeout);
                }
                return endpoint;
            });
    }
With 
ApiServer
as follows:
Copy code
public class ApiServer
{
    public ApiServer(string baseUrl)
    {
        var handler = new HttpClientHandler { ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator };
        Client = new FlurlClient(new HttpClient(handler)) { BaseUrl = baseUrl };

        HealthzPolicy = Policy.WrapAsync(
            Policy.TimeoutAsync<HttpStatusCode>(context => (TimeSpan)context["Timeout"]),
            Policy.HandleResult<HttpStatusCode>(status => status != HttpStatusCode.OK)
                .WaitAndRetryForeverAsync(
                    count => TimeSpan.FromSeconds(5),
                    (status, count, delay) => <http://Log.Info|Log.Info>($"Waiting for api server... ({count})", ephemeral: true)));
    }

    public async Task<HttpStatusCode> GetHealthzAsync()
    {
        try
        {
            return (await Client.Request("healthz").AllowAnyHttpStatus().GetAsync()).ResponseMessage.StatusCode;
        }
        catch (Exception ex)
        {
            Log.Debug($"ApiServer: {ex.GetBaseException().Message}");
            return HttpStatusCode.ServiceUnavailable;
        }
    }

    public Task<HttpStatusCode> WaitForHealthzAsync(TimeSpan timeout)
        => HealthzPolicy.ExecuteAsync(context => GetHealthzAsync(), new Dictionary<string, object> { ["Timeout"] = timeout });

    protected IFlurlClient Client { get; }

    private AsyncPolicyWrap<HttpStatusCode> HealthzPolicy { get; }
}
... using a bunch of C# libs to auto-retry every 5s for 5m
You then use the kubeconfig via the kubeprovider to create the aws-auth config map:
Copy code
var authConfigMap = new ConfigMap($"{prefix}-auth",
    new ConfigMapArgs
    {
        Metadata = new ObjectMetaArgs
        {
            Namespace = "kube-system",
            Name = "aws-auth"
        },
        Data =
        {
            ["mapRoles"] = IamHelpers.GetRoleMappings(nodeRole, nodeRoleWin, config.AwsAccountId),
            ["mapUsers"] = IamHelpers.GetUserMappings()
        }
    },
    new CustomResourceOptions { Provider = kubeProvider });
a
Thanks a lot @worried-city-86458, this is definitely helpful 👍 One part that is uncertain to me is what IamHelpers.GetUserMappings and IamHelpers.GetRoleMappings are returning. Are these Pulumi functions? (I've tried searching on the documentation, but could not find anything) I guess they are returning an iterable of strings which, in the case of GetUserMapping, look something like this?
Copy code
userarn: arn:aws:iam::111122223333:user/user.bob
      username: user.bob
      groups:
        - system:masters
Also, in the version of this which is generated by AWS, I see metadata entries for "creationTimestamp", "resourceVersion", "selfLink", and "uid". Are these automatically added in your approach? Or are they not necessary?
w
Yeah, simply returning the minimal properties like:
Copy code
public static Output<string> GetRoleMappings(Role nodeRole, Role nodeRoleWin, string accountId) =>
    Output.Tuple<string, string, string>(nodeRole.Arn, nodeRoleWin.Arn, accountId)
        .Apply(((string NodeRoleArn, string NodeRoleWinArn, string AccountId) tuple) => new[]
        {
            new
            {
                rolearn = tuple.NodeRoleArn,
                username = "system:node:{{EC2PrivateDNSName}}",
                groups = new[]
                {
                    "system:bootstrappers",
                    "system:nodes"
                }
            },
            new
            {
                rolearn = tuple.NodeRoleWinArn,
                username = "system:node:{{EC2PrivateDNSName}}",
                groups = new[]
                {
                    "system:bootstrappers",
                    "system:nodes",
                    "eks:kube-proxy-windows"
                }
            }
        }.ToYaml());
GetUserMappings
is similar but specifies 
userarn
instead of 
rolearn
a
Hello again @worried-city-86458 and everyone else who may be interested in this. Based on our discussion above, the below snippet of code is what I've found works for me. Of course it can be expanded on and further generalized later on. As @worried-city-86458 suggested, this has to be included after creating the EKS cluster, but before creating the NodeGroup
Copy code
######################################################################
## Set-up kubernetes-auth

MAP_ROLES = iam.ec2_role.arn.apply(
    lambda arn: "\n".join(
        [
            f"- groups:",
            f"    - system:bootstrappers",
            f"    - system:nodes",
            f"  rolearn: {arn}",
            f"  username: system:node:{{{{EC2PrivateDNSName}}}}",
        ]
    )
)

MAP_USERS = pulumi.Output.all().apply(
    lambda args: "\n".join(
        [
            "- userarn: arn:aws:iam::111122223333:user/bob",
            "  username: bob",
            "  groups:",
            "    - system:masters",
        ]
    ),
)

authConfigMap = pulumi_kubernetes.core.v1.ConfigMap(
    f"{cluster_name}-auth",
    metadata={"namespace": "kube-system", "name": "aws-auth"},
    data={
        "mapRoles": MAP_ROLES,
        "mapUsers": MAP_USERS,
    },
    opts=pulumi.ResourceOptions(provider=k8s_provider),
)
Open in Slack
PreviousNext
Powered by