What pattern are folks using for duplicating and amending their default provider configuration? I need to declare a second provider with same settings as default, but with a different region. Deploying locally is different than in CI.

let awsProviderDefaults;
if (!process.env.AWS_ACCESS_KEY_ID) {
  aws.sdk.config.credentials = new aws.sdk.SharedIniFileCredentials({ profile: awsConfig.get("profile") });
  awsProviderDefaults = { profile: awsConfig.get("profile") };
} else {
  awsProviderDefaults = {
    accessKey: process.env.AWS_ACCESS_KEY_ID,
    secretKey: process.env.AWS_SECRET_ACCESS_KEY,
    token: aws.sdk.config.sessionToken,
  };
}

const awsUsEast1 = new aws.Provider("east", {
  region: "us-east-1",
  ...awsProviderDefaults,
});

Hi, I’m trying to set up a MWAA environment and I’m stuck 😞 Initially I was getting this error:

aws:mwaa:Environment (dev-aqua-airflow):
    error: 1 error occurred:
        * error creating MWAA Environment: ValidationException: Failed to assume role arn:aws:iam::<account_id>:role/dev-airflow-execution-role. This could be due to the role's trust policy. Please ensure your role is assumable by '<http://airflow-env.amazonaws.com|airflow-env.amazonaws.com>' Service Principal and try again.

And this is what my execution role looks like:

mwaa_execution_role = aws.iam.Role(f"{stack}-airflow-execution-role",
                                   name=f"{stack}-airflow-execution-role",
                                   assume_role_policy=json.dumps({
                                       "Version": "2012-10-17",
                                       "Statement": [
                                           {
                                               "Action": "sts:AssumeRole",
                                               "Principal": {
                                                   "Service": [
                                                       "<http://airflow.amazonaws.com|airflow.amazonaws.com>",
                                                       "<http://airflow-env.amazonaws.com|airflow-env.amazonaws.com>"
                                                   ],
                                               },
                                               "Effect": "Allow",
                                           },
                                       ]
                                   }))

I then decided to create the S3 bucket, role and policies before creating the MWAA Environment and this is the error I’m getting now:

aws:mwaa:Environment (dev-aqua-airflow):
    error: 1 error occurred:
        * creating urn:pulumi:dev::data-ml-airflow::aws:mwaa/environment:Environment::dev-aqua-airflow: 1 error occurred:
        * error waiting for MWAA Environment (dev-aqua-airflow-dd6bc3e) creation: unexpected state 'CREATE_FAILED', wanted target 'AVAILABLE'. last error: %!s(<nil>)

When I go to the AWS Console, this is the error I see:

Error code
INCORRECT_CONFIGURATION
Message
You may need to check the execution role permissions policy for your environment, and that each of the VPC networking components required by the environment are configured to allow traffic. Troubleshooting: <https://docs.aws.amazon.com/mwaa/latest/userguide/troubleshooting.html>

And this is my Pulumi MWAA code:

airflow_env = aws.mwaa.Environment(f"{stack}-aqua-airflow",
                                   dag_s3_path="dags/",
                                   execution_role_arn=mwaa_execution_role.arn,
                                   airflow_version='2.2.2',
                                   kms_key=mwaa_kms_key.arn,
                                   logging_configuration=aws.mwaa.EnvironmentLoggingConfigurationArgs(
                                       dag_processing_logs=aws.mwaa.EnvironmentLoggingConfigurationDagProcessingLogsArgs(
                                           enabled=True,
                                           log_level="DEBUG",
                                       ),
                                       scheduler_logs=aws.mwaa.EnvironmentLoggingConfigurationSchedulerLogsArgs(
                                           enabled=True,
                                           log_level="INFO",
                                       ),
                                       task_logs=aws.mwaa.EnvironmentLoggingConfigurationTaskLogsArgs(
                                           enabled=True,
                                           log_level="WARNING",
                                       ),
                                       webserver_logs=aws.mwaa.EnvironmentLoggingConfigurationWebserverLogsArgs(
                                           enabled=True,
                                           log_level="ERROR",
                                       ),
                                       worker_logs=aws.mwaa.EnvironmentLoggingConfigurationWorkerLogsArgs(
                                           enabled=True,
                                           log_level="CRITICAL",
                                       ),
                                   ),
                                   network_configuration=aws.mwaa.EnvironmentNetworkConfigurationArgs(
                                       security_group_ids=[vpc["vpcDefaultSecurityGroupID"]],
                                       subnet_ids=[vpc["privateSubnetsIDs"][0], vpc["privateSubnetsIDs"][1]],
                                   ),
                                   source_bucket_arn=airflow_dags_bucket.arn,
                                   tags={
                                       "Environment": f"{stack}",
                                   },
                                   opts=ResourceOptions(
                                       depends_on=[mwaa_execution_role, mwaa_kms_key, airflow_dags_bucket]))

Any idea what I’m doing incorrectly? I’m trying to look into this:

That your Amazon VPC is configured to allow network traffic between the different AWS resources used by your Amazon MWAA environment, as defined in About networking on Amazon MWAA. For example, your VPC security group must either allow all traffic in a self-referencing rule, or optionally specify the port range for HTTPS port range 443 and a TCP port range 5432.

I set up a Pre-Aggregation on Source Database Snowflake. However getting an error stating ‘Insufficient privileges to operate on database’

Performing query: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1 
Error while querying: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1 (3702ms)
{
  "processingId": 1,
  "queueSize": 1,
  "queryKey": [
    [
      "CREATE TABLE arch_council_app.cube_aws_billing_cost_by_account AS SELECT\n      \"cube_aws_billing\".\"ACCOUNT_ALIAS\" \"cube_aws_billing__account_alias\", date_trunc('MONTH', CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) \"cube_aws_billing__bill_date_month\", sum(\"cube_aws_billing\".\"SERVICE_COST\") \"cube_aws_billing__service_cost\"\n    FROM\n      \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"  GROUP BY 1, 2",
      []
    ],
    [
      [
        {
          "refresh_key": "463123"
        }
      ]
    ]
  ],
  "queuePrefix": "SQL_PRE_AGGREGATIONS_STANDALONE_default",
  "timeInQueue": 1,
  "preAggregationId": "cube_aws_billing.cost_by_account",
  "newVersionEntry": {
    "table_name": "arch_council_app.cube_aws_billing_cost_by_account",
    "structure_version": "ulrf25hc",
    "content_version": "bxxvnrki",
    "last_updated_at": 1667246128534,
    "naming_version": 2
  },
  "preAggregation": {
    "preAggregationId": "cube_aws_billing.cost_by_account",
    "timezone": "UTC",
    "timestampFormat": "YYYY-MM-DD[T]HH:mm:ss.SSS[Z]",
    "tableName": "arch_council_app.cube_aws_billing_cost_by_account",
    "invalidateKeyQueries": [
      [
        "SELECT FLOOR((UNIX_TIMESTAMP()) / 3600) as refresh_key",
        [],
        {
          "external": true,
          "renewalThreshold": 120
        }
      ]
    ],
    "type": "rollup",
    "external": true,
    "previewSql": [
      "SELECT * FROM arch_council_app.cube_aws_billing_cost_by_account LIMIT 1000",
      []
    ],
    "preAggregationsSchema": "arch_council_app",
    "loadSql": [
      "CREATE TABLE arch_council_app.cube_aws_billing_cost_by_account AS SELECT\n      \"cube_aws_billing\".\"ACCOUNT_ALIAS\" \"cube_aws_billing__account_alias\", date_trunc('MONTH', CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) \"cube_aws_billing__bill_date_month\", sum(\"cube_aws_billing\".\"SERVICE_COST\") \"cube_aws_billing__service_cost\"\n    FROM\n      \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"  GROUP BY 1, 2",
      []
    ],
    "sql": [
      "SELECT\n      \"cube_aws_billing\".\"ACCOUNT_ALIAS\" \"cube_aws_billing__account_alias\", date_trunc('MONTH', CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) \"cube_aws_billing__bill_date_month\", sum(\"cube_aws_billing\".\"SERVICE_COST\") \"cube_aws_billing__service_cost\"\n    FROM\n      \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"  GROUP BY 1, 2",
      []
    ],
    "uniqueKeyColumns": [
      "\"cube_aws_billing__account_alias\"",
      "\"cube_aws_billing__bill_date_month\""
    ],
    "aggregationsColumns": [
      "sum(\"cube_aws_billing__service_cost\")"
    ],
    "dataSource": "default",
    "granularity": "month",
    "preAggregationStartEndQueries": [
      [
        "select min(CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) from \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"",
        []
      ],
      [
        "select max(CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) from \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"",
        []
      ]
    ],
    "indexesSql": [],
    "createTableIndexes": [],
    "readOnly": false
  },
  "addedToQueueTime": 1667246128534
} 
OperationFailedError: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
    at createError (/cube/node_modules/snowflake-sdk/lib/errors.js:536:15)
    at Object.exports.createOperationFailedError (/cube/node_modules/snowflake-sdk/lib/errors.js:315:10)
    at Object.callback (/cube/node_modules/snowflake-sdk/lib/services/sf.js:647:28)
    at /cube/node_modules/snowflake-sdk/lib/http/base.js:111:25
    at done (/cube/node_modules/urllib/lib/urllib.js:589:5)
    at /cube/node_modules/urllib/lib/urllib.js:953:9
    at decodeContent (/cube/node_modules/urllib/lib/urllib.js:740:14)
    at handleResponseCloseAndEnd (/cube/node_modules/urllib/lib/urllib.js:924:7)
    at IncomingMessage.<anonymous> (/cube/node_modules/urllib/lib/urllib.js:962:7)
    at IncomingMessage.emit (events.js:412:35)
    at IncomingMessage.emit (domain.js:475:12)
    at endReadableNT (internal/streams/readable.js:1333:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
Error querying db: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1 
--
"SELECT `cube_aws_billing__account_alias` `cube_aws_billing__account_alias`, sum(`cube_aws_billing__service_cost`) `cube_aws_billing__service_cost` FROM arch_council_app.cube_aws_billing_cost_by_account AS `cube_aws_billing__cost_by_account` GROUP BY 1 ORDER BY 2 DESC LIMIT 50"
--
{
  "params": []
} 
Error: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
    at QueryQueue.parseResult (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:146:13)
    at QueryQueue.executeInQueue (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:135:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at PreAggregationLoader.loadPreAggregationWithKeys (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:742:7)
    at preAggregationPromise (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:1946:28)
    at QueryOrchestrator.fetchQuery (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:158:59)
    at OrchestratorApi.executeQuery (/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:85:20)
    at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1230:21
    at async Promise.all (index 0)
    at ApiGateway.getSqlResponseInternal (/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1228:31)
    at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1357:28
    at async Promise.all (index 0)
    at ApiGateway.load (/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1348:23)
    at /cube/node_modules/@cubejs-backend/api-gateway/src/sql-server.ts:101:13
Orchestrator error: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1 (3832ms)
--
{
  "measures": [
    "cube_aws_billing.service_cost"
  ],
  "dimensions": [
    "cube_aws_billing.account_alias"
  ],
  "segments": [],
  "order": [
    [
      "cube_aws_billing.service_cost",
      "desc"
    ]
  ],
  "limit": 50
}
--
{
  "securityContext": {},
  "appName": "NULL",
  "protocol": "postgres",
  "apiType": "sql"
} 
Error: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
2022-10-31 19:55:32,267 ERROR [cubejs_native::transport] [transport] load - strange response, success which contains error: V1Error { error: "Error: SQL access control error:\nInsufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'" }
Cube SQL Error: undefined 
{
  "apiType": "sql",
  "protocol": "postgres",
  "appName": "NULL"
} 
Error during processing PostgreSQL message: Internal: Execution error: Internal: Error: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'

However I am able to execute the create table as query with the

CUBEJS_DB_USER=gbi_others_data_eng_db_arch_council_user

CREATE TABLE arch_council_app.cube_aws_billing_cost_by_account AS SELECT 
cube_aws_billing.ACCOUNT_ALIAS cube_aws_billing__account_alias, 
date_trunc('MONTH', CONVERT_TIMEZONE('UTC', cube_aws_billing.BILL_DATE::timestamp_tz)::timestamp_ntz) cube_aws_billing__bill_date_month, 
sum(cube_aws_billing.SERVICE_COST) cube_aws_billing__service_cost   
FROM ARCH_COUNCIL_APP.AWS_BILLING AS cube_aws_billing  
GROUP BY 1, 2

What am I missing here?

Hi, https://raw.githubusercontent.com/pulumi/docs/master/data/versions.json does not contain

v3.45.0

yet, so I can't install it using https://github.com/pulumi/setup-pulumi, is this WAI?

Hi, everyone, I would like to know what is the difference between the following two packages and their positioning? I see a big difference in their interfaces, some options are in 1 instead of 2

from pulumi_aws.apigateway import RestApi
from pulumi_aws_apigateway import RestAPI

Hi folks, running into an issue with pulumi-cdk / aws-native provider. It's the same issue here: https://github.com/pulumi/pulumi-aws-native/issues/610

Re: the new Web Gui; I used to see some helpful CLI commands for manipulating the stack, like what to write to delete everything, create a new .yaml file etc. Any ideas as where to find this info now?

Is

tf2pulumi

still supported? It looks like the web version's stopped working, and the repo hasn't been updated in ~18 months as far as I can see

Unit tests: How can I mock

pulumi.Output<string>

as a variable ? I’m using mocha.js & chai Error:

(property) TargetGroupArgs.vpcId: pulumi.Output<string>

Type 'string' is not assignable to type 'Output<string>'.

Helm error: incompatible version -- but it doesn't read out the right version: https://pulumi-community.slack.com/archives/CRFURDVQB/p1667384205784629

Hello, I'm getting spun up on Pulumi and having an error changing the secret provider that my co-worker is not seeing. Co-worker is on windows and using WSL while I'm on a Mac. I'm seeing the following:

pulumi stack change-secrets-provider <azurekeyvault://cg-test-kv.vault.azure.net/keys/dev-stacks>
error: open keeper <azurekeyvault://cg-test-kv.vault.azure.net/keys/dev-stacks>: failed to Dial default KeyVault: MSI not available

increasing verbosity does not add any additional output. Couldn't tell what area this is exactly related to. If I need to post in different channel, let me know. Suggestions?

https://www.pulumi.com/docs/reference/cli/pulumi_stack_graph/ I see there's a

pulumi stack graph

command that claims to "export a stack’s dependency graph to a file". How do you define a stack's dependencies? How are these dependencies used?

Hi team 👋 , When the YAML helm chart docs will be completed ?

I am using kube2pulumi to convert .yaml file to pulumi python code. but when i deploy it. configmaps which contains | with some data which is not saving. how to do that

I have multiple projects running with references, when making changes to my deployment I don’t necessarily change all the projects I’m trying to find ways to speed up the deployment (that takes over 10 minutes today) by eliminating updates on projects that haven’t changed I was thinking about running

preview --expect-no-changes

and if that command returns non-zero run

update

but I was hoping there’s a nice way of doing it instead of this ugly hack has anyone encountered something similar?

Hey, I know this is a really simple question… I’m just looking for a guide or something for setting up environment variables so that I can deploy to AWS using

pulumi up

from my laptop, and have the service running with the proper configurations. (?)

We can register additional resource outputs with

register_output

as explained here: https://www.pulumi.com/docs/intro/concepts/resources/components/#registering-component-outputs How can we hide outputs? I want to make it so that my custom components only expose a small subset of their outputs to ensure we don't accidentally develop a dependency on something unstable.

Hi! Project Level Configuration - for the win! But I must have missed something - neither in the blob post nor in the documentation do I see how, using pulumi cli, to set a secret value in the project. I understand/see that the spec can be defined for a type such that it is a secret - but now how to supply the value in the project file. My use case is to set the devops:pat + devops:org url for the Azure DevOps provider. Any hints appreciated!

Hi everybody! I am new here and haven't found any "Intro" channel, so I am saying "hi" here. I have a question: Is it possible to create infrastructure in Pulumi (single stack) using multiple programming languages? I know that there are multi-language Components but my use case is a little bit different: My main code is written in Golang and I wanted to allow other developers to write parts of the infrastructure in a different language (for example TypeScript). If not possible now, do you have plans to support multi-language stacks?

this is my pulumi code. how to order creation of the Resouces. As this i am using it for Istio installation using python https://raw.githubusercontent.com/naguait85/pulumi/main/main.py some input appreciated as i stuck from long time

Hey there, I just upgraded the pulumi cli and hit

panic: interface conversion: interface {} is string, not int

not sure if this is a bug in the new version or related to me trying to import some resources from hetzner, but wanted to flag it here in case it's the former. Happy to raise an issue

Trying to use Automation API to disable the default providers, but the

set_config()

method doesn't support using a path key?

stack.set_config("pulumi:disable-default-providers[0]", auto.ConfigValue("*"))

Results in a literal config with [0] in its name.. but if I use --path on the cli I can force it to a list of string... is there a more correct way to do this?? setting complex types via automation?

Hey everyone I'm Ganesh Patil. 👋 Professionally, I work as a community manager for open source through Aviyel. I really like the idea behind Pulumi and am looking forward to spread the word through some non-code ways (like an event or blog). I was wondering if it would be fine with the team?

Hi everyone! I'm trying to create a

DashboardJson

from an exported JSON from Datadog itself, but I'm having trouble with the definition of

query_values

. 🧵

Serious question. How on earth do I take an output and make it into a string? I have lost an entire day to this and a team of 5 people and none of us can make this work. I create a redis cache on gcp which I return as an object called cache which I would expect to have a cache.host value, which I think I do have. Only that the output type of that is string according to the documentation. When I try to use it as a string to create a dns entry I get told it is an Output and not a string. So I want to cast to a string? I have this

host = cache.host.apply(lambda k: k)

Which from what I can tell is meant to be right, but host remains an output.

👋 Hi everyone! I am using pulumi-aws, a new guru, I found it is hard to find route table via ec2.GetRouteTable. any ideas, how to get the table and add a route to it?

Hi Pulumi team, where I can check the recorded version of this workshop -> https://www.bigmarker.com/pulumi/Sharing-Code-with-Pulumi-Packages-Abstractions-and-More?utm_source=twitter&utm_medium=social&utm_campaign=workshops

Since this message is getting zero notice on #gcp and I think it is mostly due to that channel having relatively few people on it I'm sharing this here in hope that someone might actually know about this issue

I’d like some guidance on Azure Classic vs Azure Native - am I recalling right when I say classic is essentially “end of life”?

Hello all, I ran into an issue with the

deleteBeforeReplace: true

property in

CustomResourceOptions

. It seems like it's not getting applied to the resource. When I run

pulumi up

. Pulumi still tries to do a create before delete on a replacement. I put in the issue below but does anyone see an obvious error in my code?

import * as aws from '@pulumi/aws';

const cluster = new aws.ecs.Cluster('playground', {name: 'playground'});
new aws.ecs.ClusterCapacityProviders(
  'playground',
  {
    clusterName: 'playground',
    capacityProviders: ['FARGATE'],
    defaultCapacityProviderStrategies: [
      {
        base: 1,
        capacityProvider: 'FARGATE',
        weight: 100,
      },
    ],
  },
  {
    dependsOn: cluster,
    deleteBeforeReplace: true,
  }
);

https://github.com/pulumi/pulumi/issues/11259