millions-furniture-75402
10/31/2022, 3:38 PMlet awsProviderDefaults;
if (!process.env.AWS_ACCESS_KEY_ID) {
aws.sdk.config.credentials = new aws.sdk.SharedIniFileCredentials({ profile: awsConfig.get("profile") });
awsProviderDefaults = { profile: awsConfig.get("profile") };
} else {
awsProviderDefaults = {
accessKey: process.env.AWS_ACCESS_KEY_ID,
secretKey: process.env.AWS_SECRET_ACCESS_KEY,
token: aws.sdk.config.sessionToken,
};
}
const awsUsEast1 = new aws.Provider("east", {
region: "us-east-1",
...awsProviderDefaults,
});
salmon-motherboard-78006
10/31/2022, 7:45 PMaws:mwaa:Environment (dev-aqua-airflow):
error: 1 error occurred:
* error creating MWAA Environment: ValidationException: Failed to assume role arn:aws:iam::<account_id>:role/dev-airflow-execution-role. This could be due to the role's trust policy. Please ensure your role is assumable by '<http://airflow-env.amazonaws.com|airflow-env.amazonaws.com>' Service Principal and try again.
And this is what my execution role looks like:
mwaa_execution_role = aws.iam.Role(f"{stack}-airflow-execution-role",
name=f"{stack}-airflow-execution-role",
assume_role_policy=json.dumps({
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": [
"<http://airflow.amazonaws.com|airflow.amazonaws.com>",
"<http://airflow-env.amazonaws.com|airflow-env.amazonaws.com>"
],
},
"Effect": "Allow",
},
]
}))
I then decided to create the S3 bucket, role and policies before creating the MWAA Environment and this is the error I’m getting now:
aws:mwaa:Environment (dev-aqua-airflow):
error: 1 error occurred:
* creating urn:pulumi:dev::data-ml-airflow::aws:mwaa/environment:Environment::dev-aqua-airflow: 1 error occurred:
* error waiting for MWAA Environment (dev-aqua-airflow-dd6bc3e) creation: unexpected state 'CREATE_FAILED', wanted target 'AVAILABLE'. last error: %!s(<nil>)
When I go to the AWS Console, this is the error I see:
Error code
INCORRECT_CONFIGURATION
Message
You may need to check the execution role permissions policy for your environment, and that each of the VPC networking components required by the environment are configured to allow traffic. Troubleshooting: <https://docs.aws.amazon.com/mwaa/latest/userguide/troubleshooting.html>
And this is my Pulumi MWAA code:
airflow_env = aws.mwaa.Environment(f"{stack}-aqua-airflow",
dag_s3_path="dags/",
execution_role_arn=mwaa_execution_role.arn,
airflow_version='2.2.2',
kms_key=mwaa_kms_key.arn,
logging_configuration=aws.mwaa.EnvironmentLoggingConfigurationArgs(
dag_processing_logs=aws.mwaa.EnvironmentLoggingConfigurationDagProcessingLogsArgs(
enabled=True,
log_level="DEBUG",
),
scheduler_logs=aws.mwaa.EnvironmentLoggingConfigurationSchedulerLogsArgs(
enabled=True,
log_level="INFO",
),
task_logs=aws.mwaa.EnvironmentLoggingConfigurationTaskLogsArgs(
enabled=True,
log_level="WARNING",
),
webserver_logs=aws.mwaa.EnvironmentLoggingConfigurationWebserverLogsArgs(
enabled=True,
log_level="ERROR",
),
worker_logs=aws.mwaa.EnvironmentLoggingConfigurationWorkerLogsArgs(
enabled=True,
log_level="CRITICAL",
),
),
network_configuration=aws.mwaa.EnvironmentNetworkConfigurationArgs(
security_group_ids=[vpc["vpcDefaultSecurityGroupID"]],
subnet_ids=[vpc["privateSubnetsIDs"][0], vpc["privateSubnetsIDs"][1]],
),
source_bucket_arn=airflow_dags_bucket.arn,
tags={
"Environment": f"{stack}",
},
opts=ResourceOptions(
depends_on=[mwaa_execution_role, mwaa_kms_key, airflow_dags_bucket]))
Any idea what I’m doing incorrectly?
I’m trying to look into this:
That your Amazon VPC is configured to allow network traffic between the different AWS resources used by your Amazon MWAA environment, as defined in About networking on Amazon MWAA. For example, your VPC security group must either allow all traffic in a self-referencing rule, or optionally specify the port range for HTTPS port range 443 and a TCP port range 5432.
fierce-horse-21860
10/31/2022, 8:21 PMPerforming query: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1
Error while querying: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1 (3702ms)
{
"processingId": 1,
"queueSize": 1,
"queryKey": [
[
"CREATE TABLE arch_council_app.cube_aws_billing_cost_by_account AS SELECT\n \"cube_aws_billing\".\"ACCOUNT_ALIAS\" \"cube_aws_billing__account_alias\", date_trunc('MONTH', CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) \"cube_aws_billing__bill_date_month\", sum(\"cube_aws_billing\".\"SERVICE_COST\") \"cube_aws_billing__service_cost\"\n FROM\n \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\" GROUP BY 1, 2",
[]
],
[
[
{
"refresh_key": "463123"
}
]
]
],
"queuePrefix": "SQL_PRE_AGGREGATIONS_STANDALONE_default",
"timeInQueue": 1,
"preAggregationId": "cube_aws_billing.cost_by_account",
"newVersionEntry": {
"table_name": "arch_council_app.cube_aws_billing_cost_by_account",
"structure_version": "ulrf25hc",
"content_version": "bxxvnrki",
"last_updated_at": 1667246128534,
"naming_version": 2
},
"preAggregation": {
"preAggregationId": "cube_aws_billing.cost_by_account",
"timezone": "UTC",
"timestampFormat": "YYYY-MM-DD[T]HH:mm:ss.SSS[Z]",
"tableName": "arch_council_app.cube_aws_billing_cost_by_account",
"invalidateKeyQueries": [
[
"SELECT FLOOR((UNIX_TIMESTAMP()) / 3600) as refresh_key",
[],
{
"external": true,
"renewalThreshold": 120
}
]
],
"type": "rollup",
"external": true,
"previewSql": [
"SELECT * FROM arch_council_app.cube_aws_billing_cost_by_account LIMIT 1000",
[]
],
"preAggregationsSchema": "arch_council_app",
"loadSql": [
"CREATE TABLE arch_council_app.cube_aws_billing_cost_by_account AS SELECT\n \"cube_aws_billing\".\"ACCOUNT_ALIAS\" \"cube_aws_billing__account_alias\", date_trunc('MONTH', CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) \"cube_aws_billing__bill_date_month\", sum(\"cube_aws_billing\".\"SERVICE_COST\") \"cube_aws_billing__service_cost\"\n FROM\n \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\" GROUP BY 1, 2",
[]
],
"sql": [
"SELECT\n \"cube_aws_billing\".\"ACCOUNT_ALIAS\" \"cube_aws_billing__account_alias\", date_trunc('MONTH', CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) \"cube_aws_billing__bill_date_month\", sum(\"cube_aws_billing\".\"SERVICE_COST\") \"cube_aws_billing__service_cost\"\n FROM\n \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\" GROUP BY 1, 2",
[]
],
"uniqueKeyColumns": [
"\"cube_aws_billing__account_alias\"",
"\"cube_aws_billing__bill_date_month\""
],
"aggregationsColumns": [
"sum(\"cube_aws_billing__service_cost\")"
],
"dataSource": "default",
"granularity": "month",
"preAggregationStartEndQueries": [
[
"select min(CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) from \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"",
[]
],
[
"select max(CONVERT_TIMEZONE('UTC', \"cube_aws_billing\".\"BILL_DATE\"::timestamp_tz)::timestamp_ntz) from \"ARCH_COUNCIL_APP\".\"AWS_BILLING\" AS \"cube_aws_billing\"",
[]
]
],
"indexesSql": [],
"createTableIndexes": [],
"readOnly": false
},
"addedToQueueTime": 1667246128534
}
OperationFailedError: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
at createError (/cube/node_modules/snowflake-sdk/lib/errors.js:536:15)
at Object.exports.createOperationFailedError (/cube/node_modules/snowflake-sdk/lib/errors.js:315:10)
at Object.callback (/cube/node_modules/snowflake-sdk/lib/services/sf.js:647:28)
at /cube/node_modules/snowflake-sdk/lib/http/base.js:111:25
at done (/cube/node_modules/urllib/lib/urllib.js:589:5)
at /cube/node_modules/urllib/lib/urllib.js:953:9
at decodeContent (/cube/node_modules/urllib/lib/urllib.js:740:14)
at handleResponseCloseAndEnd (/cube/node_modules/urllib/lib/urllib.js:924:7)
at IncomingMessage.<anonymous> (/cube/node_modules/urllib/lib/urllib.js:962:7)
at IncomingMessage.emit (events.js:412:35)
at IncomingMessage.emit (domain.js:475:12)
at endReadableNT (internal/streams/readable.js:1333:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)
Error querying db: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1
--
"SELECT `cube_aws_billing__account_alias` `cube_aws_billing__account_alias`, sum(`cube_aws_billing__service_cost`) `cube_aws_billing__service_cost` FROM arch_council_app.cube_aws_billing_cost_by_account AS `cube_aws_billing__cost_by_account` GROUP BY 1 ORDER BY 2 DESC LIMIT 50"
--
{
"params": []
}
Error: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
at QueryQueue.parseResult (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:146:13)
at QueryQueue.executeInQueue (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:135:19)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at PreAggregationLoader.loadPreAggregationWithKeys (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:742:7)
at preAggregationPromise (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/PreAggregations.ts:1946:28)
at QueryOrchestrator.fetchQuery (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryOrchestrator.ts:158:59)
at OrchestratorApi.executeQuery (/cube/node_modules/@cubejs-backend/server-core/src/core/OrchestratorApi.ts:85:20)
at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1230:21
at async Promise.all (index 0)
at ApiGateway.getSqlResponseInternal (/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1228:31)
at /cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1357:28
at async Promise.all (index 0)
at ApiGateway.load (/cube/node_modules/@cubejs-backend/api-gateway/src/gateway.ts:1348:23)
at /cube/node_modules/@cubejs-backend/api-gateway/src/sql-server.ts:101:13
Orchestrator error: 0d3bb4b3-7837-4420-b91e-3334042f2ba1-span-1 (3832ms)
--
{
"measures": [
"cube_aws_billing.service_cost"
],
"dimensions": [
"cube_aws_billing.account_alias"
],
"segments": [],
"order": [
[
"cube_aws_billing.service_cost",
"desc"
]
],
"limit": 50
}
--
{
"securityContext": {},
"appName": "NULL",
"protocol": "postgres",
"apiType": "sql"
}
Error: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
2022-10-31 19:55:32,267 ERROR [cubejs_native::transport] [transport] load - strange response, success which contains error: V1Error { error: "Error: SQL access control error:\nInsufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'" }
Cube SQL Error: undefined
{
"apiType": "sql",
"protocol": "postgres",
"appName": "NULL"
}
Error during processing PostgreSQL message: Internal: Execution error: Internal: Error: SQL access control error:
Insufficient privileges to operate on database 'GBI_OTHERS_DATA_ENG_DB'
However I am able to execute the create table as query with the
CUBEJS_DB_USER=gbi_others_data_eng_db_arch_council_user
CREATE TABLE arch_council_app.cube_aws_billing_cost_by_account AS SELECT
cube_aws_billing.ACCOUNT_ALIAS cube_aws_billing__account_alias,
date_trunc('MONTH', CONVERT_TIMEZONE('UTC', cube_aws_billing.BILL_DATE::timestamp_tz)::timestamp_ntz) cube_aws_billing__bill_date_month,
sum(cube_aws_billing.SERVICE_COST) cube_aws_billing__service_cost
FROM ARCH_COUNCIL_APP.AWS_BILLING AS cube_aws_billing
GROUP BY 1, 2
What am I missing here?little-whale-73288
11/01/2022, 8:39 AMv3.45.0
yet, so I can't install it using https://github.com/pulumi/setup-pulumi, is this WAI?orange-airport-64592
11/01/2022, 9:58 AMfrom pulumi_aws.apigateway import RestApi
from pulumi_aws_apigateway import RestAPI
hallowed-train-1850
11/01/2022, 1:44 PMwet-gigabyte-99270
11/01/2022, 2:40 PMripe-russia-4239
11/01/2022, 4:42 PMtf2pulumi
still supported? It looks like the web version's stopped working, and the repo hasn't been updated in ~18 months as far as I can seesquare-window-64861
11/02/2022, 1:40 PMpulumi.Output<string>
as a variable ? I’m using mocha.js & chai
Error:
(property) TargetGroupArgs.vpcId: pulumi.Output<string>
Type 'string' is not assignable to type 'Output<string>'.
ambitious-agent-35343
11/02/2022, 3:16 PMambitious-alligator-62127
11/02/2022, 3:27 PMpulumi stack change-secrets-provider <azurekeyvault://cg-test-kv.vault.azure.net/keys/dev-stacks>
error: open keeper <azurekeyvault://cg-test-kv.vault.azure.net/keys/dev-stacks>: failed to Dial default KeyVault: MSI not available
increasing verbosity does not add any additional output. Couldn't tell what area this is exactly related to. If I need to post in different channel, let me know.
Suggestions?polite-ocean-13631
11/02/2022, 3:32 PMpulumi stack graph
command that claims to "export a stack’s dependency graph to a file". How do you define a stack's dependencies? How are these dependencies used?few-plastic-88435
11/02/2022, 4:14 PMkind-country-41992
11/02/2022, 5:31 PMwhite-chef-55657
11/02/2022, 5:45 PMpreview --expect-no-changes
and if that command returns non-zero run update
but I was hoping there’s a nice way of doing it instead of this ugly hack
has anyone encountered something similar?bitter-salesclerk-37287
11/02/2022, 8:10 PMpulumi up
from my laptop, and have the service running with the proper configurations. (?)polite-ocean-13631
11/02/2022, 10:13 PMregister_output
as explained here: https://www.pulumi.com/docs/intro/concepts/resources/components/#registering-component-outputs
How can we hide outputs? I want to make it so that my custom components only expose a small subset of their outputs to ensure we don't accidentally develop a dependency on something unstable.damp-honey-93158
11/03/2022, 9:41 AMpurple-coat-73595
11/03/2022, 3:23 PMkind-country-41992
11/03/2022, 3:57 PMtall-crowd-93084
11/03/2022, 7:44 PMpanic: interface conversion: interface {} is string, not int
not sure if this is a bug in the new version or related to me trying to import some resources from hetzner, but wanted to flag it here in case it's the former. Happy to raise an issuegorgeous-minister-41131
11/04/2022, 1:41 AMset_config()
method doesn't support using a path key?
stack.set_config("pulumi:disable-default-providers[0]", auto.ConfigValue("*"))
Results in a literal config with [0] in its name.. but if I use --path on the cli I can force it to a list of string... is there a more correct way to do this?? setting complex types via automation?busy-whale-18770
11/04/2022, 10:02 AMfull-pencil-66049
11/04/2022, 1:56 PMDashboardJson
from an exported JSON from Datadog itself, but I'm having trouble with the definition of query_values
.
🧵purple-minister-96588
11/04/2022, 2:29 PMhost = cache.host.apply(lambda k: k)
Which from what I can tell is meant to be right, but host remains an output.gray-fountain-32432
11/04/2022, 2:53 PMflaky-arm-38472
11/04/2022, 3:36 PMgorgeous-country-43026
11/04/2022, 4:48 PMdamp-honey-93158
11/04/2022, 5:14 PMbusy-dentist-99842
11/04/2022, 5:35 PMdeleteBeforeReplace: true
property in CustomResourceOptions
. It seems like it's not getting applied to the resource. When I run pulumi up
. Pulumi still tries to do a create before delete on a replacement.
I put in the issue below but does anyone see an obvious error in my code?
import * as aws from '@pulumi/aws';
const cluster = new aws.ecs.Cluster('playground', {name: 'playground'});
new aws.ecs.ClusterCapacityProviders(
'playground',
{
clusterName: 'playground',
capacityProviders: ['FARGATE'],
defaultCapacityProviderStrategies: [
{
base: 1,
capacityProvider: 'FARGATE',
weight: 100,
},
],
},
{
dependsOn: cluster,
deleteBeforeReplace: true,
}
);
https://github.com/pulumi/pulumi/issues/11259