https://pulumi.com logo
Title
m

most-state-94104

03/23/2023, 4:38 PM
Hello. I would like to use
preview --expect-no-changes
in combination with feature flags to confirm that no changes will happen in
prod
when I enable a feature flag in
dev
. This is mostly successful apart from aws.dms.ReplicationTask which always detects an update even if the code has not changed, regardless of whether the tasks are stopped. Is that the expected behaviour? I’ve pasted the code in the thread if helpful.
aws.dms.ReplicationTask(
                resource_name=(
                    f"{database_name}-{instance_number}-{task_number}-{migration_type}"
                    f"-{region}-{env}"
                ),
                replication_task_id=(
                    f"{database_name}-{instance_number}-{task_number}-{migration_type}"
                    f"-{region}-{env}"
                ),
                migration_type='cdc',
                cdc_start_time=null,
                replication_instance_arn=replicationInstance.replication_instance_arn,
                source_endpoint_arn=sourceEndpoint.endpoint_arn,
                target_endpoint_arn=targetEndpoint.endpoint_arn,
                replication_task_settings=json.dumps(
                    replication_task_settings, sort_keys=True, separators=(",", ":")
                ),
                table_mappings=make_table_mappings(tables, mappings["schema"]),
                tags=tagger.create_tags(
                    f"{database_name}-{instance_number}-{task_number}-{migration_type}"
                    f"-{region}-{env}"
                ),
                opts=pulumi.ResourceOptions(
                    parent=replicationInstance,
                    depends_on=[sourceEndpoint, targetEndpoint],
                    delete_before_replace=True,
                ),
            )
s

steep-toddler-94095

03/24/2023, 2:59 AM
this is probably a bug. what is/are the fields that it always has a diff for?
m

most-state-94104

03/24/2023, 9:38 AM
replicationTaskSettings
s

steep-toddler-94095

03/26/2023, 12:36 AM
what do you see in your diff? and are you sure the setting is being set in AWS on a
pulumi up
(rather than a false success like if you try to apply an IAM policy that exceeds the max length)
m

most-state-94104

03/27/2023, 10:46 AM
So I’m using a
pulumi preview --expect-no-changes
and this is a section of the output message:
@ previewing update....
    aws:iam:Role oasys-dms-dev  
    aws:dms:Endpoint oasys-source-eu-west-1-dev  
    aws:iam:Role oasys-lambda-copy-object-dev  
    aws:dms:DmsSlackNotification notify-dms-events-dev  
    aws:ec2:SecurityGroup oasys-eu-west-1-dev  
    aws:iam:RolePolicy oasys-dms-dev  
    aws:dms:Endpoint oasys-target-eu-west-1-dev  
    aws:iam:RolePolicy oasys-lambda-copy-object-dev  
    aws:sns:Topic notify-dms-events-dev  
    aws:iam:Role notify-dms-events-dev  
    aws:ec2:SecurityGroupRule egress-oasys-eu-west-1-dev  
    aws:ec2:SecurityGroupRule ingress-oasys-eu-west-1-dev  
    aws:lambda:Function notify-dms-events-dev  
    aws:dms:ReplicationInstance oasys-0-eu-west-1-dev  
    aws:iam:RolePolicyAttachment notify-dms-events-dev  
    aws:sns:TopicSubscription notify-dms-events-dev  
    aws:lambda:Permission notify-dms-events-dev  
 ~  aws:dms:ReplicationTask oasys-0-2-full-load-and-cdc-eu-west-1-dev update [diff: ~replicationTaskSettings]
    aws:dms:EventSubscription replication-instance-dev  
 ~  aws:dms:ReplicationTask oasys-0-0-full-load-eu-west-1-dev update [diff: ~replicationTaskSettings]
@ previewing update....
 ~  aws:dms:ReplicationTask oasys-0-1-cdc-eu-west-1-dev update [diff: ~replicationTaskSettings]
    aws:dms:EventSubscription replication-task-dev  
@ previewing update...........
    aws:lambda:Function delius-validate-dev  
    aws:lambda:Function oasys-validate-dev  
    aws:lambda:Permission delius-validate-dev  
    aws:s3:BucketNotification mojap-land-dev  
    aws:lambda:Permission oasys-validate-dev  
@ previewing update....
error: error: no changes were expected but changes were proposed
    pulumi:pulumi:Stack pipelines-pipelines-dev  18 warnings
s

steep-toddler-94095

03/28/2023, 5:04 AM
i meant to ask about the detailed diff. e.g. the output of a preview with the
--diff
flag. And are you able to confirm that the value you see in AWS Console matches what Pulumi has in its statefile?
m

most-state-94104

03/28/2023, 8:24 AM
Understood will send later today
Hello @steep-toddler-94095 apologies for the delay I was busy with other items and should now have more time to look into this. Are you still free to help me or should I start a new thread?
I have attached the
pulumi preview --diff
just in case 🙂
details.txt
Hello @steep-toddler-94095 I’ve also pasted the task settings for
oracle-1-0-cdc-eu-west-1-sandbox
to compare:
{
  "Logging": {
    "EnableLogging": true,
    "EnableLogContext": false,
    "LogComponents": [
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "TRANSFORMATION"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "SOURCE_UNLOAD"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "IO"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "TARGET_LOAD"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "PERFORMANCE"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "SOURCE_CAPTURE"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "SORTER"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "REST_SERVER"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "VALIDATOR_EXT"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "TARGET_APPLY"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "TASK_MANAGER"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "TABLES_MANAGER"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "METADATA_MANAGER"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "FILE_FACTORY"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "COMMON"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "ADDONS"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "DATA_STRUCTURE"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "COMMUNICATION"
      },
      {
        "Severity": "LOGGER_SEVERITY_DEFAULT",
        "Id": "FILE_TRANSFER"
      }
    ],
    "CloudWatchLogGroup": "dms-tasks-oracle-1-eu-west-1-sandbox",
    "CloudWatchLogStream": "dms-task-MEOUNVF6YGI7FP6UVZWCTUJIGDB6ORKMJ5OQN4I"
  },
  "StreamBufferSettings": {
    "StreamBufferCount": 3,
    "CtrlStreamBufferSizeInMB": 5,
    "StreamBufferSizeInMB": 8
  },
  "ErrorBehavior": {
    "FailOnNoTablesCaptured": false,
    "ApplyErrorUpdatePolicy": "LOG_ERROR",
    "FailOnTransactionConsistencyBreached": false,
    "RecoverableErrorThrottlingMax": 1800,
    "DataErrorEscalationPolicy": "STOP_TASK",
    "ApplyErrorEscalationCount": 0,
    "RecoverableErrorStopRetryAfterThrottlingMax": true,
    "RecoverableErrorThrottling": true,
    "ApplyErrorFailOnTruncationDdl": false,
    "DataTruncationErrorPolicy": "LOG_ERROR",
    "ApplyErrorInsertPolicy": "LOG_ERROR",
    "EventErrorPolicy": "IGNORE",
    "ApplyErrorEscalationPolicy": "LOG_ERROR",
    "RecoverableErrorCount": 10,
    "DataErrorEscalationCount": 0,
    "TableErrorEscalationPolicy": "STOP_TASK",
    "RecoverableErrorInterval": 30,
    "ApplyErrorDeletePolicy": "IGNORE_RECORD",
    "TableErrorEscalationCount": 0,
    "FullLoadIgnoreConflicts": true,
    "DataErrorPolicy": "LOG_ERROR",
    "TableErrorPolicy": "STOP_TASK"
  },
  "TTSettings": {
    "TTS3Settings": null,
    "TTRecordSettings": null,
    "EnableTT": false
  },
  "FullLoadSettings": {
    "CommitRate": 10000,
    "StopTaskCachedChangesApplied": false,
    "StopTaskCachedChangesNotApplied": false,
    "MaxFullLoadSubTasks": 8,
    "TransactionConsistencyTimeout": 600,
    "CreatePkAfterFullLoad": false,
    "TargetTablePrepMode": "DROP_AND_CREATE"
  },
  "TargetMetadata": {
    "ParallelApplyBufferSize": 0,
    "ParallelApplyQueuesPerThread": 0,
    "ParallelApplyThreads": 0,
    "TargetSchema": "",
    "InlineLobMaxSize": 0,
    "ParallelLoadQueuesPerThread": 0,
    "SupportLobs": true,
    "LobChunkSize": 64,
    "TaskRecoveryTableEnabled": false,
    "ParallelLoadThreads": 0,
    "LobMaxSize": 32,
    "BatchApplyEnabled": false,
    "FullLobMode": false,
    "LimitedSizeLobMode": true,
    "LoadMaxFileSize": 0,
    "ParallelLoadBufferSize": 0
  },
  "BeforeImageSettings": null,
  "ControlTablesSettings": {
    "historyTimeslotInMinutes": 5,
    "HistoryTimeslotInMinutes": 5,
    "StatusTableEnabled": false,
    "SuspendedTablesTableEnabled": false,
    "HistoryTableEnabled": false,
    "ControlSchema": "",
    "FullLoadExceptionTableEnabled": false
  },
  "LoopbackPreventionSettings": null,
  "CharacterSetSettings": null,
  "FailTaskWhenCleanTaskResourceFailed": false,
  "ChangeProcessingTuning": {
    "StatementCacheSize": 50,
    "CommitTimeout": 1,
    "BatchApplyPreserveTransaction": true,
    "BatchApplyTimeoutMin": 1,
    "BatchSplitSize": 0,
    "BatchApplyTimeoutMax": 30,
    "MinTransactionSize": 1000,
    "MemoryKeepTime": 60,
    "BatchApplyMemoryLimit": 500,
    "MemoryLimitTotal": 1024
  },
  "ChangeProcessingDdlHandlingPolicy": {
    "HandleSourceTableDropped": true,
    "HandleSourceTableTruncated": true,
    "HandleSourceTableAltered": true
  },
  "PostProcessingRules": null
}
All fixed - I had to add
"EnableLogContext": false
and
"EventErrorPolicy": "IGNORE"
- thanks for the tip to look at the
diff
!
s

steep-toddler-94095

04/13/2023, 9:27 PM
Great to hear you figured it out!