Hi All, Im new to pulumi and have used aws cdk fo...
# aws
k
Hi All, Im new to pulumi and have used aws cdk for IaC previously. I have created IaC using pulumi for a simple web application which uses RDS and EC2 instance (not fancy stuff like ecs, autoscaling etc due to limited incoming traffic at current stage). When I tried to update a role
principal
, which is not related RDS instance, pulumi returning an error saying that
it cannot update RDS instance
. This has been happening since 3 weeks involving any change not related RDS instance, associated security groups and IAM role-policies. I came across similar issues but could not find any solution. 1. https://github.com/pulumi/pulumi-aws/issues/2426 2. https://github.com/pulumi/pulumi-aws/issues/574 Here is my code,
Copy code
const subnetGrp = new aws.rds.SubnetGroup('rds-subnet-group', {
    subnetIds: [...privateSubnetIds, ...publicSubnetIds, defaultPrivateSubnetId],
  });

  const rdsInstance = new aws.rds.Instance(
    'db',
    {
      dbName: dbConfig.name,
      identifier: dbIdentifier,
      engineVersion: '8.4.3',
      engine: 'mysql',
      instanceClass: dbConfig.instance_type,
      allocatedStorage: 20,
      storageType: 'gp3',
      dbSubnetGroupName: subnetGrp.name,
      vpcSecurityGroupIds: [rdsSg.id],
      multiAz: isProd,
      iamDatabaseAuthenticationEnabled: true,
      username: dbConfig.username,
      password: dbConfig.password,
      port: dbConfig.port,
      publiclyAccessible: false,
      tags: { Name: prefix('rds-instance') },
      availabilityZone: 'ap-southeast-2a',
      skipFinalSnapshot: !isProd,
      storageEncrypted: true,
      maxAllocatedStorage: dbConfig.max_storage,
      deletionProtection: true,
      snapshotIdentifier: dbConfig.snapshot_id,
    },
    { protect: true },
  );
Here is preview log. I only updated
bitbucket-role-pipeline
Federated Principal nothing else, also this particular role is not associated to RDS instance.
Copy code
error: unable to replace resource "urn:pulumi:dev::myapp::aws:rds/instance:Instance::wa-dev-db"
 ++ aws:rds:Instance wa-dev-db create replacement [diff: ~dbName,username]; error: unable to replace resource "urn:pulumi:dev::myapp::aws:rds/instance:Instance::wa-dev-db"
 +- aws:rds:Instance wa-dev-db replace [diff: ~dbName,username]; error: unable to replace resource "urn:pulumi:dev::myapp::aws:rds/instance:Instance::wa-dev-db"
 ~  aws:iam:RolePolicy wa-dev-ec2DbConnect update [diff: ~policy]
 ~  aws:iam:Role bitbucket-role-pipeline update [diff: ~assumeRolePolicy]
 -- aws:rds:Instance wa-dev-db delete original [diff: ~dbName,username]; error: unable to replace resource "urn:pulumi:dev::myapp::aws:rds/instance:Instance::wa-dev-db"
Any help wold be appreciated. Here is my checklist to ensure im doing it right way 1. No manual changes to any resources from AWS web UI console 2. No changes tp VPC, subnet 3. No changes to IAM role - policies related to RDS instance, ec2 instance 4. Not renamed database 5. No changes to database username, password etc 6. Added
pulumi refresh
step before
pulumi preview
l
The RDS instance is being replaced because you've changed the name of the instance (property
dbName
), and AWS replaces the instance when you make this change. To see the before and after values, you can run
pulumi preview --diff
or view the diff in the Pulumi console. I see that you've set
dbName
to
dcConfig.name
, so it looks like a config change has cause this.
k
dbConfig.name
is hardcoded per stack in
<http://Pulumi.yaml.dev|Pulumi.yaml.dev>
Here is
<http://Pulumi.yaml.dev|Pulumi.yaml.dev>
Copy code
config:
  aws:region: ap-south-1
  sure-webapp:db:
    username: admin
    name: grocery
    password:
      secure: ghsdfhsdgfh=
    port: 3308
    max_storage: 50
    instance_type: db.t4g.micro
    snapshot_id: rds:rds-dev-2025-04-05-19-05
If importing these values from
Pulumi.*.yaml
is not recommended, then i will hard code them directly into the code.
l
No it's fine. I'm just wondering if that file has been edited. What's the diff showing?
k
zero git diff in that file. I have a dedicated role
bktp-iam-role
with policies that allow upload files to s3 from bitbucket pipeline. I have made few updates to ``bktp-iam-role` .
Copy code
const role = new aws.iam.Role('bktp-iam-role', {
    name: prefix('cd-pipeline'),
    assumeRolePolicy: {
      Version: '2012-10-17',
      Statement: [
        {
          Effect: 'Allow',
          Principal: pulumi.output(federatedPrincipal).apply((fp) => ({
            Federated: fp,
          })),
          Action: 'sts:AssumeRoleWithWebIdentity',
          Condition: {
            StringEquals: pulumi.all([]).apply(([k,v]) => ({
              `<http://api.bitbucket.org/2.0/workspaces/${k}/pipelines-config/identity/oidc:aud|api.bitbucket.org/2.0/workspaces/${k}/pipelines-config/identity/oidc:aud>`:
                `ari:cloud:bitbucket::workspace/${v}`,
            })),
          },
        },
      ],
    },
  });


  const s3BucketPolicy = new aws.iam.Policy(prefix('codeupload'), {
    name: prefix('code-upload-pipeline'),
    description: 'Policy for specific S3 bucket access',
    policy: pulumi.output(bucketArn).apply((bktArn) =>
      JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Action: ['s3:ListBucket', 's3:GetObject', 's3:PutObject'],
            Resource: [bktArn, `${bktArn}/*`],
          },
        ],
      }),
    ),
  });
--- Also one not i missed, is im using passing
snapshotIdentifier
from an old database that was created manually. It previously had different database name, username and password and these values are not same as one ive configured in yaml file for dev stack.
Here is the diff
Copy code
error: unable to replace resource "urn:pulumi:dev::mygrocery-webapp::aws:rds/instance:Instance::wa-dev-db"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
    +-aws:rds/instance:Instance: (replace) 🔒
        [id=db-HFgRDGRFcsddssjdfyguyf743443]
        [urn=urn:pulumi:dev::mygrocery-webapp::aws:rds/instance:Instance::wa-dev-db]
        [provider=urn:pulumi:dev::mygrocery-webapp::pulumi:providers:aws::default_6_66_2::673567-ytr23-76wtg-tf76s-7832tussdsd]
      + dbName: "grocery"
l
There you go. You've added the dbName, and that requires that the database is recreated.
Either don't add the dbName property, or allow the database to be recreated.
k
its same as previous one that was hard coded in
Pulumi.dev.yaml
.
node_modules/@pulumi/aws/rds/instance.d.ts
says
Copy code
The name of the database to create when the DB instance is created. If this parameter is not specified, no database is created in the DB instance. Note that this does not apply for Oracle or SQL Server engines. See the [AWS documentation](<https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/create-db-instance.html>) for more details on what applies for those engines. If you are providing an Oracle db name, it needs to be in all upper case. Cannot be specified for a replica.
or is it pulling difference due to snapshot ?
l
Perhaps it was
name
that had the value
grocery
before? And now dbName has? It's not due to the snapshot. The Pulumi output is very clear: the property
dbName
used to not have a value, and now it does.
It's odd that the instance has to be recreated just because a database in the instance is being created. However, the documentation is very clear, and the error message absolutely agrees with the documentation.
k
Ran
pulumi stack export
, it shows
"dbName": "",
. You are right here. Following to that, why does it doesnt have
dbName
set in pulumi stack. I have previously ran
pulumi up
once.
when i run first time it created database successfully.
l
It created the instance? Or the database inside the instance?
According to those docs, creating the instance with no dbName set should create an instance with no initial database inside it. Which is probably the best way to go.
k
it created
aws:rds:instance
with identifier
wa-dev-db
I see the the database
grocery_bus
which is from snapshot of old database when i ran below query
Copy code
show databases;
l
From which we can infer that
grocery
wasn't created when the instance was. And your current code says "make an instance that includes the initial database
grocery
. " So that's what's happening. AWS is trying to create the database .
k
1. very first time when
pulumi up
run a. It created one RDS instance and
grocery_bus
database inside it from the snapshot identifier supplied, where as ive configured
grocery
in my
pulumi.dev.yaml
b. it created ec2 and other IAM, and security groups 2. I made a change in IAM role related to bitbucket IAM role and did not changed anything related to database (snapshot, username, dbname etc) 3. Then i try to run
pulumi up
second time. This time it is failed with error saying that
failed to update rds instance
4. From your suggestions here in this thread, a. i tried to run
pulumi stack export
, i saw
dbName
is empty string. As you mentioned
dbName
was updates thats what
pulumi preview
tells. You are absolutely correct 😍. ---- Yes it didnt created
grocery
database but it created
grocery_bus
inside rds instance
wa-dev-db
in the first run. In the second run it is failing as dbName is empty string and pulumi thought i updated the dbName to
grocery
. This is my understanding, i migh have misunderstood some concepts here. https://pulumi-community.slack.com/archives/CRH5ENVDX/p1744012288014759?thread_ts=1743925123.686939&amp;cid=CRH5ENVDX
l
If it created grocery_bus, then you'd see
~dbName
, not
+dbName
. Perhaps you created that database, or as you suggested earlier, it was created from a snapshot restore? Pulumi thinks that there's no initial database in that instance.
k
it was created from pulumi
snapshotIdentifier
supplied. Just now started to
pulumi destroy
and
pulumi up
again in a fresh environment .
i see below in pulumi updates
Copy code
+  aws:rds:Instance wa-dev-db create
i didnt changed anything in the code. I ran
pulumi preview --diff
, it shows diff, but i havent changes any thing. even empty space.
Copy code
error: unable to replace resource "urn:pulumi:dev::myapp::aws:rds/instance:Instance::wa-dev-db"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
    +-aws:rds/instance:Instance: (replace) 🔒
        [id=db-5LRAGHCVMOQ7AHLNKVLAJPOM]
        [urn=urn:pulumi:dev::myapp::aws:rds/instance:Instance::wa-dev-db]
        [provider=urn:pulumi:dev::myapp::pulumi:providers:aws::default_6_66_2::d633dc46-00cc-460a-b614-d5bdf934b268]
      + dbName             : "grocery_bus"
      ~ maxAllocatedStorage: 0 => 50
      ~ port               : 0 => 3307
    ~ aws:iam/rolePolicy:RolePolicy: (update)
        [id=wa-dev-iam-role:wa-dev-db-access]
        [urn=urn:pulumi:dev::myapp::aws:iam/rolePolicy:RolePolicy::wa-dev-ec2DbConnect]
        [provider=urn:pulumi:dev::myapp::pulumi:providers:aws::default_6_66_2::d633dc46-00cc-460a-b614-d5bdf934b268]
      ~ policy: (json) {
          - Statement: [
          -     [0]: {
                  - Action  : [
                  -     [0]: "rds-db:connect"
                    ]
                  - Effect  : "Allow"
                  - Resource: [
                  -     [0]: "arn:aws:rds-db:ap-south-1:123456789:dbuser:db-5LRAGHCVMOQ7AHLNKVLAJPOM/minions"
                    ]
                }
            ]
          - Version  : "2012-10-17"
        }
also used
pulumi refresh
to ensure pulumi state got updated
l
Did you do the refresh between the two ups? That can cause it. If there are default values set by AWS but not by your code, then a refresh pulls them back into state. Then Pulumi might be seeing a difference between code and state. If this is what is happening, you can either never refresh (which is usually good enough), or update your code to match AWS' defaults.
k
👋 Paul, Previously havent not used refresh saw the same issue and see the last comment here
<https://github.com/pulumi/pulumi-aws/issues/2426>
. After that started using
refresh
and the issue still persists. Also asked some of my peers, they also said the difference is not correct. My client is little bit frustrated and last hope on the tech choice i made.
l
That ticket you linked does not apply in your case. There really is a change in your situation, you just don't know why it's there. AWS is correctly trying to replace the RDS instance given that change, you just don't want to make the change. You're going to have to figure out why the dbName property is not set when you initially create that resource, and why it is set later. If you remove the dbName property from your code, does the attempt to replace your instance go away? I presume it does.