Hello Team, I'm creating azure keyvault with some network acl

NetworkAcls: &keyvault.NetworkRuleSetArgs{
                    DefaultAction: pulumi.String("Deny"),
                    Bypass:        pulumi.String("AzureServices"),
                    IpRules: &keyvault.IPRuleArray{
                        &keyvault.IPRuleArgs{ // allow some public IP
                            Value: pulumi.String("X.X.X.X.X/32"),
                        },
                    },
                },

When running pulumi destroy from GitHub , I'm getting the following

RESPONSE 403: 403 Forbidden
2025-02-12T22:46:19.9678918Z     ERROR CODE: Forbidden
2025-02-12T22:46:19.9679282Z     --------------------------------------------------------------------------------
2025-02-12T22:46:19.9679575Z     {
2025-02-12T22:46:19.9679767Z       "error": {
2025-02-12T22:46:19.9680016Z         "code": "Forbidden",
2025-02-12T22:46:19.9682240Z         "message": "Client address is not authorized and caller is not a trusted service.\r\nClient address: 52.234.43.252\r\nCaller: appid=***;oid=***;iss=<https://sts.windows.net//|https://sts.windows.net//>\r\nVault: az-cloud-iac-kv;location=westus",
2025-02-12T22:46:19.9683381Z         "innererror": {
2025-02-12T22:46:19.9683709Z           "code": "ForbiddenByFirewall"
2025-02-12T22:46:19.9683979Z         }
2025-02-12T22:46:19.9684160Z       }
2025-02-12T22:46:19.9684330Z     }

Do I need to allow any CIDR range for GitHub ? I'm using SP with contributor access on the subscription level from GH, isn't it elevating the access for the destroy process ? how do I set my SP as "trusted service", I do have "Allow trusted Microsoft services to bypass this firewall" on the KV. please advise.