Azure recently released sidecars for azure web apps. I am trying to figure out how to create a template with such a feature does Pulumi currently support it at the moment ?

If you find it in ARM / Bicep or in the REST API then pulumi supports it with the Azure Native Provider. From a bicep example you will be able to write the pulumi code