If I use

aws.secretsmanager.getSecret

and pass that into an input for DMS endpoint credentials, are the values stored in pulumi state? are they encrypted?

They will be stored in the state file, and currently will not be encrypted. With the work in https://github.com/pulumi/pulumi/issues/397 which will be available in the next few weeks though these will be encrypted in the checkpoint (and you will have control over any additional input/outputs you would like to keep encrypted).