(Apologies in advance for posting in what looks li...
# pulumi-ai
m
(Apologies in advance for posting in what looks like a more or less restricted channel). For those interested, Claude 3.5 Sonnet does an incredible job with Pulumi TS code. I've used it about 400-600 times now, and its' accuracy / token recall is great to the point where 80% of the output or so is a direct copy/paste and it just works. Use the following system prompt: "You are an expert DevOps engineer using AWS, Pulumi, and TypeScript. Your environment is already set up, and you focus on code with concise explanations." here: https://console.anthropic.com/workbench/ with the attached settings (move max tokens to 4k from default). The model is trained up to this month, so it has all the latest APIs/interfaces/etc and uses them accurately.
w
Thanks @magnificent-soccer-44287 for calling this out. We’ve seen similarly good results from some internal testing. We’re looking at bringing Claude as an option into Pulumi AI (and Copilot) to let users take advantage of this through our products as well!
1
m
@white-balloon-205 yeah it's a pretty solid bet right now. My entire platform team (8 people) is using it with great results.
btw feel free to apply these papers to your experiments, they're fairly impactful: Accuracy check -> https://arxiv.org/abs/2406.00975v2 Infra concept linking -> https://arxiv.org/abs/2405.07500v1 A caution on fine tuning -> https://arxiv.org/abs/2405.05904v2 Luna helps a lot bringing the accuracy and reducing the hallucinations of fine-tuning on your latest API docs.
❤️ 1