When you read Terraform docs there’s a page with lifecycle attributes that you can specify for the resources. Among the others there’s prevent_destroy attribute whose behaviour is documented as:
prevent_destroy (bool) – This flag provides extra protection against the destruction of a given resource. When this is set to true, any plan that includes a destroy of this resource will return an error message.
I thought it will protect me from accidental deletion of the resource. How wrong was I!
So… how does it work?
It works in the following case: you define some resource which has attribute that – if changed – causes destruction and re-creation of the resource. If prevent_destroy is set to true that plan will fail. Terraform will prevent you from shooting yourself in the foot. That’s fine.
What is “not fine” is the case when you accidentally remove (or rename) some resource from your Terraform code. Terraform will not do anything to prevent destroying your precious resources and will happily delete them…
I – and I’m not alone – think it’s not intuitive:
The documentation for prevent_destroy says
This flag provides extra protection against the destruction of a given resource. When this is set to true, any plan that includes a destroy of this resource will return an error message
However, removing or commenting out a resource seems to still destroy it with no warning, even though technically it was a plan that includes the destruction of the resource. What I expected was that the state file would remember that it was created with prevent_destroy, and prevent its destruction even if it was no longer part of the parsed configuration. It would be reasonable to expect that the user would be forced to remove the prevent_destroy flag, terraform apply, then remove it from the configuration if they were absolutely sure they wanted to go ahead.
[ source ]
And the reply for above complaint:
Currently removing a resource from configuration is treated as intent to remove the resource, since that’s an explicit action on your part rather than it happening as a part of some other action, such as resource replacement or a whole-configuration destroy.
[ source ]
You’re not wrong @beanaroo! prevent_destroy is I think broken and arguably unusable in it’s current form. The problem is that prevent_destroy doesn’t just keep the flagged resource from being destroyed (i.e. exclude its destruction from plans), instead it blocks planning. Often I want to destroy everything except stateful resources, like a DB or EIPs or pet EBS or EFS or S3 store, terraform can’t do that. You have to leave those stateful resources outside of terraform. If you include them you can’t use prevent_destroy and plan, so instead you have to manually check plans.
Quite a few people have the same issue, but unfortunately the mod’s don’t seem to agree it is actually a problem 😢 and keep flagging this issue and an ‘enhancement’. See #3874 for a two-year long sad tale of frustrated terraform users 😝
[ source ]
The problem is real, so how can we secure our resources from destruction?
There are a couple of ways we can try to protect our resources:
Firstly, if the resource provides deletion protection (like AWS EC2 Instance or AWS RDS) we should use that. That will disable API for removing the service so Terraform will fail on apply phase. Not perfect as there are only a few services that supports that feature.
Secondly we can use Service Control Policies. This way we can blacklist some permissions, e.g. the ones related to deletion of important resources. You can find sample policy for RDS in my last post about protecting backups on RDS. The similar approach to the one showed can also work with almost any resource on AWS.
Thirdly you should comment on the issues on Terraform GitHub – maybe we’ll convince the community that this flag’s behavior has to change!
I hope the advices from the article will help you avoid some accidental deletions of important resources :)