#31309 Terraform apply destroys resources that are still referenced from other resources

Terraform Version

Terraform v1.2.3
on darwin_arm64

Terraform Configuration Files

This configuration relies on a dummy provider, see github/prashantv/terraform-dep-order-repro

resource "tftest_notifier" "n1" {
  email = "[email protected]"
}

resource "tftest_notifier" "n2" {
  email = "[email protected]"
}

resource "tftest_policy" "p1" {
  notifier_ids = [
    tftest_notifier.n1.id,
    tftest_notifier.n2.id,
  ]
}

ref

After applying the above, remove "n2" and the reference to it,

resource "tftest_notifier" "n1" {
  email = "[email protected]"
}

resource "tftest_policy" "p1" {
  notifier_ids = [
    tftest_notifier.n1.id,
  ]
}

Applying this will fail as n2 is destroyed first, while there is still a policy referencing it.

Debug Output

https://gist.github.com/prashantv/45859f8607e690ff22990c310381f8c8

Expected Behavior

When the config removing "n2" is applied, the reference is removed first, and then n2 is removed successfully.

Actual Behavior

Terraform tries to remove "n2" before updating the policy reference, and the remove fails since the notifier is still referenced by a policy.

│ Error: failed to delete Notifier: cannot delete notifier, as policies [policy-174441.453993] still refer to it

Steps to Reproduce

Full steps for reproduction are in the repo.

Summary is:

Use a dummy provider with 2 resources: notifiers/policies. policies can reference multiple notifiers.
Apply: notifiers "n1" & "n2", and policy "p1" which references "n1" & "n2"
Remove "n2" and the reference "n2" in "p2"

--- a/repro/main.tf
+++ b/repro/main.tf
@@ -11,13 +11,13 @@ resource "tftest_notifier" "n1" {
   email = "[email protected]"
 }

-resource "tftest_notifier" "n2" {
-  email = "[email protected]"
-}
+# resource "tftest_notifier" "n2" {
+# email = "[email protected]"
+# }

 resource "tftest_policy" "p1" {
   notifier_ids = [
     tftest_notifier.n1.id,
-    tftest_notifier.n2.id,
+    # tftest_notifier.n2.id,
   ]
 }

Run terraform apply, which will try to destroy "n2" first. (which is still referenced, and hence fails).

Additional Context

There is a workaround which causes the destroy to happen after the update: setting the lifecycle meta-argument create_before_destroy. This has to be set on the resoruce and applied before the delete operation. However, this workaround has some issues:

Every user of the provider would need to set this meta lifecycle argument as part of every single resource which is a poor user experience (noisy, duplicated across possibly hundreds of resources, and not very obvious that they need to do this). There's no way for a provider to specify a default create_before_destroy.
create_before_destroy is not appropriate for some resources. If a resource has a field that must be unique (e.g., name), and a separate attribute that has ForceNew, then create_before_destroy is not compatible with these resources, so the workaround does not work. We acutally have this combination -- the notifier resource has a name that must be unique, and a user-selected immutable ID field that uses ForceNew, so create_before_destroy shifts the problem to users being unable to change names.

Ideally the solution would be:

A provider configuration to indicate that destroys of a resource cannot happen till all references to the reference have been updated
The configuration doesn't cause creates to happen before destroys (so it's compatible with resources that have fields that must be unique + ForceNew fields).
The user doesn't need to do anything special.

References

There are a few existing issues which are almost all closed:

create_before_destroy is recommended in many of the above, but as mentioned in "Additional Context", it's a workaround with drawbacks rather than a solution to this problem.