Snapshot integrity error when a refresh-program update replaces a provider used by a component
[!NOTE] This issue was authored by an AI agent on @iwahbe's behalf.
What happened?
A stack contains a component resource whose state records a provider reference. The program is then run with pulumi up --refresh --run-program and the provider needs to be replaced (its diff reports replacement keys). The engine replaces the component along with the provider. The delete-replaced steps for the old provider and the old component are not ordered: they run in parallel, and when the old provider is deleted first, the old component still refers to it. The update then aborts with a snapshot integrity error. Because the order is random, roughly 40% of runs fail.
The same program with a plain pulumi up (no refresh) succeeds.
Originally observed in a TestFuzz run: https://github.com/pulumi/pulumi/actions/runs/35089763530/job/104773403788.
Example
Deterministic lifecycle test (pkg/engine/lifecycletest):
prov := &pkgresource.State{
Type: "pulumi:providers:pkgA", Custom: true, ID: "id-prov",
URN: "urn:pulumi:test-stack::test-project::pulumi:providers:pkgA::prov",
}
provRef, _ := providers.NewReference(prov.URN, prov.ID)
comp := &pkgresource.State{
Type: "pkgA:m:TypeA",
URN: "urn:pulumi:test-stack::test-project::pkgA:m:TypeA::comp",
Provider: provRef.String(),
}
snap := &deploy.Snapshot{Resources: []*pkgresource.State{prov, comp}}
// Provider whose DiffConfig always requests a replacement.
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{
DiffConfigF: func(context.Context, plugin.DiffConfigRequest) (plugin.DiffConfigResponse, error) {
return plugin.DiffResult{Changes: plugin.DiffSome, ReplaceKeys: []resource.PropertyKey{"foo"}}, nil
},
}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
resp, _ := monitor.RegisterResource("pulumi:providers:pkgA", "prov", true, deploytest.ResourceOptions{
Inputs: resource.PropertyMap{"foo": resource.NewProperty("bar")},
})
ref, _ := providers.NewReference(resp.URN, resp.ID)
_, err := monitor.RegisterResource("pkgA:m:TypeA", "comp", false, deploytest.ResourceOptions{Provider: ref.String()})
return err
})
_, err = lt.TestOp(engine.Update).RunStep(p.GetProject(), p.GetTarget(t, snap), lt.TestUpdateOptions{
T: t, HostF: deploytest.NewPluginHostF(nil, nil, programF, nil, nil, loaders...),
UpdateOptions: engine.UpdateOptions{Refresh: true, RefreshProgram: true},
}, false, p.BackendClient, nil, "0")post-step event returned an error: failed to verify snapshot: resource urn:pulumi:test-stack::test-project::pkgA:m:TypeA::comp refers to unknown provider urn:pulumi:test-stack::test-project::pulumi:providers:pkgA::prov::id-provSteps recorded in a failing run: refresh prov, create-replacement prov, replace prov, refresh comp, create-replacement comp, replace comp, delete-replaced prov (fails here; delete-replaced comp has not run yet). A passing run deletes comp before prov.
Without Refresh/RefreshProgram, the deletes are always ordered comp then prov and the update succeeds.
Expected: the update succeeds every time. The old component is deleted before the old provider it refers to.
Regression
git bisect with the test above (30 runs per commit) points at 6215b8211b, "create refresh steps for custom and provider resources" (https://github.com/pulumi/pulumi/pull/21817, 2026-04-13). Its parent passes 30 consecutive runs. The engine used to skip refresh steps for components and providers; since that change they get persisted internal refresh steps, and the delete scheduler no longer sees the dependency between the refreshed copies.
Environment
pulumi/pulumi master at ceb2e86de7 (v3.263.0 release PR #24679).
Source: pulumi/pulumi