Registry - manifest deletion leaves dangling tags in case of error on storage

Hi,

I'm using private registry (version 2.6.1) with swift storage driver as a part of our infrastructure. There are some images which are tagged (and untagged) by test environment using HTTP v2 api few thousand times a day.

It happened few times, that (different) manifest deletion requests for given image were all failing with 500 constantly from some point in time until action was taken to manually repair storage.

I looked into the code and found two causes of this issue, which together make manifest deleting impossible.

First, manifest deletion process looks like this:

1. delete manifest blob
2. scan all tags and find those linking to deleted manifest
3. remove found tag links

If there is any error during tag links scan (it happens ocasionally, this is remote storage), further search is aborted and we leave dangling tag link(s) behind. From now on broken tags show in tag list, but there is no way to delete them, because further calls stop at 1 since manifest is already gone.

I think we should check if manifest exists and then scan and delete linking tags. Only after succesful tag removal we can remove manifest. If we fail to remove manifest, nothing bad happens, because client can try again and (hopefully) delete manifest.

Now, imagine that some tag link is broken, so we get persistent error, i.e path not found. From now on we will always fail in tag scan phase and client will get 500. That way new tags appear, but can not be removed. This is undesirable, since delete's computational complexity is proportional to number of tags and even after manual intervention we can quickly reach swift's limit of directory listing (10000) and have even more problems later.

In my cases, such storage corruption started from unsuccesful PUT manifest call.

PUT logic looks like this:

1. store manifest
2. store index link in tag directory
3. store current link in tag directory

If we fail at 3, we leave corrupted storage which ends in state described before. To recover, we have to manually delete tag directory.

Since (remote) storage errors are inevitable and can have arbitrary timespan (net link down, etc) we need to cope with such inconsistencies. This leads us back to tag link scan.

Tags are scanned on per tag directory basis. For each directory, a current/link is checked if it matches given manifest. If this link does not exist, we leave with error. Instead, we should ignore PathNotFoundError and continue scanning. We could also log, if not delete, such dangling tags since they are useless.

Apologizing for such a long post, let me summarize what I'm proposing:

1. Scan and delete tag links first, then if no error delete manifest
2. Scan tags allowing for storage inconsistencies and remove dangling links if they appear