Test #5181

Updated by bherring over 2 years ago

h3. Original Dev Problem

Removing all docker manifests from a large docker repo seems to take a long time:

~300 manifests takes ~2 minutes
~2000 manifests takes ~30-40 minutes

h3. Reproducer:

1. Create and sync a docker repo such as: datawire/ambassador
2. Remove all docker manifests from the repo: pulp-admin docker repo remove manifest --repo-id=1-docker-dev-7915f7d0-7a98-4131-9c41-1be7b578d442 --not id=foo

h3. Discussion with asmacado

* A lot of the delay were caused by the sequential removals
* If there was an outage while the task was on-going, the removal was left in a non-deterministic state
* The timing model for removal for general use cases grew incredibly fast

h3. Solutions with the new refactor:

* All BLOBS and MANIFESTS are all removed at once
* Higher level units are all removed second
* Timing shrank from 300 ~=1hr to 300 ~=a few seconds

h3. Concerns:

* This refactor would affect removals and syncs
* The removal, if interrupted, would leave Orphans which could affect upstream verification

h3. Actions:

h4. Upstream

* Need an upstream verification that ensures there is no change in functionality with remove units and sync units

h4. Downstream

* Need to verify downstream verification are not affected by this change/patch
* BZ verification will just be the removal of a large number of repos takes a short amount of time

h3. Test Permutations

h4. Remove


* (Permutations) Shared or Not Shared [manifest_lists, manifests]
* type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob']

Verification Scenarios:

* For Not Shared, remove $REMOVAL_TYPES and verify all units have been removed
* For Shared, remove $REMOVAL_TYPES and verify only non-shared units have been removed. Shared units should remain.


* Removal Criteria short-cut for relevant type_id being removed.

criteria = {
"type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob'],
"filters": {"unit": {"_id": {"$in": all_units}}},

* There are already tests covering tag removal