Test #5181

Updated by bherring almost 5 years ago

h3.    Original Dev Problem 

 Removing all docker manifests from a large docker repo seems to take a long time: 

 ~300 manifests takes ~2 minutes 
 ~2000 manifests takes ~30-40 minutes 

 h3.    Reproducer: 

 1.    Create and sync a docker repo such as:         datawire/ambassador 
 2.    Remove all docker manifests from the    repo:      pulp-admin    docker repo remove manifest --repo-id=1-docker-dev-7915f7d0-7a98-4131-9c41-1be7b578d442     --not id=foo 

 h3.    Discussion with asmacado 

 * A lot of the delay were caused by the sequential removals 
 * If there was an outage while the task was on-going, the removal was left in a non-deterministic state 
 * The timing model for removal for general use cases grew incredibly fast 

 h3.    Solutions with the new refactor: 

 * All BLOBS and MANIFESTS are all removed at once 
 * Higher level units are all removed second 
 * Timing shrank from 300 ~=1hr to 300 ~=a few seconds 

 h3. Concerns: 

 * This refactor would affect removals and syncs 
 * The removal, if interrupted, would leave Orphans which could affect upstream verification 

 h3.    Actions: 

 h4. Upstream 

 *    Need an upstream verification that ensures there is no change in functionality with remove units and sync units 

 h4. Downstream 

 * Need to verify downstream verification are not affected by this change/patch 
 * BZ verification will just be the removal of a large number of repos takes a short amount of time 

 h3. Test Permutations 

 h4. Remove 


 * (Permutations) Shared or Not Shared [manifest_lists, manifests] 
 * type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob'] 

 Verification Scenarios: 

 * For Not Shared, remove $REMOVAL_TYPES and verify all units have been removed 
 * For Shared, remove $REMOVAL_TYPES and verify only non-shared units have been removed. Shared units should remain. 


 * Removal Criteria short-cut for relevant type_id being removed. 

     criteria = { 
         "type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob'], 
         "filters": {"unit": {"_id": {"$in": all_units}}}, 

 * There are already tests covering tag removal