Test #5181

Removing docker manifests from a docker repository takes a long time

Added by bherring over 1 year ago. Updated about 1 year ago.

Platform Release:
Pulp 2


Original Dev Problem

Removing all docker manifests from a large docker repo seems to take a long time:

~300 manifests takes ~2 minutes
~2000 manifests takes ~30-40 minutes


1. Create and sync a docker repo such as: datawire/ambassador
2. Remove all docker manifests from the repo: pulp-admin docker repo remove manifest --repo-id=1-docker-dev-7915f7d0-7a98-4131-9c41-1be7b578d442 --not id=foo

Discussion with asmacado

  • A lot of the delay were caused by the sequential removals
  • If there was an outage while the task was on-going, the removal was left in a non-deterministic state
  • The timing model for removal for general use cases grew incredibly fast

Solutions with the new refactor:

  • All BLOBS and MANIFESTS are all removed at once
  • Higher level units are all removed second
  • Timing shrank from 300 ~=1hr to 300 ~=a few seconds


  • This refactor would affect removals and syncs
  • The removal, if interrupted, would leave Orphans which could affect upstream verification



  • Need an upstream verification that ensures there is no change in functionality with remove units and sync units


  • Need to verify downstream verification are not affected by this change/patch
  • BZ verification will just be the removal of a large number of repos takes a short amount of time

Test Permutations



  • (Permutations) Shared or Not Shared [manifest_lists, manifests]
  • type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob']

Verification Scenarios:

  • For Not Shared, remove $REMOVAL_TYPES and verify all units have been removed
  • For Shared, remove $REMOVAL_TYPES and verify only non-shared units have been removed. Shared units should remain.


  • Removal Criteria short-cut for relevant type_id being removed.
    criteria = {
        "type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob'],
        "filters": {"unit": {"_id": {"$in": all_units}}},
  • There are already tests covering tag removal
almost-all-man-lists-busybox (210 KB) almost-all-man-lists-busybox cProfile: Removal of all manifest lists in busybox repository, 07/10/2019 12:00 AM (5.76 KB), 07/25/2019 02:48 PM (15 KB), 07/25/2019 02:48 PM

Related issues

Copied from Docker Support - Issue #4549: Removing docker manifests from a docker repository takes a long timeCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 086b660a View on GitHub
Added by bherring about 1 year ago

Adding tests to docker plugin for removing and syncing of docker units

Issue #4549 refactors the handling of removing tags, maniftest_lists, manifests, and blobs within the docker plugin.

Adding test permutations to ensure the behavior of removing all, shared, and non-shared units remains the same.

This test will not address the large performance gain by the refactor.

Refs #5181

Revision ecba1db4 View on GitHub
Added by bherring about 1 year ago

Refactor of with changes from 4549

With the refactor of the docker importer's remove function to
increase performance, content removal needs to be functional verified.

The cases covered with content post-count verification for all units:

1. Remove all manifest_lists sequentially.
2. Remove all manifests sequentially.
3. Remove all blobs sequentially.
4. Remove all manifest_lists batch.
5. Remove all manifests batch.
6. Remove all blobs batch.
7. Remove some non-shared manifest lists.
8. Remove some non-shared manifest.
9. Remove some shared manifests lists and verify shared units are not
   recursively removed.
10. Remove some shared manifests and verify shared units are not
    recursively removed.

The fixture includes:

* 2 relatively independent manifest lists (no shared manifests,
  no shared blobs between them)
* 2 manifest lists that share some (but not all) manifests, and those
  manifest share some (but not all) blobs. This only requires the creation
  of 1 manifest list that shares some content with one of the first
  “independent manifest lists”.
* 2 relatively independent manifests
* 2 manifests that share (some but not all) blobs

In order to sync the content, each content unit must be recursively related
to at least 1 tag.

refs #4549 #5161

closes #5181


#1 Updated by bherring over 1 year ago

  • Copied from Issue #4549: Removing docker manifests from a docker repository takes a long time added

#2 Updated by bherring over 1 year ago

  • Description updated (diff)

#4 Updated by bherring about 1 year ago

  • Description updated (diff)
  • Status changed from NEW to ASSIGNED

#5 Updated by bherring about 1 year ago

  • Status changed from ASSIGNED to CLOSED - COMPLETE


All testing complete.

All information aggregated in PR via

Please register to edit this issue

Also available in: Atom PDF