Docker Support

Original Dev Problem¶

Removing all docker manifests from a large docker repo seems to take a long time:

~300 manifests takes ~2 minutes
~2000 manifests takes ~30-40 minutes

Reproducer:¶

1. Create and sync a docker repo such as: https://quay.io datawire/ambassador
2. Remove all docker manifests from the repo: pulp-admin docker repo remove manifest --repo-id=1-docker-dev-7915f7d0-7a98-4131-9c41-1be7b578d442 --not id=foo

Discussion with asmacado¶

• A lot of the delay were caused by the sequential removals
• If there was an outage while the task was on-going, the removal was left in a non-deterministic state
• The timing model for removal for general use cases grew incredibly fast

Solutions with the new refactor:¶

• All BLOBS and MANIFESTS are all removed at once
• Higher level units are all removed second
• Timing shrank from 300 ~=1hr to 300 ~=a few seconds

Concerns:¶

• This refactor would affect removals and syncs
• The removal, if interrupted, would leave Orphans which could affect upstream verification

Actions:¶

Upstream¶

• Need an upstream verification that ensures there is no change in functionality with remove units and sync units

Downstream¶

• Need to verify downstream verification are not affected by this change/patch
• BZ verification will just be the removal of a large number of repos takes a short amount of time

Test Permutations¶

Remove¶

Variables:

• (Permutations) Shared or Not Shared [manifest_lists, manifests]
• type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob']

Verification Scenarios:

• For Not Shared, remove $REMOVAL_TYPES and verify all units have been removed
• For Shared, remove $REMOVAL_TYPES and verify only non-shared units have been removed. Shared units should remain.

Note:

• Removal Criteria short-cut for relevant type_id being removed.

    criteria = {
        "type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob'],
        "filters": {"unit": {"_id": {"$in": all_units}}},
    }

• There are already tests covering tag removal