Project

Profile

Help

Issue #1638

closed

docker v2: report counts for manifests, tags, and blobs during syncs

Added by bbuckingham@redhat.com over 6 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Docker:
Platform Release:
Target Release - Docker:
2.0.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

The progress_report for syncing a docker v2 repository does not appear to be fully updated to show the progress of a sync. Below is an example syncing library/elasticsearch from dockerhub.

The following are the content unit counts for the repository after a sync has completed:

Docker Blob: 209 Docker Manifest: 40 Docker Tag: 40

The following is the final progress report (copied from katello's dynflow console):

progress_report: docker_importer: - num_success: 1 description: Downloading manifests step_type: sync_step_metadata items_total: 1 state: FINISHED error_details: [] details: '' num_failures: 0 step_id: 0455ffee-185e-471b-a746-15ac40a8c49e num_processed: 1 - num_success: 1 description: Copying units already in pulp step_type: get_local items_total: 1 state: FINISHED error_details: [] details: '' num_failures: 0 step_id: e60e5252-06e1-4a12-bf15-331ec7d27ef9 num_processed: 1 - num_success: 1 description: Copying units already in pulp step_type: get_local items_total: 1 state: FINISHED error_details: [] details: '' num_failures: 0 step_id: cf55788b-70c0-473e-9762-33cc2686d175 num_processed: 1 - num_success: 1 description: Downloading remote files step_type: sync_step_download items_total: 209 state: FINISHED error_details: [] details: '' num_failures: 0 step_id: dee58d5a-185b-4569-8437-6e69ae95ff8a num_processed: 1 - num_success: 1 description: Saving Manifests and Blobs step_type: sync_step_save items_total: 1 state: FINISHED error_details: [] details: '' num_failures: 0 step_id: 8566eef2-9da8-473b-983b-ddfc755152fc num_processed: 1 - num_success: 1 description: Saving Tags step_type: sync_step_save items_total: 1 state: FINISHED error_details: [] details: '' num_failures: 0 step_id: fdca141a-dd29-4a56-a24f-1ba1ed013553 num_processed: 1

From the progress report, there do not appear to be any counts related to the manifests(40) and tags(40). Since user's are not likely to be familiar with blobs, shouldn't these also be reflected in the progress?

In addition, the step that appears to take the longest is sync_step_download ('Downloading remote files') which appears to be downloading the 209 blobs; however, while that downloading is occurring the counts for num_success, num_failure, num_process and num_total only appear to get updated at the start and finish. E.g.

success=0/failure=0/processed=0/total=209 success=1/failure=0/processed=1/total=209

Actions #1

Updated by amacdona@redhat.com over 6 years ago

  • Subject changed from docker v2: improved sync reporting to docker v2: report counts for manifests, tags, and blobs during syncs

The 'Downloading remote files' reporting was a problem I introduced https://github.com/pulp/pulp_docker/pull/121 and it should be fixed by this PR https://github.com/pulp/pulp_docker/pull/128

The remaining issue is that manifests, tags, and blobs should have counts so their progress can be better reported.

Actions #2

Updated by bbuckingham@redhat.com over 6 years ago

At Randy's recommendation, I updated to the latest nightly.

With nightly, I am seeing the counts for blobs get updated during the sync. Awesome!

The only part that remains, if possible, will be showing the user the count for manifests and tags as part of the progress report.

Actions #3

Updated by amacdona@redhat.com over 6 years ago

  • Project changed from Pulp to Docker Support
  • Category deleted (14)
  • Target Release - Docker set to 2.0.0

Adding this after pulp_docker 2.0 would represent a backwards incompatible change, so it should be done now if possible.

Actions #4

Updated by pcreech over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to pcreech
Actions #5

Updated by pcreech over 6 years ago

  • Status changed from ASSIGNED to POST

Added by pcreech over 6 years ago

Revision 62c9df3c

Update tag and manifest counts during sync

Update tag and manifest counts during sync for the sync report

closes #1638 https://pulp.plan.io/issues/1638

Added by pcreech over 6 years ago

Revision 62c9df3c

Update tag and manifest counts during sync

Update tag and manifest counts during sync for the sync report

closes #1638 https://pulp.plan.io/issues/1638

Added by pcreech over 6 years ago

Revision 62c9df3c

Update tag and manifest counts during sync

Update tag and manifest counts during sync for the sync report

closes #1638 https://pulp.plan.io/issues/1638

Added by pcreech over 6 years ago

Revision 62c9df3c

Update tag and manifest counts during sync

Update tag and manifest counts during sync for the sync report

closes #1638 https://pulp.plan.io/issues/1638

Actions #6

Updated by pcreech over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #7

Updated by ipanova@redhat.com over 6 years ago

  • Triaged changed from No to Yes
Actions #8

Updated by dkliban@redhat.com about 6 years ago

  • Status changed from MODIFIED to 5
Actions #9

Updated by semyers almost 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #10

Updated by bmbouter about 3 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF