Project

Profile

Help

Story #4503

As a user, I can see in a sync task report how much work is left to do

Added by jsherril@redhat.com almost 3 years ago. Updated almost 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 53
Quarter:

Description

This story should set the total in the progress report if possible.
It is likely that for RPMs there is a total number of them specified in the beginning of primary.xml.
There is no counter available right away for other content types like Errata, Modules and so on.

When syncing a large yum repo, such as http://mirror.math.princeton.edu/pub/fedora/linux/releases/28/Everything/x86_64/os/, after ~15 minutes i still see no indication of how much work is to be done for each step:

This was done using 'on_demand'

{
    "_created": "2019-03-05T16:21:56.054367Z", 
    "_href": "/pulp/api/v3/tasks/3/", 
    "created_resources": [
        null
    ], 
    "error": null, 
    "finished_at": null, 
    "job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd", 
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 0, 
            "message": "Un-Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4, 
            "message": "Downloading and Parsing Metadata", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 0, 
            "message": "Downloading Artifacts", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 3750, 
            "message": "Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-03-05T16:21:56.163701Z", 
    "state": "running", 
    "worker": "/pulp/api/v3/workers/2/"
}

Related issues

Related to File Support - Issue #4895: As a user, I can see in a sync task report how much work is left to do CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Container Support - Issue #5193: Progress presenter has tasks with no value for totalCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision ae158745 View on GitHub
Added by bmbouter over 2 years ago

Add total counts to pulp_rpm

Previously the user didn't have a good idea of two things:

  1. How many packages and erratum are there?
  2. How far into those counts are we?

This add two new progress bars that users can see the total counts on.

https://pulp.plan.io/issues/4503 closes #4503

History

#1 Updated by jsherril@redhat.com almost 3 years ago

Completed task:

{
    "_created": "2019-03-05T16:21:56.054367Z", 
    "_href": "/pulp/api/v3/tasks/3/", 
    "created_resources": [
        null
    ], 
    "error": null, 
    "finished_at": null, 
    "job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd", 
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 0, 
            "message": "Un-Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4, 
            "message": "Downloading and Parsing Metadata", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 0, 
            "message": "Downloading Artifacts", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4350, 
            "message": "Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-03-05T16:21:56.163701Z", 
    "state": "running", 
    "worker": "/pulp/api/v3/workers/2/"
}

#2 Updated by ttereshc almost 3 years ago

  • Subject changed from Sync task provides no indication of how much work is left to do to As a user, I can see in a sync task report how much work is left to do
  • Sprint Candidate changed from No to Yes
  • Tags Pulp 3 added

#3 Updated by ttereshc almost 3 years ago

  • Description updated (diff)

#4 Updated by daviddavis almost 3 years ago

  • Groomed changed from No to Yes

#5 Updated by ttereshc almost 3 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#6 Updated by ttereshc over 2 years ago

  • Sprint set to Sprint 50

#7 Updated by dalley over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#8 Updated by rchan over 2 years ago

  • Sprint changed from Sprint 50 to Sprint 51

#9 Updated by bmbouter over 2 years ago

The only thing I can think of to improve this is to set the count of the # of metadata files to total=4.

For the downloading, we don't know until we need to perform downloading based on what is already local. AKA this is not something we compute up front, but as we go in batches (stream processing).

Similarly for the associating and unassociating, we learn those as we go as well due to the total differences not being known until the entire stream is processed.

#10 Updated by daviddavis over 2 years ago

  • Sprint changed from Sprint 51 to Sprint 52

#11 Updated by bmbouter over 2 years ago

  • Tags deleted (Pulp 3)

#12 Updated by dalley over 2 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

Since we don't know if/how this can be resolved, I will unassign for now. It looks like this issue may either be closed or we should discuss options.

#13 Updated by rchan over 2 years ago

  • Sprint changed from Sprint 52 to Sprint 53

#14 Updated by bmbouter over 2 years ago

I can think of one improvement that can be done for rpm (but maybe not all plugins). Let's add a progress report that FirstStage will populate that shows the number of units parsed in the remote metadata. This would give you a reasonable idea of the maximum number of units handled.

@katello would this be good?

#15 Updated by bmbouter over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

So I tried this idea, and it did not work. I don't think there is anything we can do to improve it. I added another progress bar that incremented with each DeclarativeContent object created. For a fresh sync this number will equal the association number too. As it ran the "associated" count was roughly in-pace with the new progress bar in FirstStage.

{
    "_created": "2019-05-30T18:27:43.885795Z",
    "_href": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
    "created_resources": [
        null
    ],
    "error": null,
    "finished_at": null,
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [
        {
            "done": 4,
            "message": "Downloading Metadata",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 3102,
            "message": "Parsed Unit Count",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 2850,
            "message": "Associating Content",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        }
    ],
    "spawned_tasks": [],
    "started_at": "2019-05-30T18:27:44.000766Z",
    "state": "running",
    "worker": "/pulp/api/v3/workers/4861aee5-892f-42ea-87bc-cc7ec022af6b/"
}

See how 2850 associated, yet at the head end only shows 3102 items. This ultimately will go to roughly 51000. So early on, it's very difficult to get the total count due to stream processing. I'm closing as WONTFIX.

#16 Updated by jsherril@redhat.com over 2 years ago

  • Related to Issue #4895: As a user, I can see in a sync task report how much work is left to do added

#17 Updated by bmbouter over 2 years ago

  • Status changed from CLOSED - WONTFIX to ASSIGNED
  • Assignee set to bmbouter

After IRC discussion we realized we could ask createrepo_c for totals and set the total on a package count. It looks like this:

    "parent": null,
    "progress_reports": [
        {
            "done": 0,
            "message": "Parsed Erratum",
            "state": "waiting",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 4,
            "message": "Downloading Metadata Files",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 13300,
            "message": "Associating Content",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 13614,
            "message": "Parsed Packages",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": 57327
        }

#18 Updated by bmbouter over 2 years ago

  • Status changed from ASSIGNED to POST

#19 Updated by bmbouter over 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#20 Updated by ipanova@redhat.com over 2 years ago

  • Related to Issue #5193: Progress presenter has tasks with no value for total added

#21 Updated by ttereshc almost 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF