Project

Profile

Help

Story #4503

closed

As a user, I can see in a sync task report how much work is left to do

Added by jsherril@redhat.com over 5 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 53
Quarter:

Description

This story should set the total in the progress report if possible.
It is likely that for RPMs there is a total number of them specified in the beginning of primary.xml.
There is no counter available right away for other content types like Errata, Modules and so on.

When syncing a large yum repo, such as http://mirror.math.princeton.edu/pub/fedora/linux/releases/28/Everything/x86_64/os/, after ~15 minutes i still see no indication of how much work is to be done for each step:

This was done using 'on_demand'

{
    "_created": "2019-03-05T16:21:56.054367Z", 
    "_href": "/pulp/api/v3/tasks/3/", 
    "created_resources": [
        null
    ], 
    "error": null, 
    "finished_at": null, 
    "job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd", 
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 0, 
            "message": "Un-Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4, 
            "message": "Downloading and Parsing Metadata", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 0, 
            "message": "Downloading Artifacts", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 3750, 
            "message": "Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-03-05T16:21:56.163701Z", 
    "state": "running", 
    "worker": "/pulp/api/v3/workers/2/"
}

Related issues

Related to File Support - Issue #4895: As a user, I can see in a sync task report how much work is left to do CLOSED - CURRENTRELEASEbmbouterActions
Related to Container Support - Issue #5193: Progress presenter has tasks with no value for totalCLOSED - WONTFIXActions
Actions #1

Updated by jsherril@redhat.com over 5 years ago

Completed task:

{
    "_created": "2019-03-05T16:21:56.054367Z", 
    "_href": "/pulp/api/v3/tasks/3/", 
    "created_resources": [
        null
    ], 
    "error": null, 
    "finished_at": null, 
    "job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd", 
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 0, 
            "message": "Un-Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4, 
            "message": "Downloading and Parsing Metadata", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 0, 
            "message": "Downloading Artifacts", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4350, 
            "message": "Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-03-05T16:21:56.163701Z", 
    "state": "running", 
    "worker": "/pulp/api/v3/workers/2/"
}
Actions #2

Updated by ttereshc over 5 years ago

  • Subject changed from Sync task provides no indication of how much work is left to do to As a user, I can see in a sync task report how much work is left to do
  • Sprint Candidate changed from No to Yes
  • Tags Pulp 3 added
Actions #3

Updated by ttereshc over 5 years ago

  • Description updated (diff)
Actions #4

Updated by daviddavis over 5 years ago

  • Groomed changed from No to Yes
Actions #5

Updated by ttereshc over 5 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0
Actions #6

Updated by ttereshc over 5 years ago

  • Sprint set to Sprint 50
Actions #7

Updated by dalley over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #8

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 50 to Sprint 51
Actions #9

Updated by bmbouter over 5 years ago

The only thing I can think of to improve this is to set the count of the # of metadata files to total=4.

For the downloading, we don't know until we need to perform downloading based on what is already local. AKA this is not something we compute up front, but as we go in batches (stream processing).

Similarly for the associating and unassociating, we learn those as we go as well due to the total differences not being known until the entire stream is processed.

Actions #10

Updated by daviddavis over 5 years ago

  • Sprint changed from Sprint 51 to Sprint 52
Actions #11

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions #12

Updated by dalley over 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

Since we don't know if/how this can be resolved, I will unassign for now. It looks like this issue may either be closed or we should discuss options.

Actions #13

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 52 to Sprint 53
Actions #14

Updated by bmbouter over 5 years ago

I can think of one improvement that can be done for rpm (but maybe not all plugins). Let's add a progress report that FirstStage will populate that shows the number of units parsed in the remote metadata. This would give you a reasonable idea of the maximum number of units handled.

@katello would this be good?

Actions #15

Updated by bmbouter over 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX

So I tried this idea, and it did not work. I don't think there is anything we can do to improve it. I added another progress bar that incremented with each DeclarativeContent object created. For a fresh sync this number will equal the association number too. As it ran the "associated" count was roughly in-pace with the new progress bar in FirstStage.

{
    "_created": "2019-05-30T18:27:43.885795Z",
    "_href": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
    "created_resources": [
        null
    ],
    "error": null,
    "finished_at": null,
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [
        {
            "done": 4,
            "message": "Downloading Metadata",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 3102,
            "message": "Parsed Unit Count",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 2850,
            "message": "Associating Content",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        }
    ],
    "spawned_tasks": [],
    "started_at": "2019-05-30T18:27:44.000766Z",
    "state": "running",
    "worker": "/pulp/api/v3/workers/4861aee5-892f-42ea-87bc-cc7ec022af6b/"
}

See how 2850 associated, yet at the head end only shows 3102 items. This ultimately will go to roughly 51000. So early on, it's very difficult to get the total count due to stream processing. I'm closing as WONTFIX.

Actions #16

Updated by jsherril@redhat.com over 5 years ago

  • Related to Issue #4895: As a user, I can see in a sync task report how much work is left to do added
Actions #17

Updated by bmbouter over 5 years ago

  • Status changed from CLOSED - WONTFIX to ASSIGNED
  • Assignee set to bmbouter

After IRC discussion we realized we could ask createrepo_c for totals and set the total on a package count. It looks like this:

    "parent": null,
    "progress_reports": [
        {
            "done": 0,
            "message": "Parsed Erratum",
            "state": "waiting",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 4,
            "message": "Downloading Metadata Files",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 13300,
            "message": "Associating Content",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 13614,
            "message": "Parsed Packages",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": 57327
        }

Added by bmbouter over 5 years ago

Revision ae158745 | View on GitHub

Add total counts to pulp_rpm

Previously the user didn't have a good idea of two things:

  1. How many packages and erratum are there?
  2. How far into those counts are we?

This add two new progress bars that users can see the total counts on.

https://pulp.plan.io/issues/4503 closes #4503

Actions #18

Updated by bmbouter over 5 years ago

  • Status changed from ASSIGNED to POST
Actions #19

Updated by bmbouter over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #20

Updated by ipanova@redhat.com about 5 years ago

  • Related to Issue #5193: Progress presenter has tasks with no value for total added
Actions #21

Updated by ttereshc almost 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF