Story #4503: As a user, I can see in a sync task report how much work is left to do - RPM Support - Pulp

Actions

#1

Updated by jsherril@redhat.com over 5 years ago

Completed task:

{
    "_created": "2019-03-05T16:21:56.054367Z", 
    "_href": "/pulp/api/v3/tasks/3/", 
    "created_resources": [
        null
    ], 
    "error": null, 
    "finished_at": null, 
    "job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd", 
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 0, 
            "message": "Un-Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4, 
            "message": "Downloading and Parsing Metadata", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 0, 
            "message": "Downloading Artifacts", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }, 
        {
            "done": 4350, 
            "message": "Associating Content", 
            "state": "running", 
            "suffix": "", 
            "task": "/pulp/api/v3/tasks/3/", 
            "total": null
        }
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-03-05T16:21:56.163701Z", 
    "state": "running", 
    "worker": "/pulp/api/v3/workers/2/"
}

Actions

Copy link

#2

Updated by ttereshc over 5 years ago

Subject changed from Sync task provides no indication of how much work is left to do to As a user, I can see in a sync task report how much work is left to do
Sprint Candidate changed from No to Yes
Tags Pulp 3 added

Actions

Copy link

#3

Updated by ttereshc over 5 years ago

Description updated (diff)

Actions

Copy link

#4

Updated by daviddavis over 5 years ago

Groomed changed from No to Yes

Actions

Copy link

#5

Updated by ttereshc over 5 years ago

Tracker changed from Issue to Story
% Done set to 0

Actions

Copy link

#6

Updated by ttereshc over 5 years ago

Sprint set to Sprint 50

Actions

Copy link

#7

Updated by dalley over 5 years ago

Status changed from NEW to ASSIGNED
Assignee set to dalley

Actions

Copy link

#8

Updated by rchan over 5 years ago

Sprint changed from Sprint 50 to Sprint 51

Actions

Copy link

#9

Updated by bmbouter over 5 years ago

The only thing I can think of to improve this is to set the count of the # of metadata files to total=4.

For the downloading, we don't know until we need to perform downloading based on what is already local. AKA this is not something we compute up front, but as we go in batches (stream processing).

Similarly for the associating and unassociating, we learn those as we go as well due to the total differences not being known until the entire stream is processed.

Actions

Copy link

#10

Updated by daviddavis over 5 years ago

Sprint changed from Sprint 51 to Sprint 52

Actions

Copy link

#11

Updated by bmbouter over 5 years ago

Tags deleted (~~Pulp 3~~)

Actions

Copy link

#12

Updated by dalley over 5 years ago

Status changed from ASSIGNED to NEW
Assignee deleted (~~dalley~~)

Since we don't know if/how this can be resolved, I will unassign for now. It looks like this issue may either be closed or we should discuss options.

Actions

Copy link

#13

Updated by rchan over 5 years ago

Sprint changed from Sprint 52 to Sprint 53

Actions

Copy link

#14

Updated by bmbouter over 5 years ago

I can think of one improvement that can be done for rpm (but maybe not all plugins). Let's add a progress report that FirstStage will populate that shows the number of units parsed in the remote metadata. This would give you a reasonable idea of the maximum number of units handled.

@katello would this be good?

Actions

Copy link

#15

Updated by bmbouter over 5 years ago

Status changed from NEW to CLOSED - WONTFIX

So I tried this idea, and it did not work. I don't think there is anything we can do to improve it. I added another progress bar that incremented with each DeclarativeContent object created. For a fresh sync this number will equal the association number too. As it ran the "associated" count was roughly in-pace with the new progress bar in FirstStage.

{
    "_created": "2019-05-30T18:27:43.885795Z",
    "_href": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
    "created_resources": [
        null
    ],
    "error": null,
    "finished_at": null,
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [
        {
            "done": 4,
            "message": "Downloading Metadata",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 3102,
            "message": "Parsed Unit Count",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        },
        {
            "done": 2850,
            "message": "Associating Content",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
            "total": null
        }
    ],
    "spawned_tasks": [],
    "started_at": "2019-05-30T18:27:44.000766Z",
    "state": "running",
    "worker": "/pulp/api/v3/workers/4861aee5-892f-42ea-87bc-cc7ec022af6b/"
}

See how 2850 associated, yet at the head end only shows 3102 items. This ultimately will go to roughly 51000. So early on, it's very difficult to get the total count due to stream processing. I'm closing as WONTFIX.

Actions

Copy link

#16

Updated by jsherril@redhat.com over 5 years ago

Related to Issue #4895: As a user, I can see in a sync task report how much work is left to do added

Actions

Copy link

#17

Updated by bmbouter over 5 years ago

Status changed from CLOSED - WONTFIX to ASSIGNED
Assignee set to bmbouter

After IRC discussion we realized we could ask createrepo_c for totals and set the total on a package count. It looks like this:

    "parent": null,
    "progress_reports": [
        {
            "done": 0,
            "message": "Parsed Erratum",
            "state": "waiting",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 4,
            "message": "Downloading Metadata Files",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 13300,
            "message": "Associating Content",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": null
        },
        {
            "done": 13614,
            "message": "Parsed Packages",
            "state": "running",
            "suffix": null,
            "task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
            "total": 57327
        }