Story #4503
closed
As a user, I can see in a sync task report how much work is left to do
100%
Description
This story should set the total in the progress report if possible.
It is likely that for RPMs there is a total number of them specified in the beginning of primary.xml.
There is no counter available right away for other content types like Errata, Modules and so on.
When syncing a large yum repo, such as http://mirror.math.princeton.edu/pub/fedora/linux/releases/28/Everything/x86_64/os/, after ~15 minutes i still see no indication of how much work is to be done for each step:
This was done using 'on_demand'
{
"_created": "2019-03-05T16:21:56.054367Z",
"_href": "/pulp/api/v3/tasks/3/",
"created_resources": [
null
],
"error": null,
"finished_at": null,
"job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 0,
"message": "Un-Associating Content",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
},
{
"done": 4,
"message": "Downloading and Parsing Metadata",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
},
{
"done": 0,
"message": "Downloading Artifacts",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
},
{
"done": 3750,
"message": "Associating Content",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
}
],
"spawned_tasks": [],
"started_at": "2019-03-05T16:21:56.163701Z",
"state": "running",
"worker": "/pulp/api/v3/workers/2/"
}
Related issues
Updated by jsherril@redhat.com over 5 years ago
Completed task:
{
"_created": "2019-03-05T16:21:56.054367Z",
"_href": "/pulp/api/v3/tasks/3/",
"created_resources": [
null
],
"error": null,
"finished_at": null,
"job_id": "155c9a3e-4853-4c08-9c24-fa4484cb48dd",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 0,
"message": "Un-Associating Content",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
},
{
"done": 4,
"message": "Downloading and Parsing Metadata",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
},
{
"done": 0,
"message": "Downloading Artifacts",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
},
{
"done": 4350,
"message": "Associating Content",
"state": "running",
"suffix": "",
"task": "/pulp/api/v3/tasks/3/",
"total": null
}
],
"spawned_tasks": [],
"started_at": "2019-03-05T16:21:56.163701Z",
"state": "running",
"worker": "/pulp/api/v3/workers/2/"
}
Updated by ttereshc over 5 years ago
- Subject changed from Sync task provides no indication of how much work is left to do to As a user, I can see in a sync task report how much work is left to do
- Sprint Candidate changed from No to Yes
- Tags Pulp 3 added
Updated by ttereshc over 5 years ago
- Tracker changed from Issue to Story
- % Done set to 0
Updated by dalley over 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
Updated by bmbouter over 5 years ago
The only thing I can think of to improve this is to set the count of the # of metadata files to total=4.
For the downloading, we don't know until we need to perform downloading based on what is already local. AKA this is not something we compute up front, but as we go in batches (stream processing).
Similarly for the associating and unassociating, we learn those as we go as well due to the total differences not being known until the entire stream is processed.
Updated by daviddavis over 5 years ago
- Sprint changed from Sprint 51 to Sprint 52
Updated by dalley over 5 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
dalley)
Since we don't know if/how this can be resolved, I will unassign for now. It looks like this issue may either be closed or we should discuss options.
Updated by bmbouter over 5 years ago
I can think of one improvement that can be done for rpm (but maybe not all plugins). Let's add a progress report that FirstStage will populate that shows the number of units parsed in the remote metadata. This would give you a reasonable idea of the maximum number of units handled.
@katello would this be good?
Updated by bmbouter over 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
So I tried this idea, and it did not work. I don't think there is anything we can do to improve it. I added another progress bar that incremented with each DeclarativeContent object created. For a fresh sync this number will equal the association number too. As it ran the "associated" count was roughly in-pace with the new progress bar in FirstStage.
{
"_created": "2019-05-30T18:27:43.885795Z",
"_href": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
"created_resources": [
null
],
"error": null,
"finished_at": null,
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 4,
"message": "Downloading Metadata",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
"total": null
},
{
"done": 3102,
"message": "Parsed Unit Count",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
"total": null
},
{
"done": 0,
"message": "Downloading Artifacts",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
"total": null
},
{
"done": 2850,
"message": "Associating Content",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/b09cfbd4-2437-4f3b-8522-859358eea3f1/",
"total": null
}
],
"spawned_tasks": [],
"started_at": "2019-05-30T18:27:44.000766Z",
"state": "running",
"worker": "/pulp/api/v3/workers/4861aee5-892f-42ea-87bc-cc7ec022af6b/"
}
See how 2850 associated, yet at the head end only shows 3102 items. This ultimately will go to roughly 51000. So early on, it's very difficult to get the total count due to stream processing. I'm closing as WONTFIX.
Updated by jsherril@redhat.com over 5 years ago
- Related to Issue #4895: As a user, I can see in a sync task report how much work is left to do added
Updated by bmbouter over 5 years ago
- Status changed from CLOSED - WONTFIX to ASSIGNED
- Assignee set to bmbouter
After IRC discussion we realized we could ask createrepo_c for totals and set the total on a package count. It looks like this:
"parent": null,
"progress_reports": [
{
"done": 0,
"message": "Parsed Erratum",
"state": "waiting",
"suffix": null,
"task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
"total": null
},
{
"done": 4,
"message": "Downloading Metadata Files",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
"total": null
},
{
"done": 13300,
"message": "Associating Content",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
"total": null
},
{
"done": 0,
"message": "Downloading Artifacts",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
"total": null
},
{
"done": 13614,
"message": "Parsed Packages",
"state": "running",
"suffix": null,
"task": "/pulp/api/v3/tasks/1133b7c8-1861-4515-95aa-2f84c614951e/",
"total": 57327
}
Added by bmbouter over 5 years ago
Updated by bmbouter over 5 years ago
- Status changed from ASSIGNED to POST
PR available at: https://github.com/pulp/pulp_rpm/pull/1360
Updated by bmbouter over 5 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset ae1587453c5ce548c77908c662eee0df4ac69d65.
Updated by ipanova@redhat.com over 5 years ago
- Related to Issue #5193: Progress presenter has tasks with no value for total added
Updated by ttereshc almost 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Add total counts to pulp_rpm
Previously the user didn't have a good idea of two things:
This add two new progress bars that users can see the total counts on.
https://pulp.plan.io/issues/4503 closes #4503