Actions
Issue #9398
closedNo optimized syncs after a change to repo's remote?
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 107
Quarter:
Q4-2021
Description
My environment:
pulpcore - 3.14.5 pulp-rpm - 3.14.2 RHEL 8
I am fighting a weird performance issue. After a change to a remote that is attached to some repository (for example when a client certificate is changed), all further sync attempts for that repository involves reloading of the repository metadata and parsing it, even when no new content is available and no new repository version is created afterwards. This repeats for any subsequent sync attempt of that repo, and is a significant resource hog (eg. RHEL7 repos carry 30K+ packages). A sample task with progress reports looks like this:
{
"pulp_href": "/pulp/api/v3/tasks/a9ec4ba1-a804-483d-b9ae-02874a8f19ad/",
"pulp_created": "2021-09-15T08:43:25.974820Z",
"state": "running",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid": "4bd0de408bce49f2932790a4c6075e5e",
"started_at": "2021-09-15T09:03:54.732570Z",
"finished_at": null,
"error": null,
"worker": "/pulp/api/v3/workers/0c1e8308-37a8-43e3-8a9b-fb91ebdc18b6/",
"parent_task": null,
"child_tasks": [],
"task_group": null,
"progress_reports": [
{
"message": "Downloading Metadata Files",
"code": "sync.downloading.metadata",
"state": "completed",
"total": null,
"done": 11,
"suffix": null
},
{
"message": "Downloading Artifacts",
"code": "sync.downloading.artifacts",
"state": "running",
"total": null,
"done": 0,
"suffix": null
},
{
"message": "Associating Content",
"code": "associating.content",
"state": "running",
"total": null,
"done": 0,
"suffix": null
},
{
"message": "Parsed Modulemd",
"code": "sync.parsing.modulemds",
"state": "completed",
"total": 374,
"done": 374,
"suffix": null
},
{
"message": "Parsed Modulemd-defaults",
"code": "sync.parsing.modulemd_defaults",
"state": "completed",
"total": 45,
"done": 45,
"suffix": null
},
{
"message": "Parsed Packages",
"code": "sync.parsing.packages",
"state": "completed",
"total": 19122,
"done": 19122,
"suffix": null
},
{
"message": "Parsed Advisories",
"code": "sync.parsing.advisories",
"state": "completed",
"total": 1566,
"done": 1566,
"suffix": null
},
{
"message": "Parsed Comps",
"code": "sync.parsing.comps",
"state": "completed",
"total": 67,
"done": 67,
"suffix": null
}
],
"created_resources": [
null
],
"reserved_resources_record": [
"/pulp/api/v3/remotes/rpm/rpm/9b11994b-f7ea-4df3-a7ed-094f4b183a6c/",
"/pulp/api/v3/repositories/rpm/rpm/c610ead3-8517-47c4-9ed8-d9dc9cc731ee/"
]
},
Actions
Fix sync optimization when remote changes
closes: #9398 https://pulp.plan.io/issues/9398