Issue #5865
closedImprove the speed when syncing large repository
Description
There are some improvements that we can do to speed up the syncing process, such as:
- prevent unnecessary looping the xml metadata files when there is no new packages.
- prevent unnecessary looping the xml metadata files when purging the missing units.
Updated by hyu over 4 years ago
Added a pull request for this.
https://github.com/pulp/pulp_rpm/pull/1558
Not sure about the autotest, it seems that many tests have already been broken.
Updated by hyu over 4 years ago
Syncing a rhel 7 server rpms repository.
pulp-admin --username admin --password $pass rpm repo sync run --repo-id 0a32fe81-6f52-4fd2-92b1-2e617ce39f7a --force-full
Before patch:
Dec 10 19:11:43 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Parsing metadata.
Dec 10 19:11:46 hao-capsule-2 pulp: pulp.server.controllers.repository:INFO: [ccac130c] Fully resyncing due to use of force_full in config
Dec 10 19:11:46 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Downloading metadata files.
Dec 10 19:14:42 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Generating metadata databases.
Dec 10 19:20:58 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Determining which units need to be downloaded.
Dec 10 19:41:36 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [ccac130c] Removing missing units.
Dec 10 19:48:14 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Downloading additional units.
Dec 10 19:50:51 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Sync complete.
Dec 10 19:50:52 hao-capsule-2 pulp: celery.app.trace:INFO: [ccac130c] Task pulp.server.managers.repo.sync.sync[ccac130c-d512-49f6-8261-144efc515f10] succeeded in 2358.83260471s: <pulp.server.async.tasks.TaskResult object at 0x7fbbbe987bd0>
After patch:
Dec 10 18:31:18 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Parsing metadata.
Dec 10 18:31:21 hao-capsule-2 pulp: pulp.server.controllers.repository:INFO: [94218322] Fully resyncing due to use of force_full in config
Dec 10 18:31:21 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Downloading metadata files.
Dec 10 18:33:35 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Generating metadata databases.
Dec 10 18:39:37 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Determining which units need to be downloaded.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing rpms.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing drpms.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Downloading additional units.
Dec 10 18:57:31 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing erratums.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_groups.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_categorys.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_environments.
Dec 10 18:57:42 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Sync complete.
Dec 10 18:57:43 hao-capsule-2 pulp: celery.app.trace:INFO: [94218322] Task pulp.server.managers.repo.sync.sync[94218322-ef24-4d57-aa12-028138a7937b] succeeded in 1594.37815083s: <pulp.server.async.tasks.TaskResult object at 0x7f77a01be4d0>
Updated by dalley over 4 years ago
- Status changed from NEW to POST
- Assignee set to hyu
Updated by ttereshc about 4 years ago
- Description updated (diff)
- Status changed from POST to CLOSED - WONTFIX
@hyu thank you for this work. Apologies for not responding sooner.
Since the release of Pulp 3.0 GA, we're only merging critical bug fixes for Pulp 2.
Updated by bmbouter almost 4 years ago
- Tags Performance added
- Tags deleted (
Sync Performance)