Issue #5865
closed
Improve the speed when syncing large repository
Description
There are some improvements that we can do to speed up the syncing process, such as:
- prevent unnecessary looping the xml metadata files when there is no new packages.
- prevent unnecessary looping the xml metadata files when purging the missing units.
Syncing a rhel 7 server rpms repository.
pulp-admin --username admin --password $pass rpm repo sync run --repo-id 0a32fe81-6f52-4fd2-92b1-2e617ce39f7a --force-full
Before patch:
Dec 10 19:11:43 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Parsing metadata.
Dec 10 19:11:46 hao-capsule-2 pulp: pulp.server.controllers.repository:INFO: [ccac130c] Fully resyncing due to use of force_full in config
Dec 10 19:11:46 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Downloading metadata files.
Dec 10 19:14:42 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Generating metadata databases.
Dec 10 19:20:58 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Determining which units need to be downloaded.
Dec 10 19:41:36 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [ccac130c] Removing missing units.
Dec 10 19:48:14 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Downloading additional units.
Dec 10 19:50:51 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Sync complete.
Dec 10 19:50:52 hao-capsule-2 pulp: celery.app.trace:INFO: [ccac130c] Task pulp.server.managers.repo.sync.sync[ccac130c-d512-49f6-8261-144efc515f10] succeeded in 2358.83260471s: <pulp.server.async.tasks.TaskResult object at 0x7fbbbe987bd0>
After patch:
Dec 10 18:31:18 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Parsing metadata.
Dec 10 18:31:21 hao-capsule-2 pulp: pulp.server.controllers.repository:INFO: [94218322] Fully resyncing due to use of force_full in config
Dec 10 18:31:21 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Downloading metadata files.
Dec 10 18:33:35 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Generating metadata databases.
Dec 10 18:39:37 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Determining which units need to be downloaded.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing rpms.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing drpms.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Downloading additional units.
Dec 10 18:57:31 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing erratums.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_groups.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_categorys.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_environments.
Dec 10 18:57:42 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Sync complete.
Dec 10 18:57:43 hao-capsule-2 pulp: celery.app.trace:INFO: [94218322] Task pulp.server.managers.repo.sync.sync[94218322-ef24-4d57-aa12-028138a7937b] succeeded in 1594.37815083s: <pulp.server.async.tasks.TaskResult object at 0x7f77a01be4d0>
- Status changed from NEW to POST
- Assignee set to hyu
- Description updated (diff)
- Status changed from POST to CLOSED - WONTFIX
@hyu thank you for this work. Apologies for not responding sooner.
Since the release of Pulp 3.0 GA, we're only merging critical bug fixes for Pulp 2.
- Tags Performance added
- Tags deleted (
Sync Performance)
Also available in: Atom
PDF
.