Project

Profile

Help

Issue #5865

closed

Improve the speed when syncing large repository

Added by hyu over 4 years ago. Updated almost 4 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.21.0
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Performance, Pulp 2
Sprint:
Quarter:

Description

There are some improvements that we can do to speed up the syncing process, such as:

- prevent unnecessary looping the xml metadata files when there is no new packages.
- prevent unnecessary looping the xml metadata files when purging the missing units.

Actions #1

Updated by hyu over 4 years ago

Added a pull request for this.

https://github.com/pulp/pulp_rpm/pull/1558

Not sure about the autotest, it seems that many tests have already been broken.

Actions #2

Updated by hyu over 4 years ago

Syncing a rhel 7 server rpms repository.

pulp-admin --username admin --password $pass rpm repo sync run --repo-id 0a32fe81-6f52-4fd2-92b1-2e617ce39f7a --force-full

Before patch:
Dec 10 19:11:43 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Parsing metadata.
Dec 10 19:11:46 hao-capsule-2 pulp: pulp.server.controllers.repository:INFO: [ccac130c] Fully resyncing due to use of force_full in config
Dec 10 19:11:46 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Downloading metadata files.
Dec 10 19:14:42 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Generating metadata databases.
Dec 10 19:20:58 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Determining which units need to be downloaded.
Dec 10 19:41:36 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [ccac130c] Removing missing units.
Dec 10 19:48:14 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Downloading additional units.
Dec 10 19:50:51 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [ccac130c] Sync complete.
Dec 10 19:50:52 hao-capsule-2 pulp: celery.app.trace:INFO: [ccac130c] Task pulp.server.managers.repo.sync.sync[ccac130c-d512-49f6-8261-144efc515f10] succeeded in 2358.83260471s: <pulp.server.async.tasks.TaskResult object at 0x7fbbbe987bd0>

After patch:
Dec 10 18:31:18 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Parsing metadata.
Dec 10 18:31:21 hao-capsule-2 pulp: pulp.server.controllers.repository:INFO: [94218322] Fully resyncing due to use of force_full in config
Dec 10 18:31:21 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Downloading metadata files.
Dec 10 18:33:35 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Generating metadata databases.
Dec 10 18:39:37 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Determining which units need to be downloaded.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing rpms.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing drpms.
Dec 10 18:54:55 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Downloading additional units.
Dec 10 18:57:31 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing erratums.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_groups.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_categorys.
Dec 10 18:57:33 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [94218322] Removing missing package_environments.
Dec 10 18:57:42 hao-capsule-2 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [94218322] Sync complete.
Dec 10 18:57:43 hao-capsule-2 pulp: celery.app.trace:INFO: [94218322] Task pulp.server.managers.repo.sync.sync[94218322-ef24-4d57-aa12-028138a7937b] succeeded in 1594.37815083s: <pulp.server.async.tasks.TaskResult object at 0x7f77a01be4d0>

Actions #3

Updated by dalley over 4 years ago

  • Status changed from NEW to POST
  • Assignee set to hyu
Actions #4

Updated by dalley over 4 years ago

  • Tags Pulp 2 added
Actions #5

Updated by ttereshc about 4 years ago

  • Description updated (diff)
  • Status changed from POST to CLOSED - WONTFIX

@hyu thank you for this work. Apologies for not responding sooner.

Since the release of Pulp 3.0 GA, we're only merging critical bug fixes for Pulp 2.

Actions #6

Updated by bmbouter almost 4 years ago

  • Tags Performance added
  • Tags deleted (Sync Performance)

Also available in: Atom PDF