Actions
Issue #3701
closed
Metadata recalculated even if not changes on huge repos like OL7/OL6 latest
Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
Hi,
seems that Pulp has a problem with large metadata files. For standard sized repos I got ```Skipped: Repository content has not changed since last publish``` but for OL7 and OL6 latest nothing like that is visible in logs and takes about 20-30 minutes to task finish (due download, analyze and metadata recreate). I didn't find any difference just metadata size in OL6/7. Any advice ?
pulp-admin -v rpm repo sync run --repo-id public-ol7-latest
Id: public-ol7-latest
Display Name: Oracle Linux 7 Latest (x86_64)
Description: Oracle Linux 7 Latest (x86_64)
Content Unit Counts:
Package Category: 12
Package Environment: 6
Package Group: 76
Package Langpacks: 1
Rpm: 21158
Notes:
Scratchpad:
Checksum Type: sha1
Importers:
Config:
Feed: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86
_64/
Max Downloads: 20
Remove Missing: True
Type Skip List: erratum, srpm, distribution
Validate: True
Id: yum_importer
Importer Type Id: yum_importer
Last Override Config:
Last Sync: 2018-05-24T09:47:35Z
Last Updated: 2018-05-16T20:26:07Z
Repo Id: public-ol7-latest
Scratchpad:
Repomd Revision: None
Distributors:
Auto Publish: True
Config:
Checksum Type: sha1
Generate Sqlite: False
Gpgkey: -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.5
(GNU/Linux)
mQENBEwtJWoBCACpiY2rGA6T0ceBi92X88/QclytVBjtDRohOVzs3pmIPh3
ULqsW
G323nmyKbKQBBSjh9OnuO9Y09VG8mzr/w9YV0Ix4cI9/HDTERZ2+TR5u+VN
n5J5h
yvwQSN/FEK6oH2+mz7O0yUNleN7UltR4MOEkHIoAhIvv+1AQQKN3OM8oalz
+3gv/
zz9rAoQczQzT7QWOtBrsRMZgBrKXY/TpJrpVSO3Hx8CnbhKGtLxCCrxZ5v7
hh1ht
3CRAr2+h5bDA9KP6vBZWKEs7NgcvtZFDY6EJc7AoApr3phX9hHE2+snTxe8
2DkFT
uA69C8wLyjPCoSy+tcaCqJKOZelNy9RN/WKRABEBAAG0RE9yYWNsZSBPU1M
gZ3Jv
dXAgKE9wZW4gU291cmNlIFNvZnR3YXJlIGdyb3VwKSA8YnVpbGRAb3NzLm9
yYWNs
ZS5jb20+iQE8BBMBAgAmAhsDBgsJCAcDAgQVAggDBBYCAwECHgECF4AFAlN
hkUEF
CSaOl9cACgkQcvl7dOxVHwNHFQf9G2ZI5ZH8T1VASvQteTyUR7uNgqXEbJh
i9zZO
7EO26+wzkj9EGLmH1COdUApZ1cINsYfGGgUJT5mfcRV3yYZbwc4AZJbJe0z
7C5Yu
ZLs5W0ryV4bzIqcWnVphIAOwmxMxIVGz8Cr1Dsyyal7ORgYzdfOppYetwtZ
+J+Wn
/oVgFkh9v19l/CltBkRh2SqgUZYfCoELo7hUzLNh7cw2av8rcSUKSH3ra9M
vpYfS
ANCnouzbgKix1gD6niT3pm1s5II3VuQ2vEcJunnoRFci9FzLXelTuL00Mvu
xERr7 Fsqm1/D2JfKDbE09qy5bLWrWaTM6zOFQKN7F2edY2uaukLT6/w==
=Djed -----END PGP PUBLIC KEY BLOCK-----
Http: False
Https: True
Relative URL: x86_64/oel/7/latest/qa
Skip: erratum, srpm, distribution
Distributor Type Id: yum_distributor
Id: yum_distributor
Last Override Config:
Last Publish: 2018-05-24T06:51:36Z
Last Updated: 2018-05-16T20:57:08Z
Repo Id: public-ol7-latest
Scratchpad:
Auto Publish: False
Config:
Generate Sqlite: False
Http: False
Https: True
Relative URL: x86_64/oel/7/latest/qa
Skip: erratum, srpm, distribution
Distributor Type Id: export_distributor
Id: export_distributor
Last Override Config:
Last Publish: None
Last Updated: 2018-05-16T20:26:07Z
Repo Id: public-ol7-latest
Scratchpad:
May 24 02:31:17 pulp-master pulp[2940]: pulp.server.event.http:INFO: [8738607c] {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/8738607c-237c-4a45-b98b-1d7472e54c8a/", "task_id": "8738607c-237c-4a45-b98b-1d7472e54c8a", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:sync"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:31:17Z", "traceback": null, "spawned_tasks": [], "progress_report": {}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068665cd154abeb79e5acf"}, "id": "5b068665cd154abeb79e5acf"}, "event_type": "repo.sync.start", "payload": {"repo_id": "public-ol7-latest"}}
May 24 02:31:17 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [8738607c] Starting new HTTP connection (1): localhost
May 24 02:31:17 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:17 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:18 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:18 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:18 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:18 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:19 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:19 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:19 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:19 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:19 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:20 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:20 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:20 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:20 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:21 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:22 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata files.
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (5): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/comps.xml.
May 24 02:31:22 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/updateinfo.xml.gz.
May 24 02:31:24 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/filelists.xml.gz.
May 24 02:31:25 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/primary.xml.gz.
May 24 02:31:49 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/other.xml.gz.
May 24 02:31:49 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Generating metadata databases.
May 24 02:36:06 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Determining which units need to be downloaded.
May 24 02:42:40 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading 0 RPMs.
May 24 02:44:54 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [8738607c] The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/
May 24 02:44:54 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.purge:INFO: [8738607c] Removing missing units.
May 24 02:47:27 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading additional units.
May 24 02:47:35 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Sync complete.
May 24 02:47:35 pulp-master pulp[2940]: pulp.server.event.http:INFO: [8738607c] (2940-32480) {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/8738607c-237c-4a45-b98b-1d7472e54c8a/", "task_id": "8738607c-237c-4a45-b98b-1d7472e54c8a", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:sync"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:31:17Z", "traceback": null, "spawned_tasks": [], "progress_report": {"yum_importer": {"content": {"items_total": 0, "state": "FINISHED", "error_details": [], "details": {"rpm_total": 0, "rpm_done": 0, "drpm_total": 0, "drpm_done": 0}, "size_total": 0, "size_left": 0, "items_left": 0}, "comps": {"state": "FINISHED"}, "purge_duplicates": {"state": "FINISHED"}, "distribution": {"items_total": 0, "state": "SKIPPED", "error_details": [], "items_left": 0}, "errata": {"state": "SKIPPED"}, "metadata": {"state": "FINISHED"}}}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068665cd154abeb79e5acf"}, "id": "5b068665cd154abeb79e5acf"}, "event_type": "repo.sync.finish", "payload": {"importer_id": "yum_importer", "exception": null, "repo_id": "public-ol7-latest", "traceback": null, "started": "2018-05-24T09:31:17Z", "_ns": "repo_sync_results", "completed": "2018-05-24T09:47:35Z", "importer_type_id": "yum_importer", "error_message": null, "summary": {"content": {"state": "FINISHED"}, "comps": {"state": "FINISHED"}, "purge_duplicates": {"state": "FINISHED"}, "distribution": {"state": "SKIPPED"}, "errata": {"state": "SKIPPED"}, "metadata": {"state": "FINISHED"}}, "added_count": 0, "result": "success", "updated_count": 95, "details": {"content": {"size_total": 0, "items_left": 0, "items_total": 0, "state": "FINISHED", "size_left": 0, "details": {"rpm_total": 0, "rpm_done": 0, "drpm_total": 0, "drpm_done": 0}, "error_details": []}, "comps": {"state": "FINISHED"}, "purge_duplicates": {"state": "FINISHED"}, "distribution": {"ite
May 24 02:47:35 pulp-master pulp[2940]: pulp.server.event.http:INFO: [8738607c] (2940-32480) ms_total": 0, "state": "SKIPPED", "error_details": [], "items_left": 0}, "errata": {"state": "SKIPPED"}, "metadata": {"state": "FINISHED"}}, "id": "5b068a374949c40b7c67a33a", "removed_count": 0}}
May 24 02:47:35 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [8738607c] Starting new HTTP connection (1): localhost
May 24 02:47:35 pulp-master pulp[1303]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[e0385edc-189e-41f0-acdd-3f736dd44ec9]
May 24 02:47:35 pulp-master pulp[2940]: celery.app.trace:INFO: [8738607c] Task pulp.server.managers.repo.sync.sync[8738607c-237c-4a45-b98b-1d7472e54c8a] succeeded in 977.64816999s: <pulp.server.async.tasks.TaskResult object at 0x4184fd0>
May 24 02:47:35 pulp-master pulp[2940]: celery.app.trace:INFO: [28d45b3d] Task pulp.server.async.tasks._release_resource[28d45b3d-9bb1-414e-bd00-bd1c942592e9] succeeded in 0.00138283002889s: None
May 24 02:47:35 pulp-master pulp[1367]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d]
May 24 02:47:35 pulp-master pulp[3050]: celery.app.trace:INFO: [e0385edc] Task pulp.server.async.tasks._queue_reserved_task[e0385edc-189e-41f0-acdd-3f736dd44ec9] succeeded in 0.0156079180306s: None
May 24 02:47:35 pulp-master pulp[1367]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[50437933-2545-45ff-8f75-1303cfcc495d]
May 24 02:47:35 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d/", "task_id": "cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:publish"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:47:35Z", "traceback": null, "spawned_tasks": [], "progress_report": {}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068a37cd154abeb79e5ad0"}, "id": "5b068a37cd154abeb79e5ad0"}, "event_type": "repo.publish.start", "payload": {"repo_id": "public-ol7-latest", "distributor_id": "yum_distributor"}}
May 24 02:47:35 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [cdc55e7d] Starting new HTTP connection (1): localhost
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d/", "task_id": "cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:publish"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:47:35Z", "traceback": null, "spawned_tasks": [], "progress_report": {"yum_distributor": [{"num_success": 1, "description": "Copying files", "step_type": "save_tar", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "52f9feaf-3783-44a3-9ae0-136c58f6134c", "num_processed": 1}, {"num_success": 1, "description": "Initializing repo metadata", "step_type": "initialize_repo_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "f4a5eda2-da6d-493b-91d0-4dec63ccee4d", "num_processed": 1}, {"num_success": 0, "description": "Publishing Distribution files", "step_type": "distribution", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "2104e32c-895b-400f-8c70-cc8664c41bb5", "num_processed": 0}, {"num_success": 0, "description": "Publishing RPMs", "step_type": "rpms", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "58f597e6-d530-45e0-af8c-fc664a2da594", "num_processed": 0}, {"num_success": 0, "description": "Publishing Delta RPMs", "step_type": "drpms", "items_total": 1, "state": "SKIPPED", "error_details": [], "details": "", "num_failures": 0, "step_id": "e234e535-8ff6-4cae-8d92-635c4ef19439", "num_processed": 0}, {"num_success": 0, "description": "Publishing Errata", "step_type": "errata", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "83e0d352-5a0b-42bd-b15c-ccbfcf1f9717", "num_processed": 0}, {"num_success": 95, "description
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) ": "Publishing Comps file", "step_type": "comps", "items_total": 95, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "ac51c974-d7cc-46fc-bf21-63f5e730e004", "num_processed": 95}, {"num_success": 0, "description": "Publishing Metadata.", "step_type": "metadata", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "24aa03ef-d1d7-491f-96af-eb82fd6cae76", "num_processed": 0}, {"num_success": 1, "description": "Closing repo metadata", "step_type": "close_repo_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "14d7da15-47c6-4b51-88e5-c564ae317334", "num_processed": 1}, {"num_success": 0, "description": "Generating sqlite files", "step_type": "generate sqlite", "items_total": 1, "state": "SKIPPED", "error_details": [], "details": "", "num_failures": 0, "step_id": "15b91f9f-a459-4c9c-a720-e0831a37644b", "num_processed": 0}, {"num_success": 1, "description": "Removing old repodata", "step_type": "remove_old_repodata", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "309786a9-80ed-492d-97af-69a907f7bdd2", "num_processed": 1}, {"num_success": 0, "description": "Generating HTML files", "step_type": "repoview", "items_total": 1, "state": "SKIPPED", "error_details": [], "details": "", "num_failures": 0, "step_id": "d9062311-59f8-4cb8-b7a9-b2c69db6f9a8", "num_processed": 0}, {"num_success": 1, "description": "Publishing files to web", "step_type": "publish_directory", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "320ea7e8-adc3-4e27-ba0c-b87e4c8acd46", "num_processed": 1}, {"num_success": 1, "description": "Writing Listings File", "step_type": "initialize_repo_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "3c258b0f-8245-402d-
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) 8647-f2ef0867ac7b", "num_processed": 1}]}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068a37cd154abeb79e5ad0"}, "id": "5b068a37cd154abeb79e5ad0"}, "event_type": "repo.publish.finish", "payload": {"exception": null, "repo_id": "public-ol7-latest", "traceback": null, "started": "2018-05-24T09:47:35Z", "_ns": "repo_publish_results", "completed": {"$date": 1527155593262}, "error_message": null, "distributor_type_id": "yum_distributor", "distributor_id": "yum_distributor", "summary": {"generate sqlite": "SKIPPED", "initialize_repo_metadata": "FINISHED", "remove_old_repodata": "FINISHED", "rpms": "FINISHED", "repoview": "SKIPPED", "close_repo_metadata": "FINISHED", "drpms": "SKIPPED", "comps": "FINISHED", "distribution": "FINISHED", "save_tar": "FINISHED", "publish_directory": "FINISHED", "errata": "FINISHED", "metadata": "FINISHED"}, "result": "success", "id": "5b068b894949c40b7c67a33b", "details": [{"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Copying files", "num_failures": 0, "step_id": "52f9feaf-3783-44a3-9ae0-136c58f6134c", "step_type": "save_tar", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Initializing repo metadata", "num_failures": 0, "step_id": "f4a5eda2-da6d-493b-91d0-4dec63ccee4d", "step_type": "initialize_repo_metadata", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing Distribution files", "num_failures": 0, "step_id": "2104e32c-895b-400f-8c70-cc8664c41bb5", "step_type": "distribution", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing RPMs", "num_failures": 0, "step_id": "58f597e6-d530-45e0-af8c-fc664
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) a2da594", "step_type": "rpms", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "SKIPPED", "num_success": 0, "error_details": [], "description": "Publishing Delta RPMs", "num_failures": 0, "step_id": "e234e535-8ff6-4cae-8d92-635c4ef19439", "step_type": "drpms", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing Errata", "num_failures": 0, "step_id": "83e0d352-5a0b-42bd-b15c-ccbfcf1f9717", "step_type": "errata", "details": ""}, {"num_processed": 95, "items_total": 95, "state": "FINISHED", "num_success": 95, "error_details": [], "description": "Publishing Comps file", "num_failures": 0, "step_id": "ac51c974-d7cc-46fc-bf21-63f5e730e004", "step_type": "comps", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing Metadata.", "num_failures": 0, "step_id": "24aa03ef-d1d7-491f-96af-eb82fd6cae76", "step_type": "metadata", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Closing repo metadata", "num_failures": 0, "step_id": "14d7da15-47c6-4b51-88e5-c564ae317334", "step_type": "close_repo_metadata", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "SKIPPED", "num_success": 0, "error_details": [], "description": "Generating sqlite files", "num_failures": 0, "step_id": "15b91f9f-a459-4c9c-a720-e0831a37644b", "step_type": "generate sqlite", "details": ""}, {"num_processed": 1, "items_total": 0, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Removing old repodata", "num_failures": 0, "step_id": "309786a9-80ed-492d-97af-69a907f7bdd2", "step_type": "remove_old_repodata", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "SKIPPED", "num_success": 0, "error_details": [], "description": "Generating HTML files", "num_failures": 0, "step_id"
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) : "d9062311-59f8-4cb8-b7a9-b2c69db6f9a8", "step_type": "repoview", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Publishing files to web", "num_failures": 0, "step_id": "320ea7e8-adc3-4e27-ba0c-b87e4c8acd46", "step_type": "publish_directory", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Writing Listings File", "num_failures": 0, "step_id": "3c258b0f-8245-402d-8647-f2ef0867ac7b", "step_type": "initialize_repo_metadata", "details": ""}]}}
May 24 02:53:13 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [cdc55e7d] Starting new HTTP connection (1): localhost
May 24 02:53:13 pulp-master pulp[2940]: celery.app.trace:INFO: [cdc55e7d] Task pulp.server.managers.repo.publish.publish[cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d] succeeded in 337.718775363s: {'exception': None, 'repo_id': 'public-ol7-latest', 'traceback': None, 'started': '2018-05-24T09:47:35Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2018, 5, 24, 9, 53, 13, 262891, tzinfo=<isodate.tzinfo.Utc object at 0x1a06a90>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'repoview': 'SKIPPED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'save_tar': 'FINISHED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5b068b894949c40b7c67a33b', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Copying files', 'num_failures': 0, 'step_id': '52f9feaf-3783-44a3-9ae0-136c58f6134c', 'step_type': 'save_tar', 'details', ...}]}
Updated by dkliban@redhat.com over 5 years ago
- Project changed from Pulp to RPM Support
There are a couple of things that come to mind:
The repository got republished and the repomd file has a different version. Pulp assumes that the metadata changed and needs to be processed.
The importer config was updated after the last sync. In this case Pulp does not perform any optimizations for the sync the follows the importer config update.
Could either of those happened here?
Updated by vaclav.adamec@suchy-zleb.cz over 5 years ago
No updates to configuration and based on timestamps of http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/ no changes at feed side. It tooks about 20-30 minutes to run sync everytime, other repos are ok, I can run it several times and get skipped, no change. But OL6 and OL7 latest refresh metadata everytime.
Updated by ipanova@redhat.com over 5 years ago
it behaves like this because repomd revision tag is missing completely.
The code checks previous and current revision and based on the number it can tell whether the content changed on the remote repo or not.
In this case we cannot tell what happened with the remote content that's why we parse metadata with every sync.
previous_revision = None
current_revision = None
skip_sync_steps = not force_full_sync and \
0 < self.current_revision <= previous_revision and \
(self.download_deferred or not missing_units)
0 < self.current_revision <= previous_revision
In [9]: 0 < None <= None
Out [9]: False
which makes overall result of skip_sync_steps False.
Updated by vaclav.adamec@suchy-zleb.cz over 5 years ago
H, you are right, I missed that. Ok, try to open ticket to Oracle, wish me luck :-)
Actions
.