Project

Profile

Help

Issue #3701

closed

Metadata recalculated even if not changes on huge repos like OL7/OL6 latest

Added by vaclav.adamec@suchy-zleb.cz almost 6 years ago. Updated almost 5 years ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Hi,
seems that Pulp has a problem with large metadata files. For standard sized repos I got ```Skipped: Repository content has not changed since last publish``` but for OL7 and OL6 latest nothing like that is visible in logs and takes about 20-30 minutes to task finish (due download, analyze and metadata recreate). I didn't find any difference just metadata size in OL6/7. Any advice ?

pulp-admin -v rpm repo sync run --repo-id public-ol7-latest
Id:                   public-ol7-latest
Display Name:         Oracle Linux 7 Latest (x86_64)
Description:          Oracle Linux 7 Latest (x86_64)
Content Unit Counts:
  Package Category:    12
  Package Environment: 6
  Package Group:       76
  Package Langpacks:   1
  Rpm:                 21158
Notes:
Scratchpad:
  Checksum Type: sha1
Importers:
  Config:
    Feed:           http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86
                    _64/
    Max Downloads:  20
    Remove Missing: True
    Type Skip List: erratum, srpm, distribution
    Validate:       True
  Id:                   yum_importer
  Importer Type Id:     yum_importer
  Last Override Config:
  Last Sync:            2018-05-24T09:47:35Z
  Last Updated:         2018-05-16T20:26:07Z
  Repo Id:              public-ol7-latest
  Scratchpad:
    Repomd Revision: None
Distributors:
  Auto Publish:         True
  Config:
    Checksum Type:   sha1
    Generate Sqlite: False
    Gpgkey:          -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.5
                     (GNU/Linux)
                     mQENBEwtJWoBCACpiY2rGA6T0ceBi92X88/QclytVBjtDRohOVzs3pmIPh3
                     ULqsW
                     G323nmyKbKQBBSjh9OnuO9Y09VG8mzr/w9YV0Ix4cI9/HDTERZ2+TR5u+VN
                     n5J5h
                     yvwQSN/FEK6oH2+mz7O0yUNleN7UltR4MOEkHIoAhIvv+1AQQKN3OM8oalz
                     +3gv/
                     zz9rAoQczQzT7QWOtBrsRMZgBrKXY/TpJrpVSO3Hx8CnbhKGtLxCCrxZ5v7
                     hh1ht
                     3CRAr2+h5bDA9KP6vBZWKEs7NgcvtZFDY6EJc7AoApr3phX9hHE2+snTxe8
                     2DkFT
                     uA69C8wLyjPCoSy+tcaCqJKOZelNy9RN/WKRABEBAAG0RE9yYWNsZSBPU1M
                     gZ3Jv
                     dXAgKE9wZW4gU291cmNlIFNvZnR3YXJlIGdyb3VwKSA8YnVpbGRAb3NzLm9
                     yYWNs
                     ZS5jb20+iQE8BBMBAgAmAhsDBgsJCAcDAgQVAggDBBYCAwECHgECF4AFAlN
                     hkUEF
                     CSaOl9cACgkQcvl7dOxVHwNHFQf9G2ZI5ZH8T1VASvQteTyUR7uNgqXEbJh
                     i9zZO
                     7EO26+wzkj9EGLmH1COdUApZ1cINsYfGGgUJT5mfcRV3yYZbwc4AZJbJe0z
                     7C5Yu
                     ZLs5W0ryV4bzIqcWnVphIAOwmxMxIVGz8Cr1Dsyyal7ORgYzdfOppYetwtZ
                     +J+Wn
                     /oVgFkh9v19l/CltBkRh2SqgUZYfCoELo7hUzLNh7cw2av8rcSUKSH3ra9M
                     vpYfS
                     ANCnouzbgKix1gD6niT3pm1s5II3VuQ2vEcJunnoRFci9FzLXelTuL00Mvu
                     xERr7 Fsqm1/D2JfKDbE09qy5bLWrWaTM6zOFQKN7F2edY2uaukLT6/w==
                     =Djed -----END PGP PUBLIC KEY BLOCK-----
    Http:            False
    Https:           True
    Relative URL:    x86_64/oel/7/latest/qa
    Skip:            erratum, srpm, distribution
  Distributor Type Id:  yum_distributor
  Id:                   yum_distributor
  Last Override Config:
  Last Publish:         2018-05-24T06:51:36Z
  Last Updated:         2018-05-16T20:57:08Z
  Repo Id:              public-ol7-latest
  Scratchpad:
  Auto Publish:         False
  Config:
    Generate Sqlite: False
    Http:            False
    Https:           True
    Relative URL:    x86_64/oel/7/latest/qa
    Skip:            erratum, srpm, distribution
  Distributor Type Id:  export_distributor
  Id:                   export_distributor
  Last Override Config:
  Last Publish:         None
  Last Updated:         2018-05-16T20:26:07Z
  Repo Id:              public-ol7-latest
  Scratchpad:
May 24 02:31:17 pulp-master pulp[2940]: pulp.server.event.http:INFO: [8738607c] {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/8738607c-237c-4a45-b98b-1d7472e54c8a/", "task_id": "8738607c-237c-4a45-b98b-1d7472e54c8a", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:sync"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:31:17Z", "traceback": null, "spawned_tasks": [], "progress_report": {}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068665cd154abeb79e5acf"}, "id": "5b068665cd154abeb79e5acf"}, "event_type": "repo.sync.start", "payload": {"repo_id": "public-ol7-latest"}}
May 24 02:31:17 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [8738607c] Starting new HTTP connection (1): localhost
May 24 02:31:17 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:17 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:18 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:18 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:18 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:18 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:19 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:19 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:19 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:19 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:19 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:20 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:20 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata from http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/.
May 24 02:31:20 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): public-yum.oracle.com
May 24 02:31:20 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/repomd.xml.
May 24 02:31:21 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Parsing metadata.
May 24 02:31:22 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading metadata files.
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (5): public-yum.oracle.com
May 24 02:31:22 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/comps.xml.
May 24 02:31:22 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/updateinfo.xml.gz.
May 24 02:31:24 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/filelists.xml.gz.
May 24 02:31:25 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/primary.xml.gz.
May 24 02:31:49 pulp-master pulp[2940]: nectar.downloaders.threaded:INFO: Download succeeded: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/other.xml.gz.

May 24 02:31:49 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Generating metadata databases.
May 24 02:36:06 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Determining which units need to be downloaded.
May 24 02:42:40 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading 0 RPMs.

May 24 02:44:54 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [8738607c] The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/

May 24 02:44:54 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.purge:INFO: [8738607c] Removing missing units.
May 24 02:47:27 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Downloading additional units.
May 24 02:47:35 pulp-master pulp[2940]: pulp_rpm.plugins.importers.yum.sync:INFO: [8738607c] Sync complete.

May 24 02:47:35 pulp-master pulp[2940]: pulp.server.event.http:INFO: [8738607c] (2940-32480) {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/8738607c-237c-4a45-b98b-1d7472e54c8a/", "task_id": "8738607c-237c-4a45-b98b-1d7472e54c8a", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:sync"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:31:17Z", "traceback": null, "spawned_tasks": [], "progress_report": {"yum_importer": {"content": {"items_total": 0, "state": "FINISHED", "error_details": [], "details": {"rpm_total": 0, "rpm_done": 0, "drpm_total": 0, "drpm_done": 0}, "size_total": 0, "size_left": 0, "items_left": 0}, "comps": {"state": "FINISHED"}, "purge_duplicates": {"state": "FINISHED"}, "distribution": {"items_total": 0, "state": "SKIPPED", "error_details": [], "items_left": 0}, "errata": {"state": "SKIPPED"}, "metadata": {"state": "FINISHED"}}}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068665cd154abeb79e5acf"}, "id": "5b068665cd154abeb79e5acf"}, "event_type": "repo.sync.finish", "payload": {"importer_id": "yum_importer", "exception": null, "repo_id": "public-ol7-latest", "traceback": null, "started": "2018-05-24T09:31:17Z", "_ns": "repo_sync_results", "completed": "2018-05-24T09:47:35Z", "importer_type_id": "yum_importer", "error_message": null, "summary": {"content": {"state": "FINISHED"}, "comps": {"state": "FINISHED"}, "purge_duplicates": {"state": "FINISHED"}, "distribution": {"state": "SKIPPED"}, "errata": {"state": "SKIPPED"}, "metadata": {"state": "FINISHED"}}, "added_count": 0, "result": "success", "updated_count": 95, "details": {"content": {"size_total": 0, "items_left": 0, "items_total": 0, "state": "FINISHED", "size_left": 0, "details": {"rpm_total": 0, "rpm_done": 0, "drpm_total": 0, "drpm_done": 0}, "error_details": []}, "comps": {"state": "FINISHED"}, "purge_duplicates": {"state": "FINISHED"}, "distribution": {"ite
May 24 02:47:35 pulp-master pulp[2940]: pulp.server.event.http:INFO: [8738607c] (2940-32480) ms_total": 0, "state": "SKIPPED", "error_details": [], "items_left": 0}, "errata": {"state": "SKIPPED"}, "metadata": {"state": "FINISHED"}}, "id": "5b068a374949c40b7c67a33a", "removed_count": 0}}
May 24 02:47:35 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [8738607c] Starting new HTTP connection (1): localhost
May 24 02:47:35 pulp-master pulp[1303]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[e0385edc-189e-41f0-acdd-3f736dd44ec9]
May 24 02:47:35 pulp-master pulp[2940]: celery.app.trace:INFO: [8738607c] Task pulp.server.managers.repo.sync.sync[8738607c-237c-4a45-b98b-1d7472e54c8a] succeeded in 977.64816999s: <pulp.server.async.tasks.TaskResult object at 0x4184fd0>
May 24 02:47:35 pulp-master pulp[2940]: celery.app.trace:INFO: [28d45b3d] Task pulp.server.async.tasks._release_resource[28d45b3d-9bb1-414e-bd00-bd1c942592e9] succeeded in 0.00138283002889s: None
May 24 02:47:35 pulp-master pulp[1367]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d]
May 24 02:47:35 pulp-master pulp[3050]: celery.app.trace:INFO: [e0385edc] Task pulp.server.async.tasks._queue_reserved_task[e0385edc-189e-41f0-acdd-3f736dd44ec9] succeeded in 0.0156079180306s: None
May 24 02:47:35 pulp-master pulp[1367]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[50437933-2545-45ff-8f75-1303cfcc495d]
May 24 02:47:35 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d/", "task_id": "cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:publish"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:47:35Z", "traceback": null, "spawned_tasks": [], "progress_report": {}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068a37cd154abeb79e5ad0"}, "id": "5b068a37cd154abeb79e5ad0"}, "event_type": "repo.publish.start", "payload": {"repo_id": "public-ol7-latest", "distributor_id": "yum_distributor"}}
May 24 02:47:35 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [cdc55e7d] Starting new HTTP connection (1): localhost
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d/", "task_id": "cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d", "tags": ["pulp:repository:public-ol7-latest", "pulp:action:publish"], "finish_time": null, "_ns": "task_status", "start_time": "2018-05-24T09:47:35Z", "traceback": null, "spawned_tasks": [], "progress_report": {"yum_distributor": [{"num_success": 1, "description": "Copying files", "step_type": "save_tar", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "52f9feaf-3783-44a3-9ae0-136c58f6134c", "num_processed": 1}, {"num_success": 1, "description": "Initializing repo metadata", "step_type": "initialize_repo_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "f4a5eda2-da6d-493b-91d0-4dec63ccee4d", "num_processed": 1}, {"num_success": 0, "description": "Publishing Distribution files", "step_type": "distribution", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "2104e32c-895b-400f-8c70-cc8664c41bb5", "num_processed": 0}, {"num_success": 0, "description": "Publishing RPMs", "step_type": "rpms", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "58f597e6-d530-45e0-af8c-fc664a2da594", "num_processed": 0}, {"num_success": 0, "description": "Publishing Delta RPMs", "step_type": "drpms", "items_total": 1, "state": "SKIPPED", "error_details": [], "details": "", "num_failures": 0, "step_id": "e234e535-8ff6-4cae-8d92-635c4ef19439", "num_processed": 0}, {"num_success": 0, "description": "Publishing Errata", "step_type": "errata", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "83e0d352-5a0b-42bd-b15c-ccbfcf1f9717", "num_processed": 0}, {"num_success": 95, "description
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) ": "Publishing Comps file", "step_type": "comps", "items_total": 95, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "ac51c974-d7cc-46fc-bf21-63f5e730e004", "num_processed": 95}, {"num_success": 0, "description": "Publishing Metadata.", "step_type": "metadata", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "24aa03ef-d1d7-491f-96af-eb82fd6cae76", "num_processed": 0}, {"num_success": 1, "description": "Closing repo metadata", "step_type": "close_repo_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "14d7da15-47c6-4b51-88e5-c564ae317334", "num_processed": 1}, {"num_success": 0, "description": "Generating sqlite files", "step_type": "generate sqlite", "items_total": 1, "state": "SKIPPED", "error_details": [], "details": "", "num_failures": 0, "step_id": "15b91f9f-a459-4c9c-a720-e0831a37644b", "num_processed": 0}, {"num_success": 1, "description": "Removing old repodata", "step_type": "remove_old_repodata", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "309786a9-80ed-492d-97af-69a907f7bdd2", "num_processed": 1}, {"num_success": 0, "description": "Generating HTML files", "step_type": "repoview", "items_total": 1, "state": "SKIPPED", "error_details": [], "details": "", "num_failures": 0, "step_id": "d9062311-59f8-4cb8-b7a9-b2c69db6f9a8", "num_processed": 0}, {"num_success": 1, "description": "Publishing files to web", "step_type": "publish_directory", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "320ea7e8-adc3-4e27-ba0c-b87e4c8acd46", "num_processed": 1}, {"num_success": 1, "description": "Writing Listings File", "step_type": "initialize_repo_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "3c258b0f-8245-402d-
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) 8647-f2ef0867ac7b", "num_processed": 1}]}, "state": "running", "worker_name": "reserved_resource_worker-1@pulp-master", "result": null, "error": null, "_id": {"$oid": "5b068a37cd154abeb79e5ad0"}, "id": "5b068a37cd154abeb79e5ad0"}, "event_type": "repo.publish.finish", "payload": {"exception": null, "repo_id": "public-ol7-latest", "traceback": null, "started": "2018-05-24T09:47:35Z", "_ns": "repo_publish_results", "completed": {"$date": 1527155593262}, "error_message": null, "distributor_type_id": "yum_distributor", "distributor_id": "yum_distributor", "summary": {"generate sqlite": "SKIPPED", "initialize_repo_metadata": "FINISHED", "remove_old_repodata": "FINISHED", "rpms": "FINISHED", "repoview": "SKIPPED", "close_repo_metadata": "FINISHED", "drpms": "SKIPPED", "comps": "FINISHED", "distribution": "FINISHED", "save_tar": "FINISHED", "publish_directory": "FINISHED", "errata": "FINISHED", "metadata": "FINISHED"}, "result": "success", "id": "5b068b894949c40b7c67a33b", "details": [{"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Copying files", "num_failures": 0, "step_id": "52f9feaf-3783-44a3-9ae0-136c58f6134c", "step_type": "save_tar", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Initializing repo metadata", "num_failures": 0, "step_id": "f4a5eda2-da6d-493b-91d0-4dec63ccee4d", "step_type": "initialize_repo_metadata", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing Distribution files", "num_failures": 0, "step_id": "2104e32c-895b-400f-8c70-cc8664c41bb5", "step_type": "distribution", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing RPMs", "num_failures": 0, "step_id": "58f597e6-d530-45e0-af8c-fc664
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) a2da594", "step_type": "rpms", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "SKIPPED", "num_success": 0, "error_details": [], "description": "Publishing Delta RPMs", "num_failures": 0, "step_id": "e234e535-8ff6-4cae-8d92-635c4ef19439", "step_type": "drpms", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing Errata", "num_failures": 0, "step_id": "83e0d352-5a0b-42bd-b15c-ccbfcf1f9717", "step_type": "errata", "details": ""}, {"num_processed": 95, "items_total": 95, "state": "FINISHED", "num_success": 95, "error_details": [], "description": "Publishing Comps file", "num_failures": 0, "step_id": "ac51c974-d7cc-46fc-bf21-63f5e730e004", "step_type": "comps", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Publishing Metadata.", "num_failures": 0, "step_id": "24aa03ef-d1d7-491f-96af-eb82fd6cae76", "step_type": "metadata", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Closing repo metadata", "num_failures": 0, "step_id": "14d7da15-47c6-4b51-88e5-c564ae317334", "step_type": "close_repo_metadata", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "SKIPPED", "num_success": 0, "error_details": [], "description": "Generating sqlite files", "num_failures": 0, "step_id": "15b91f9f-a459-4c9c-a720-e0831a37644b", "step_type": "generate sqlite", "details": ""}, {"num_processed": 1, "items_total": 0, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Removing old repodata", "num_failures": 0, "step_id": "309786a9-80ed-492d-97af-69a907f7bdd2", "step_type": "remove_old_repodata", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "SKIPPED", "num_success": 0, "error_details": [], "description": "Generating HTML files", "num_failures": 0, "step_id"
May 24 02:53:13 pulp-master pulp[2940]: pulp.server.event.http:INFO: [cdc55e7d] (2940-32480) : "d9062311-59f8-4cb8-b7a9-b2c69db6f9a8", "step_type": "repoview", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Publishing files to web", "num_failures": 0, "step_id": "320ea7e8-adc3-4e27-ba0c-b87e4c8acd46", "step_type": "publish_directory", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Writing Listings File", "num_failures": 0, "step_id": "3c258b0f-8245-402d-8647-f2ef0867ac7b", "step_type": "initialize_repo_metadata", "details": ""}]}}
May 24 02:53:13 pulp-master pulp[2940]: requests.packages.urllib3.connectionpool:INFO: [cdc55e7d] Starting new HTTP connection (1): localhost
May 24 02:53:13 pulp-master pulp[2940]: celery.app.trace:INFO: [cdc55e7d] Task pulp.server.managers.repo.publish.publish[cdc55e7d-fc82-4c98-9f12-8b83cedd3e3d] succeeded in 337.718775363s: {'exception': None, 'repo_id': 'public-ol7-latest', 'traceback': None, 'started': '2018-05-24T09:47:35Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2018, 5, 24, 9, 53, 13, 262891, tzinfo=<isodate.tzinfo.Utc object at 0x1a06a90>), 'error_message': None, 'distributor_type_id': 'yum_distributor', 'distributor_id': 'yum_distributor', 'summary': {'generate sqlite': 'SKIPPED', 'initialize_repo_metadata': 'FINISHED', 'remove_old_repodata': 'FINISHED', 'rpms': 'FINISHED', 'repoview': 'SKIPPED', 'close_repo_metadata': 'FINISHED', 'drpms': 'SKIPPED', 'comps': 'FINISHED', 'distribution': 'FINISHED', 'save_tar': 'FINISHED', 'publish_directory': 'FINISHED', 'errata': 'FINISHED', 'metadata': 'FINISHED'}, 'result': 'success', 'id': '5b068b894949c40b7c67a33b', 'details': [{'num_processed': 1, 'items_total': 1, 'state': 'FINISHED', 'num_success': 1, 'error_details': [...], 'description': 'Copying files', 'num_failures': 0, 'step_id': '52f9feaf-3783-44a3-9ae0-136c58f6134c', 'step_type': 'save_tar', 'details', ...}]}
Actions #1

Updated by dkliban@redhat.com almost 6 years ago

  • Project changed from Pulp to RPM Support

There are a couple of things that come to mind:

The repository got republished and the repomd file has a different version. Pulp assumes that the metadata changed and needs to be processed.

The importer config was updated after the last sync. In this case Pulp does not perform any optimizations for the sync the follows the importer config update.

Could either of those happened here?

Actions #2

Updated by vaclav.adamec@suchy-zleb.cz almost 6 years ago

No updates to configuration and based on timestamps of http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/repodata/ no changes at feed side. It tooks about 20-30 minutes to run sync everytime, other repos are ok, I can run it several times and get skipped, no change. But OL6 and OL7 latest refresh metadata everytime.

Actions #3

Updated by ipanova@redhat.com almost 6 years ago

it behaves like this because repomd revision tag is missing completely.
The code checks previous and current revision and based on the number it can tell whether the content changed on the remote repo or not.
In this case we cannot tell what happened with the remote content that's why we parse metadata with every sync.

previous_revision = None
current_revision = None

skip_sync_steps = not force_full_sync and \
0 < self.current_revision <= previous_revision and \
(self.download_deferred or not missing_units)

0 < self.current_revision <= previous_revision

In [9]: 0 < None <= None
Out [9]: False

which makes overall result of skip_sync_steps False.

Actions #4

Updated by vaclav.adamec@suchy-zleb.cz almost 6 years ago

H, you are right, I missed that. Ok, try to open ticket to Oracle, wish me luck :-)

Actions #5

Updated by milan almost 6 years ago

  • Status changed from NEW to CLOSED - NOTABUG

Good luck! :)

Actions #6

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF