Project

Profile

Help

Issue #2827

Pulp resource lock mechanism is not working correctly

Added by jluza over 3 years ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

I just found out there are two running cdn publishes at the same time:

[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ date
Wed Jun 21 15:14:42 CEST 2017
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID 4cc17c8c-6204-4f0b-9133-527f6862e1a1 | grep state
                "state": "FINISHED",
                "state": "FINISHED",
                "state": "FINISHED",
                "state": "FINISHED",
                "state": "IN_PROGRESS",
                "state": "NOT_STARTED",
    "state": "running",
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ date
Wed Jun 21 15:14:48 CEST 2017
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca | grep state
                "state": "FINISHED",
                "state": "FINISHED",
                "state": "FINISHED",
                "state": "FINISHED",
                "state": "IN_PROGRESS",
                "state": "NOT_STARTED",
    "state": "running",
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID 4cc17c8c-6204-4f0b-9133-527f6862e1a1 | grep repository
        "pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca | grep repository
        "pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",

History

#1 Updated by bmbouter over 3 years ago

jluza, thanks for filing. Can you list the following info on the pulp server: rpm -qa | grep pulp

Also can you provide the -vv output of pulp-admin tasks as attachments to this issue for those tasks. Specifically those commands would be:

pulp-admin -vv tasks details --task-id 4cc17c8c-6204-4f0b-9133-527f6862e1a1
pulp-admin -vv tasks details --task-id e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca

#2 Updated by jluza over 3 years ago

mod_wsgi-3.4-1.pulp.el6.x86_64
pulp-admin-client-2.7.1-1.1.12.el6eng.noarch
pulp-cdn-distributor-plugins-4.1.1-1.el6eng.noarch
pulp-rpm-admin-extensions-2.7.1-1.1.18.el6eng.noarch
pulp-rpm-handlers-2.7.1-1.1.37.el6eng.noarch
pulp-rpm-plugins-2.7.1-1.1.37.el6eng.noarch
pulp-rpm-yumplugins-2.7.1-1.1.37.el6eng.noarch
pulp-selinux-2.7.1-1.1.12.el6eng.noarch
pulp-server-2.7.1-1.1.12.el6eng.noarch
python-celery-3.1.11-1.el6.noarch
python-gofer-qpid-2.6.6-2.el6_6sat.noarch
python-isodate-0.5.0-4.pulp.el6.noarch
python-kombu-3.0.33-5.pulp.el6.noarch
python-pulp-agent-lib-2.7.1-1.1.2.el6eng.noarch
python-pulp-bindings-2.7.1-1.1.12.el6eng.noarch
python-pulp-cdn-distributor-common-4.1.1-1.el6eng.noarch
python-pulp-client-lib-2.7.1-1.1.12.el6eng.noarch
python-pulp-common-2.7.1-1.1.12.el6eng.noarch
python-pulp-repoauth-2.7.1-1.1.12.el6eng.noarch
python-pulp-rpm-common-2.7.1-1.1.37.el6eng.noarch
python-qpid-0.30-9.el6sat.noarch
python-qpid-qmf-0.30-6.el6.x86_64
qpid-cpp-client-0.30-9.el6.x86_64
qpid-cpp-server-0.30-9.el6.x86_64
qpid-proton-c-0.9-3.el6.x86_64
qpid-qmf-0.30-6.el6.x86_64
qpid-tools-0.30-5.el6.noarch
{
    "_href": "/pulp/api/v2/tasks/e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca/",
    "_id": {
        "$oid": "594a36999cd7d8282fa40d2a"
    },
    "_ns": "task_status",
    "error": null,
    "exception": null,
    "finish_time": "2017-06-21T13:32:40Z",
    "id": "594a36999cd7d8282fa40d2a",
    "progress_report": {
        "cdn_distributor": [
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 13579,
                "num_failures": 0,
                "num_processed": 13579,
                "num_success": 13579,
                "state": "FINISHED",
                "step_id": "1ec72330-6c5a-4a88-a4cf-d0c9e06d67e3",
                "step_type": "Unit query step (rpm)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "1d2e5362-9c1b-49e2-8042-5d2fd4122f73",
                "step_type": "Rsync step (origin)"
            },
            {
                "description": "Copying selected files",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "303201cc-d291-42d3-85fb-7c78358326ec",
                "step_type": "save_tar"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "1c89b1cd-8b43-4571-96e5-f1f7f7e5f789",
                "step_type": "Rsync step (content)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "e948563d-f4cc-42b4-83be-79e0479c130b",
                "step_type": "Rsync step (repodata)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "147fd78c-7979-45fd-bbed-5aeef511cbde",
                "step_type": "UpdateLastPredistDate"
            }
        ]
    },
    "queue": "reserved_resource_worker-0@pulp04.web.stage.ext.phx2.redhat.com.dq",
    "result": {
        "_ns": "repo_publish_results",
        "completed": "2017-06-21T13:32:40Z",
        "details": [
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 13579,
                "num_failures": 0,
                "num_processed": 13579,
                "num_success": 13579,
                "state": "FINISHED",
                "step_id": "1ec72330-6c5a-4a88-a4cf-d0c9e06d67e3",
                "step_type": "Unit query step (rpm)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "1d2e5362-9c1b-49e2-8042-5d2fd4122f73",
                "step_type": "Rsync step (origin)"
            },
            {
                "description": "Copying selected files",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "303201cc-d291-42d3-85fb-7c78358326ec",
                "step_type": "save_tar"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "1c89b1cd-8b43-4571-96e5-f1f7f7e5f789",
                "step_type": "Rsync step (content)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "e948563d-f4cc-42b4-83be-79e0479c130b",
                "step_type": "Rsync step (repodata)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "147fd78c-7979-45fd-bbed-5aeef511cbde",
                "step_type": "UpdateLastPredistDate"
            }
        ],
        "distributor_config": {
            "cdn_rel_prefix": "/content/origin/rpms/",
            "content_types": [
                "rpm"
            ],
            "content_units_only": false,
            "force_full": true,
            "handler_type": "rsync",
            "http": true,
            "https": false,
            "relative_url": "repo",
            "remote": {
                "auth_type": "publickey",
                "cdn_login": "rcm-cron",
                "host": "pulp04.web.stage.ext.phx2.redhat.com",
                "root": "/mnt/cdn/cdn-stage",
                "skip_repodata": false,
                "ssh_identity_file": "/etc/httpd/id_rsa",
                "ssh_user": "rcm-cron"
            },
            "skips": [
                "extra"
            ]
        },
        "distributor_id": "cdn_distributor",
        "distributor_type_id": "cdn_distributor",
        "error_message": null,
        "exception": null,
        "id": "594a7578b43c1421dd5a629a",
        "repo_id": "rhel-7-desktop-rpms__7Client__x86_64",
        "result": "success",
        "started": "2017-06-21T12:35:29Z",
        "summary": {
            "Rsync step (content)": "FINISHED",
            "Rsync step (origin)": "FINISHED",
            "Rsync step (repodata)": "FINISHED",
            "Unit query step (rpm)": "FINISHED",
            "UpdateLastPredistDate": "FINISHED",
            "save_tar": "FINISHED"
        },
        "traceback": null
    },
    "spawned_tasks": [],
    "start_time": "2017-06-21T12:35:29Z",
    "state": "finished",
    "tags": [
        "pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
        "pulp:action:publish"
    ],
    "task_id": "e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca",
    "task_type": "pulp.server.managers.repo.publish.publish",
    "traceback": null,
    "worker_name": "reserved_resource_worker-0@pulp04.web.stage.ext.phx2.redhat.com"
}
{
    "_href": "/pulp/api/v2/tasks/4cc17c8c-6204-4f0b-9133-527f6862e1a1/",
    "_id": {
        "$oid": "594a68149cd7d8282fa40eba"
    },
    "_ns": "task_status",
    "error": null,
    "exception": null,
    "finish_time": "2017-06-21T13:38:05Z",
    "id": "594a68149cd7d8282fa40eba",
    "progress_report": {
        "cdn_distributor": [
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 13579,
                "num_failures": 0,
                "num_processed": 13579,
                "num_success": 13579,
                "state": "FINISHED",
                "step_id": "7e9e3413-187c-4370-8e13-48ea148493ee",
                "step_type": "Unit query step (rpm)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "4af8eb19-3cdf-4852-8b02-e806fc788822",
                "step_type": "Rsync step (origin)"
            },
            {
                "description": "Copying selected files",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "fe90afab-52bd-470f-b95f-0dc7a5ca2fd2",
                "step_type": "save_tar"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "b6a1309b-a270-42e8-a895-a4182763b1e5",
                "step_type": "Rsync step (content)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "5ed4e4c2-02b2-4b7e-b20a-7ca2b51aa41f",
                "step_type": "Rsync step (repodata)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "a7c9bb25-047d-4710-aa42-36c4783e9e57",
                "step_type": "UpdateLastPredistDate"
            }
        ]
    },
    "queue": "reserved_resource_worker-6@pulp05.web.stage.ext.phx2.redhat.com.dq",
    "result": {
        "_ns": "repo_publish_results",
        "completed": "2017-06-21T13:38:05Z",
        "details": [
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 13579,
                "num_failures": 0,
                "num_processed": 13579,
                "num_success": 13579,
                "state": "FINISHED",
                "step_id": "7e9e3413-187c-4370-8e13-48ea148493ee",
                "step_type": "Unit query step (rpm)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "4af8eb19-3cdf-4852-8b02-e806fc788822",
                "step_type": "Rsync step (origin)"
            },
            {
                "description": "Copying selected files",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "fe90afab-52bd-470f-b95f-0dc7a5ca2fd2",
                "step_type": "save_tar"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "b6a1309b-a270-42e8-a895-a4182763b1e5",
                "step_type": "Rsync step (content)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "5ed4e4c2-02b2-4b7e-b20a-7ca2b51aa41f",
                "step_type": "Rsync step (repodata)"
            },
            {
                "description": "",
                "details": "",
                "error_details": [],
                "items_total": 1,
                "num_failures": 0,
                "num_processed": 1,
                "num_success": 1,
                "state": "FINISHED",
                "step_id": "a7c9bb25-047d-4710-aa42-36c4783e9e57",
                "step_type": "UpdateLastPredistDate"
            }
        ],
        "distributor_config": {
            "cdn_rel_prefix": "/content/origin/rpms/",
            "content_types": [
                "rpm"
            ],
            "content_units_only": false,
            "force_full": true,
            "handler_type": "rsync",
            "http": true,
            "https": false,
            "relative_url": "repo",
            "remote": {
                "auth_type": "publickey",
                "cdn_login": "rcm-cron",
                "host": "pulp04.web.stage.ext.phx2.redhat.com",
                "root": "/mnt/cdn/cdn-stage",
                "skip_repodata": false,
                "ssh_identity_file": "/etc/httpd/id_rsa",
                "ssh_user": "rcm-cron"
            },
            "skips": [
                "extra"
            ]
        },
        "distributor_id": "cdn_distributor",
        "distributor_type_id": "cdn_distributor",
        "error_message": null,
        "exception": null,
        "id": "594a76bd46ac687190da9269",
        "repo_id": "rhel-7-desktop-rpms__7Client__x86_64",
        "result": "success",
        "started": "2017-06-21T12:35:32Z",
        "summary": {
            "Rsync step (content)": "FINISHED",
            "Rsync step (origin)": "FINISHED",
            "Rsync step (repodata)": "FINISHED",
            "Unit query step (rpm)": "FINISHED",
            "UpdateLastPredistDate": "FINISHED",
            "save_tar": "FINISHED"
        },
        "traceback": null
    },
    "spawned_tasks": [],
    "start_time": "2017-06-21T12:35:32Z",
    "state": "finished",
    "tags": [
        "pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
        "pulp:action:publish"
    ],
    "task_id": "4cc17c8c-6204-4f0b-9133-527f6862e1a1",
    "task_type": "pulp.server.managers.repo.publish.publish",
    "traceback": null,
    "worker_name": "reserved_resource_worker-6@pulp05.web.stage.ext.phx2.redhat.com"
}

#3 Updated by bmbouter over 3 years ago

jluza, thanks for the paste. The rpms being used are not upstream rpms. Also it's using a distributor that is not present in upstream Pulp.

Can this be reproduced against a vanilla Pulp system?

#4 Updated by jluza over 3 years ago

Today we just discovered we have two celery beats running on our pulp nodes. And that's the most probable problem. However we are not sure. Also, it's very hard to reproduce issues like this. Rohan tried to reproduce it on very same system but he didn't succeed.

#5 Updated by ttereshc over 3 years ago

  • Status changed from NEW to CLOSED - NOTABUG
  • Triaged changed from No to Yes

The issue is observed in a fork and not in the upstream Pulp.
As for the multiple celerybeats, it is supported in the upstream Pulp.

#6 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF