Actions
Issue #2827
closed
Pulp resource lock mechanism is not working correctly
Added by jluza almost 7 years ago. Updated about 5 years ago.
Status:
CLOSED - NOTABUG
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
I just found out there are two running cdn publishes at the same time:
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ date
Wed Jun 21 15:14:42 CEST 2017
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID 4cc17c8c-6204-4f0b-9133-527f6862e1a1 | grep state
"state": "FINISHED",
"state": "FINISHED",
"state": "FINISHED",
"state": "FINISHED",
"state": "IN_PROGRESS",
"state": "NOT_STARTED",
"state": "running",
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ date
Wed Jun 21 15:14:48 CEST 2017
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca | grep state
"state": "FINISHED",
"state": "FINISHED",
"state": "FINISHED",
"state": "FINISHED",
"state": "IN_PROGRESS",
"state": "NOT_STARTED",
"state": "running",
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID 4cc17c8c-6204-4f0b-9133-527f6862e1a1 | grep repository
"pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
[jluza@dhcp-24-116 rcm-pulp-cdn-distributor](master)$ pa_tool --env stage task get --TASK_ID e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca | grep repository
"pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
Updated by bmbouter almost 7 years ago
jluza, thanks for filing. Can you list the following info on the pulp server: rpm -qa | grep pulp
Also can you provide the -vv output of pulp-admin tasks as attachments to this issue for those tasks. Specifically those commands would be:
pulp-admin -vv tasks details --task-id 4cc17c8c-6204-4f0b-9133-527f6862e1a1
pulp-admin -vv tasks details --task-id e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca
Updated by jluza almost 7 years ago
mod_wsgi-3.4-1.pulp.el6.x86_64
pulp-admin-client-2.7.1-1.1.12.el6eng.noarch
pulp-cdn-distributor-plugins-4.1.1-1.el6eng.noarch
pulp-rpm-admin-extensions-2.7.1-1.1.18.el6eng.noarch
pulp-rpm-handlers-2.7.1-1.1.37.el6eng.noarch
pulp-rpm-plugins-2.7.1-1.1.37.el6eng.noarch
pulp-rpm-yumplugins-2.7.1-1.1.37.el6eng.noarch
pulp-selinux-2.7.1-1.1.12.el6eng.noarch
pulp-server-2.7.1-1.1.12.el6eng.noarch
python-celery-3.1.11-1.el6.noarch
python-gofer-qpid-2.6.6-2.el6_6sat.noarch
python-isodate-0.5.0-4.pulp.el6.noarch
python-kombu-3.0.33-5.pulp.el6.noarch
python-pulp-agent-lib-2.7.1-1.1.2.el6eng.noarch
python-pulp-bindings-2.7.1-1.1.12.el6eng.noarch
python-pulp-cdn-distributor-common-4.1.1-1.el6eng.noarch
python-pulp-client-lib-2.7.1-1.1.12.el6eng.noarch
python-pulp-common-2.7.1-1.1.12.el6eng.noarch
python-pulp-repoauth-2.7.1-1.1.12.el6eng.noarch
python-pulp-rpm-common-2.7.1-1.1.37.el6eng.noarch
python-qpid-0.30-9.el6sat.noarch
python-qpid-qmf-0.30-6.el6.x86_64
qpid-cpp-client-0.30-9.el6.x86_64
qpid-cpp-server-0.30-9.el6.x86_64
qpid-proton-c-0.9-3.el6.x86_64
qpid-qmf-0.30-6.el6.x86_64
qpid-tools-0.30-5.el6.noarch
{
"_href": "/pulp/api/v2/tasks/e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca/",
"_id": {
"$oid": "594a36999cd7d8282fa40d2a"
},
"_ns": "task_status",
"error": null,
"exception": null,
"finish_time": "2017-06-21T13:32:40Z",
"id": "594a36999cd7d8282fa40d2a",
"progress_report": {
"cdn_distributor": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "1ec72330-6c5a-4a88-a4cf-d0c9e06d67e3",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1d2e5362-9c1b-49e2-8042-5d2fd4122f73",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "303201cc-d291-42d3-85fb-7c78358326ec",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1c89b1cd-8b43-4571-96e5-f1f7f7e5f789",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "e948563d-f4cc-42b4-83be-79e0479c130b",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "147fd78c-7979-45fd-bbed-5aeef511cbde",
"step_type": "UpdateLastPredistDate"
}
]
},
"queue": "reserved_resource_worker-0@pulp04.web.stage.ext.phx2.redhat.com.dq",
"result": {
"_ns": "repo_publish_results",
"completed": "2017-06-21T13:32:40Z",
"details": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "1ec72330-6c5a-4a88-a4cf-d0c9e06d67e3",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1d2e5362-9c1b-49e2-8042-5d2fd4122f73",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "303201cc-d291-42d3-85fb-7c78358326ec",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1c89b1cd-8b43-4571-96e5-f1f7f7e5f789",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "e948563d-f4cc-42b4-83be-79e0479c130b",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "147fd78c-7979-45fd-bbed-5aeef511cbde",
"step_type": "UpdateLastPredistDate"
}
],
"distributor_config": {
"cdn_rel_prefix": "/content/origin/rpms/",
"content_types": [
"rpm"
],
"content_units_only": false,
"force_full": true,
"handler_type": "rsync",
"http": true,
"https": false,
"relative_url": "repo",
"remote": {
"auth_type": "publickey",
"cdn_login": "rcm-cron",
"host": "pulp04.web.stage.ext.phx2.redhat.com",
"root": "/mnt/cdn/cdn-stage",
"skip_repodata": false,
"ssh_identity_file": "/etc/httpd/id_rsa",
"ssh_user": "rcm-cron"
},
"skips": [
"extra"
]
},
"distributor_id": "cdn_distributor",
"distributor_type_id": "cdn_distributor",
"error_message": null,
"exception": null,
"id": "594a7578b43c1421dd5a629a",
"repo_id": "rhel-7-desktop-rpms__7Client__x86_64",
"result": "success",
"started": "2017-06-21T12:35:29Z",
"summary": {
"Rsync step (content)": "FINISHED",
"Rsync step (origin)": "FINISHED",
"Rsync step (repodata)": "FINISHED",
"Unit query step (rpm)": "FINISHED",
"UpdateLastPredistDate": "FINISHED",
"save_tar": "FINISHED"
},
"traceback": null
},
"spawned_tasks": [],
"start_time": "2017-06-21T12:35:29Z",
"state": "finished",
"tags": [
"pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
"pulp:action:publish"
],
"task_id": "e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca",
"task_type": "pulp.server.managers.repo.publish.publish",
"traceback": null,
"worker_name": "reserved_resource_worker-0@pulp04.web.stage.ext.phx2.redhat.com"
}
{
"_href": "/pulp/api/v2/tasks/4cc17c8c-6204-4f0b-9133-527f6862e1a1/",
"_id": {
"$oid": "594a68149cd7d8282fa40eba"
},
"_ns": "task_status",
"error": null,
"exception": null,
"finish_time": "2017-06-21T13:38:05Z",
"id": "594a68149cd7d8282fa40eba",
"progress_report": {
"cdn_distributor": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "7e9e3413-187c-4370-8e13-48ea148493ee",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "4af8eb19-3cdf-4852-8b02-e806fc788822",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "fe90afab-52bd-470f-b95f-0dc7a5ca2fd2",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "b6a1309b-a270-42e8-a895-a4182763b1e5",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "5ed4e4c2-02b2-4b7e-b20a-7ca2b51aa41f",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "a7c9bb25-047d-4710-aa42-36c4783e9e57",
"step_type": "UpdateLastPredistDate"
}
]
},
"queue": "reserved_resource_worker-6@pulp05.web.stage.ext.phx2.redhat.com.dq",
"result": {
"_ns": "repo_publish_results",
"completed": "2017-06-21T13:38:05Z",
"details": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "7e9e3413-187c-4370-8e13-48ea148493ee",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "4af8eb19-3cdf-4852-8b02-e806fc788822",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "fe90afab-52bd-470f-b95f-0dc7a5ca2fd2",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "b6a1309b-a270-42e8-a895-a4182763b1e5",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "5ed4e4c2-02b2-4b7e-b20a-7ca2b51aa41f",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "a7c9bb25-047d-4710-aa42-36c4783e9e57",
"step_type": "UpdateLastPredistDate"
}
],
"distributor_config": {
"cdn_rel_prefix": "/content/origin/rpms/",
"content_types": [
"rpm"
],
"content_units_only": false,
"force_full": true,
"handler_type": "rsync",
"http": true,
"https": false,
"relative_url": "repo",
"remote": {
"auth_type": "publickey",
"cdn_login": "rcm-cron",
"host": "pulp04.web.stage.ext.phx2.redhat.com",
"root": "/mnt/cdn/cdn-stage",
"skip_repodata": false,
"ssh_identity_file": "/etc/httpd/id_rsa",
"ssh_user": "rcm-cron"
},
"skips": [
"extra"
]
},
"distributor_id": "cdn_distributor",
"distributor_type_id": "cdn_distributor",
"error_message": null,
"exception": null,
"id": "594a76bd46ac687190da9269",
"repo_id": "rhel-7-desktop-rpms__7Client__x86_64",
"result": "success",
"started": "2017-06-21T12:35:32Z",
"summary": {
"Rsync step (content)": "FINISHED",
"Rsync step (origin)": "FINISHED",
"Rsync step (repodata)": "FINISHED",
"Unit query step (rpm)": "FINISHED",
"UpdateLastPredistDate": "FINISHED",
"save_tar": "FINISHED"
},
"traceback": null
},
"spawned_tasks": [],
"start_time": "2017-06-21T12:35:32Z",
"state": "finished",
"tags": [
"pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
"pulp:action:publish"
],
"task_id": "4cc17c8c-6204-4f0b-9133-527f6862e1a1",
"task_type": "pulp.server.managers.repo.publish.publish",
"traceback": null,
"worker_name": "reserved_resource_worker-6@pulp05.web.stage.ext.phx2.redhat.com"
}
Updated by bmbouter almost 7 years ago
jluza, thanks for the paste. The rpms being used are not upstream rpms. Also it's using a distributor that is not present in upstream Pulp.
Can this be reproduced against a vanilla Pulp system?
Updated by jluza almost 7 years ago
Today we just discovered we have two celery beats running on our pulp nodes. And that's the most probable problem. However we are not sure. Also, it's very hard to reproduce issues like this. Rohan tried to reproduce it on very same system but he didn't succeed.
Updated by ttereshc almost 7 years ago
- Status changed from NEW to CLOSED - NOTABUG
- Triaged changed from No to Yes
The issue is observed in a fork and not in the upstream Pulp.
As for the multiple celerybeats, it is supported in the upstream Pulp.
Actions
.