https://pulp.plan.io/
https://pulp.plan.io/favicon.ico
2017-06-21T16:05:35Z
Pulp
Pulp - Issue #2827: Pulp resource lock mechanism is not working correctly
https://pulp.plan.io/issues/2827?journal_id=20336
2017-06-21T16:05:35Z
bmbouter
bmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/24">jluza</a>, thanks for filing. Can you list the following info on the pulp server: <code>rpm -qa | grep pulp</code></p>
<p>Also can you provide the -vv output of pulp-admin tasks as attachments to this issue for those tasks. Specifically those commands would be:</p>
<pre><code>pulp-admin -vv tasks details --task-id 4cc17c8c-6204-4f0b-9133-527f6862e1a1
pulp-admin -vv tasks details --task-id e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca
</code></pre>
Pulp - Issue #2827: Pulp resource lock mechanism is not working correctly
https://pulp.plan.io/issues/2827?journal_id=20337
2017-06-21T16:13:29Z
jluza
jluza@redhat.com
<ul></ul><pre><code>mod_wsgi-3.4-1.pulp.el6.x86_64
pulp-admin-client-2.7.1-1.1.12.el6eng.noarch
pulp-cdn-distributor-plugins-4.1.1-1.el6eng.noarch
pulp-rpm-admin-extensions-2.7.1-1.1.18.el6eng.noarch
pulp-rpm-handlers-2.7.1-1.1.37.el6eng.noarch
pulp-rpm-plugins-2.7.1-1.1.37.el6eng.noarch
pulp-rpm-yumplugins-2.7.1-1.1.37.el6eng.noarch
pulp-selinux-2.7.1-1.1.12.el6eng.noarch
pulp-server-2.7.1-1.1.12.el6eng.noarch
python-celery-3.1.11-1.el6.noarch
python-gofer-qpid-2.6.6-2.el6_6sat.noarch
python-isodate-0.5.0-4.pulp.el6.noarch
python-kombu-3.0.33-5.pulp.el6.noarch
python-pulp-agent-lib-2.7.1-1.1.2.el6eng.noarch
python-pulp-bindings-2.7.1-1.1.12.el6eng.noarch
python-pulp-cdn-distributor-common-4.1.1-1.el6eng.noarch
python-pulp-client-lib-2.7.1-1.1.12.el6eng.noarch
python-pulp-common-2.7.1-1.1.12.el6eng.noarch
python-pulp-repoauth-2.7.1-1.1.12.el6eng.noarch
python-pulp-rpm-common-2.7.1-1.1.37.el6eng.noarch
python-qpid-0.30-9.el6sat.noarch
python-qpid-qmf-0.30-6.el6.x86_64
qpid-cpp-client-0.30-9.el6.x86_64
qpid-cpp-server-0.30-9.el6.x86_64
qpid-proton-c-0.9-3.el6.x86_64
qpid-qmf-0.30-6.el6.x86_64
qpid-tools-0.30-5.el6.noarch
</code></pre>
<pre><code>{
"_href": "/pulp/api/v2/tasks/e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca/",
"_id": {
"$oid": "594a36999cd7d8282fa40d2a"
},
"_ns": "task_status",
"error": null,
"exception": null,
"finish_time": "2017-06-21T13:32:40Z",
"id": "594a36999cd7d8282fa40d2a",
"progress_report": {
"cdn_distributor": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "1ec72330-6c5a-4a88-a4cf-d0c9e06d67e3",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1d2e5362-9c1b-49e2-8042-5d2fd4122f73",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "303201cc-d291-42d3-85fb-7c78358326ec",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1c89b1cd-8b43-4571-96e5-f1f7f7e5f789",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "e948563d-f4cc-42b4-83be-79e0479c130b",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "147fd78c-7979-45fd-bbed-5aeef511cbde",
"step_type": "UpdateLastPredistDate"
}
]
},
"queue": "reserved_resource_worker-0@pulp04.web.stage.ext.phx2.redhat.com.dq",
"result": {
"_ns": "repo_publish_results",
"completed": "2017-06-21T13:32:40Z",
"details": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "1ec72330-6c5a-4a88-a4cf-d0c9e06d67e3",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1d2e5362-9c1b-49e2-8042-5d2fd4122f73",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "303201cc-d291-42d3-85fb-7c78358326ec",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "1c89b1cd-8b43-4571-96e5-f1f7f7e5f789",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "e948563d-f4cc-42b4-83be-79e0479c130b",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "147fd78c-7979-45fd-bbed-5aeef511cbde",
"step_type": "UpdateLastPredistDate"
}
],
"distributor_config": {
"cdn_rel_prefix": "/content/origin/rpms/",
"content_types": [
"rpm"
],
"content_units_only": false,
"force_full": true,
"handler_type": "rsync",
"http": true,
"https": false,
"relative_url": "repo",
"remote": {
"auth_type": "publickey",
"cdn_login": "rcm-cron",
"host": "pulp04.web.stage.ext.phx2.redhat.com",
"root": "/mnt/cdn/cdn-stage",
"skip_repodata": false,
"ssh_identity_file": "/etc/httpd/id_rsa",
"ssh_user": "rcm-cron"
},
"skips": [
"extra"
]
},
"distributor_id": "cdn_distributor",
"distributor_type_id": "cdn_distributor",
"error_message": null,
"exception": null,
"id": "594a7578b43c1421dd5a629a",
"repo_id": "rhel-7-desktop-rpms__7Client__x86_64",
"result": "success",
"started": "2017-06-21T12:35:29Z",
"summary": {
"Rsync step (content)": "FINISHED",
"Rsync step (origin)": "FINISHED",
"Rsync step (repodata)": "FINISHED",
"Unit query step (rpm)": "FINISHED",
"UpdateLastPredistDate": "FINISHED",
"save_tar": "FINISHED"
},
"traceback": null
},
"spawned_tasks": [],
"start_time": "2017-06-21T12:35:29Z",
"state": "finished",
"tags": [
"pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
"pulp:action:publish"
],
"task_id": "e799fa1c-3b5b-43c0-8bad-ba5d3ec794ca",
"task_type": "pulp.server.managers.repo.publish.publish",
"traceback": null,
"worker_name": "reserved_resource_worker-0@pulp04.web.stage.ext.phx2.redhat.com"
}
</code></pre>
<pre><code>{
"_href": "/pulp/api/v2/tasks/4cc17c8c-6204-4f0b-9133-527f6862e1a1/",
"_id": {
"$oid": "594a68149cd7d8282fa40eba"
},
"_ns": "task_status",
"error": null,
"exception": null,
"finish_time": "2017-06-21T13:38:05Z",
"id": "594a68149cd7d8282fa40eba",
"progress_report": {
"cdn_distributor": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "7e9e3413-187c-4370-8e13-48ea148493ee",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "4af8eb19-3cdf-4852-8b02-e806fc788822",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "fe90afab-52bd-470f-b95f-0dc7a5ca2fd2",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "b6a1309b-a270-42e8-a895-a4182763b1e5",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "5ed4e4c2-02b2-4b7e-b20a-7ca2b51aa41f",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "a7c9bb25-047d-4710-aa42-36c4783e9e57",
"step_type": "UpdateLastPredistDate"
}
]
},
"queue": "reserved_resource_worker-6@pulp05.web.stage.ext.phx2.redhat.com.dq",
"result": {
"_ns": "repo_publish_results",
"completed": "2017-06-21T13:38:05Z",
"details": [
{
"description": "",
"details": "",
"error_details": [],
"items_total": 13579,
"num_failures": 0,
"num_processed": 13579,
"num_success": 13579,
"state": "FINISHED",
"step_id": "7e9e3413-187c-4370-8e13-48ea148493ee",
"step_type": "Unit query step (rpm)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "4af8eb19-3cdf-4852-8b02-e806fc788822",
"step_type": "Rsync step (origin)"
},
{
"description": "Copying selected files",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "fe90afab-52bd-470f-b95f-0dc7a5ca2fd2",
"step_type": "save_tar"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "b6a1309b-a270-42e8-a895-a4182763b1e5",
"step_type": "Rsync step (content)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "5ed4e4c2-02b2-4b7e-b20a-7ca2b51aa41f",
"step_type": "Rsync step (repodata)"
},
{
"description": "",
"details": "",
"error_details": [],
"items_total": 1,
"num_failures": 0,
"num_processed": 1,
"num_success": 1,
"state": "FINISHED",
"step_id": "a7c9bb25-047d-4710-aa42-36c4783e9e57",
"step_type": "UpdateLastPredistDate"
}
],
"distributor_config": {
"cdn_rel_prefix": "/content/origin/rpms/",
"content_types": [
"rpm"
],
"content_units_only": false,
"force_full": true,
"handler_type": "rsync",
"http": true,
"https": false,
"relative_url": "repo",
"remote": {
"auth_type": "publickey",
"cdn_login": "rcm-cron",
"host": "pulp04.web.stage.ext.phx2.redhat.com",
"root": "/mnt/cdn/cdn-stage",
"skip_repodata": false,
"ssh_identity_file": "/etc/httpd/id_rsa",
"ssh_user": "rcm-cron"
},
"skips": [
"extra"
]
},
"distributor_id": "cdn_distributor",
"distributor_type_id": "cdn_distributor",
"error_message": null,
"exception": null,
"id": "594a76bd46ac687190da9269",
"repo_id": "rhel-7-desktop-rpms__7Client__x86_64",
"result": "success",
"started": "2017-06-21T12:35:32Z",
"summary": {
"Rsync step (content)": "FINISHED",
"Rsync step (origin)": "FINISHED",
"Rsync step (repodata)": "FINISHED",
"Unit query step (rpm)": "FINISHED",
"UpdateLastPredistDate": "FINISHED",
"save_tar": "FINISHED"
},
"traceback": null
},
"spawned_tasks": [],
"start_time": "2017-06-21T12:35:32Z",
"state": "finished",
"tags": [
"pulp:repository:rhel-7-desktop-rpms__7Client__x86_64",
"pulp:action:publish"
],
"task_id": "4cc17c8c-6204-4f0b-9133-527f6862e1a1",
"task_type": "pulp.server.managers.repo.publish.publish",
"traceback": null,
"worker_name": "reserved_resource_worker-6@pulp05.web.stage.ext.phx2.redhat.com"
}
</code></pre>
Pulp - Issue #2827: Pulp resource lock mechanism is not working correctly
https://pulp.plan.io/issues/2827?journal_id=20339
2017-06-21T18:12:33Z
bmbouter
bmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/24">jluza</a>, thanks for the paste. The rpms being used are not upstream rpms. Also it's using a distributor that is not present in upstream Pulp.</p>
<p>Can this be reproduced against a vanilla Pulp system?</p>
Pulp - Issue #2827: Pulp resource lock mechanism is not working correctly
https://pulp.plan.io/issues/2827?journal_id=20348
2017-06-22T07:35:04Z
jluza
jluza@redhat.com
<ul></ul><p>Today we just discovered we have two celery beats running on our pulp nodes. And that's the most probable problem. However we are not sure. Also, it's very hard to reproduce issues like this. Rohan tried to reproduce it on very same system but he didn't succeed.</p>
Pulp - Issue #2827: Pulp resource lock mechanism is not working correctly
https://pulp.plan.io/issues/2827?journal_id=20366
2017-06-23T15:04:31Z
ttereshc
ttereshc@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>CLOSED - NOTABUG</i></li><li><strong>Triaged</strong> changed from <i>No</i> to <i>Yes</i></li></ul><p>The issue is observed in a fork and not in the upstream Pulp.<br>
As for the multiple celerybeats, it is supported in the upstream Pulp.</p>
Pulp - Issue #2827: Pulp resource lock mechanism is not working correctly
https://pulp.plan.io/issues/2827?journal_id=38228
2019-04-15T20:17:24Z
bmbouter
bmbouter@redhat.com
<ul><li><strong>Tags</strong> <i>Pulp 2</i> added</li></ul>