Actions
Issue #5906
closed
tasks can be assigned to dead workers
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 65
Quarter:
Description
I hit a situation with pulp3 where i synced a repository and the task never got picked up. After restarting ALL pulp3 services, i was still seeing this behavior. Other repositories synced without issue.
Turns out, all these stuck syncs are being assigned to the same worker that no longer existed:
- pulp_href: "/pulp/api/v3/tasks/04b2f793-ef3e-49fa-8698-b3f86654d48e/"
pulp_created: '2019-12-19T20:28:02.060+00:00'
state: waiting
name: pulp_container.app.tasks.synchronize.synchronize
worker: "/pulp/api/v3/workers/3880936c-1a64-46ff-9651-7fefef088c69/"
progress_reports: []
created_resources: []
reserved_resources_record:
- "/pulp/api/v3/remotes/container/container/366ec86d-5910-499f-ab51-07bc73cafb80/"
- "/pulp/api/v3/repositories/container/container/1026248f-1e62-497d-9dee-d8c959dea8d4/"
stutus output:
{
"versions": [
{
"component": "pulpcore",
"version": "3.0.0rc9"
},
{
"component": "pulp_2to3_migration",
"version": "0.0.1a1.dev0"
},
{
"component": "pulp_file",
"version": "0.1.0.dev0"
},
{
"component": "pulp_container",
"version": "1.0.0rc2.dev0"
},
{
"component": "pulp_ansible",
"version": "0.2.0b7.dev0"
},
{
"component": "pulp_rpm",
"version": "3.0.0rc2.dev0"
}
],
"online_workers": [
{
"pulp_href": "/pulp/api/v3/workers/7f91ec97-98e3-47f2-a1d8-10e0f2bf13ec/",
"pulp_created": "2019-12-19T20:27:44.849421Z",
"name": "23430@devel.balmora.example.com",
"last_heartbeat": "2019-12-19T20:38:33.138047Z"
},
{
"pulp_href": "/pulp/api/v3/workers/19bf0a63-d8c9-4f9b-b7e2-4597c903c31c/",
"pulp_created": "2019-12-19T20:27:45.006246Z",
"name": "23429@devel.balmora.example.com",
"last_heartbeat": "2019-12-19T20:38:33.138315Z"
},
{
"pulp_href": "/pulp/api/v3/workers/68acaa8e-5b2e-4fb5-b17a-f903171691db/",
"pulp_created": "2019-12-11T17:10:07.192264Z",
"name": "resource-manager",
"last_heartbeat": "2019-12-19T20:38:35.546686Z"
}
],
"online_content_apps": [
{
"name": "23444@devel.balmora.example.com",
"last_heartbeat": "2019-12-19T20:38:31.081544Z"
},
{
"name": "23442@devel.balmora.example.com",
"last_heartbeat": "2019-12-19T20:38:31.088967Z"
}
],
"database_connection": {
"connected": true
},
"redis_connection": {
"connected": true
},
"storage": {
"total": 42927656960,
"used": 23027937280,
"free": 19899719680
}
}
Dead worker details:
$ curl https://`hostname`/pulp/api/v3/workers/3880936c-1a64-46ff-9651-7fefef088c69/ -u admin:password | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 206 100 206 0 0 655 0 --:--:-- --:--:-- --:--:-- 658
{
"pulp_href": "/pulp/api/v3/workers/3880936c-1a64-46ff-9651-7fefef088c69/",
"pulp_created": "2019-12-16T15:04:54.639299Z",
"name": "1989@devel.balmora.example.com",
"last_heartbeat": "2019-12-19T19:53:36.786369Z"
}
Actions
.