Project

Profile

Help

Issue #5906

closed

tasks can be assigned to dead workers

Added by jsherril@redhat.com over 4 years ago. Updated about 4 years ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 65
Quarter:

Description

I hit a situation with pulp3 where i synced a repository and the task never got picked up. After restarting ALL pulp3 services, i was still seeing this behavior. Other repositories synced without issue.

Turns out, all these stuck syncs are being assigned to the same worker that no longer existed:

- pulp_href: "/pulp/api/v3/tasks/04b2f793-ef3e-49fa-8698-b3f86654d48e/"
  pulp_created: '2019-12-19T20:28:02.060+00:00'
  state: waiting
  name: pulp_container.app.tasks.synchronize.synchronize
  worker: "/pulp/api/v3/workers/3880936c-1a64-46ff-9651-7fefef088c69/"
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/remotes/container/container/366ec86d-5910-499f-ab51-07bc73cafb80/"
  - "/pulp/api/v3/repositories/container/container/1026248f-1e62-497d-9dee-d8c959dea8d4/"

stutus output:

{
  "versions": [
    {
      "component": "pulpcore",
      "version": "3.0.0rc9"
    },
    {
      "component": "pulp_2to3_migration",
      "version": "0.0.1a1.dev0"
    },
    {
      "component": "pulp_file",
      "version": "0.1.0.dev0"
    },
    {
      "component": "pulp_container",
      "version": "1.0.0rc2.dev0"
    },
    {
      "component": "pulp_ansible",
      "version": "0.2.0b7.dev0"
    },
    {
      "component": "pulp_rpm",
      "version": "3.0.0rc2.dev0"
    }
  ],
  "online_workers": [
    {
      "pulp_href": "/pulp/api/v3/workers/7f91ec97-98e3-47f2-a1d8-10e0f2bf13ec/",
      "pulp_created": "2019-12-19T20:27:44.849421Z",
      "name": "23430@devel.balmora.example.com",
      "last_heartbeat": "2019-12-19T20:38:33.138047Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/19bf0a63-d8c9-4f9b-b7e2-4597c903c31c/",
      "pulp_created": "2019-12-19T20:27:45.006246Z",
      "name": "23429@devel.balmora.example.com",
      "last_heartbeat": "2019-12-19T20:38:33.138315Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/68acaa8e-5b2e-4fb5-b17a-f903171691db/",
      "pulp_created": "2019-12-11T17:10:07.192264Z",
      "name": "resource-manager",
      "last_heartbeat": "2019-12-19T20:38:35.546686Z"
    }
  ],
  "online_content_apps": [
    {
      "name": "23444@devel.balmora.example.com",
      "last_heartbeat": "2019-12-19T20:38:31.081544Z"
    },
    {
      "name": "23442@devel.balmora.example.com",
      "last_heartbeat": "2019-12-19T20:38:31.088967Z"
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": true
  },
  "storage": {
    "total": 42927656960,
    "used": 23027937280,
    "free": 19899719680
  }
}

Dead worker details:

$ curl https://`hostname`/pulp/api/v3/workers/3880936c-1a64-46ff-9651-7fefef088c69/  -u admin:password | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   206  100   206    0     0    655      0 --:--:-- --:--:-- --:--:--   658
{
  "pulp_href": "/pulp/api/v3/workers/3880936c-1a64-46ff-9651-7fefef088c69/",
  "pulp_created": "2019-12-16T15:04:54.639299Z",
  "name": "1989@devel.balmora.example.com",
  "last_heartbeat": "2019-12-19T19:53:36.786369Z"
}

Also available in: Atom PDF