Project

Profile

Help

Issue #7847

closed

Worker has gone missing during migration

Added by iballou over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 91
Quarter:

Description

I'm trying to migrate Pulp 2 -> Pulp 3. The migration has been working consistently until I tried 180K RPMs and 31 repos in Katello. Now, consistently, after about 15 minutes, the migration fails with the following errors:

...
Nov 17 14:52:49 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:52:49 +0000] "GET /pulp/api/v3/tasks/dfa912b7-6b4c-4d29-b546-4926e8d00a93/ HTTP/1.1" 200 3678 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:52:50 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:52:50 +0000] "GET /pulp/api/v3/task-groups/5a30cf02-4354-4b72-a8c5-a3d0d4ce6fcb/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:53:06 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:53:06 +0000] "GET /pulp/api/v3/tasks/dfa912b7-6b4c-4d29-b546-4926e8d00a93/ HTTP/1.1" 200 3678 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:53:06 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:53:06 +0000] "GET /pulp/api/v3/task-groups/5a30cf02-4354-4b72-a8c5-a3d0d4ce6fcb/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:53:52 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22312@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:52 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22312@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-7: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-5: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-5: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-7: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22319@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22319@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-6: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-6: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22323@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22323@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22315@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22315@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker 'resource-manager' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named resource-manager is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22307@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22307@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22314@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22314@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22310@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22310@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '22311@centos7-katello-nightly-2.cannolo.example.com' has gone missing, removing from list of workers
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 22311@centos7-katello-nightly-2.cannolo.example.com is missing. Canceling the tasks in its queue.
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:53:54 +0000] "GET /pulp/api/v3/tasks/dfa912b7-6b4c-4d29-b546-4926e8d00a93/ HTTP/1.1" 200 7283 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:53:54 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:53:54 +0000] "GET /pulp/api/v3/task-groups/5a30cf02-4354-4b72-a8c5-a3d0d4ce6fcb/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:54:23 centos7-katello-nightly-2 pulpcore-worker-7: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:23 centos7-katello-nightly-2 pulpcore-worker-6: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:23 centos7-katello-nightly-2 pulpcore-resource-manager: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-1: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-8: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-5: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:54:24 +0000] "GET /pulp/api/v3/tasks/dfa912b7-6b4c-4d29-b546-4926e8d00a93/ HTTP/1.1" 200 7284 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-2: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-4: pulp: rq.worker:INFO: Cleaning registries for queue: 22311@centos7-katello-nightly-2.cannolo.example.com
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-3: pulp: pulpcore.tasking.util:INFO: Task canceled: dfa912b7-6b4c-4d29-b546-4926e8d00a93.
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-4: pulp: rq.worker:INFO: 22311@centos7-katello-nightly-2.cannolo.example.com: e698be4e-6943-4f7c-9614-07e7f81c2265
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-api: - - [17/Nov/2020:14:54:24 +0000] "GET /pulp/api/v3/task-groups/5a30cf02-4354-4b72-a8c5-a3d0d4ce6fcb/ HTTP/1.1" 200 440 "-" "OpenAPI-Generator/3.7.1/ruby"
Nov 17 14:54:24 centos7-katello-nightly-2 pulpcore-worker-4: pulp: rq.worker:INFO: 22311@centos7-katello-nightly-2.cannolo.example.com: Job OK (e698be4e-6943-4f7c-9614-07e7f81c2265)
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-6: pulp: pulpcore.tasking.services.worker_watcher:INFO: Worker '22312@centos7-katello-nightly-2.cannolo.example.com' is back online.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-6: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:INFO: Worker '22319@centos7-katello-nightly-2.cannolo.example.com' is back online.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-8: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-7: pulp: pulpcore.tasking.services.worker_watcher:INFO: Worker '22315@centos7-katello-nightly-2.cannolo.example.com' is back online.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-7: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:INFO: Worker '22314@centos7-katello-nightly-2.cannolo.example.com' is back online.
Nov 17 14:54:39 centos7-katello-nightly-2 pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:INFO: Worker 'resource-manager' is back online.

Also available in: Atom PDF