Project

Profile

Help

Issue #3882

pulp_resource_manager is unable to start in pulp 2.16.4 beta due to celery issue 3620.

Added by dustinmcnabb over 3 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.16.4
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 40
Quarter:

Description

Pulp 2.16.4 has this issue https://github.com/celery/celery/issues/3620. It was fixed in celery 4.1. You need multiple tasks in the queue upon initial startup to reproduce this issue. You can also see more discussion about this issue here: https://github.com/celery/celery/pull/3752 and here https://github.com/celery/celery/issues/4109#issuecomment-314367686. I implemented this fix in my lab environment and it resolved my issue.

  1. Problem with pulp_resource_manager service
    Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1486, in apply_async
    Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) self._quick_put((TASK, (result._job, None, func, args, kwds)))
    Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) TypeError: 'NoneType' object is not callable
  1. Here is the fix that was ultimately implemented which moves the consumption step below the object registration in worker/loops.py.
    - consumer.consume()
    - obj.on_ready()
    obj.controller.register_with_event_loop(hub)
    obj.register_with_event_loop(hub)
    + consumer.consume()
    + obj.on_ready()

Associated revisions

Revision c79fe114 View on GitHub
Added by dalley over 3 years ago

Add patch for celery worker crashing issue

closes #3882 https://pulp.plan.io/issues/3882

Revision c79fe114 View on GitHub
Added by dalley over 3 years ago

Add patch for celery worker crashing issue

closes #3882 https://pulp.plan.io/issues/3882

History

#1 Updated by mihai.ibanescu@gmail.com over 3 years ago

I have also observed this while testing the debian changes.

pulp-admin tasks list shows nothing, yet the resource manager fails to start.

#2 Updated by daviddavis over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 40

#3 Updated by daviddavis over 3 years ago

  • Platform Release set to 2.16.4

#4 Updated by dalley over 3 years ago

  • Status changed from NEW to MODIFIED
  • Assignee set to dalley

Updated packages should be released soon.

PR: https://github.com/pulp/pulp-packaging/pull/50

#5 Updated by daviddavis over 3 years ago

  • Status changed from MODIFIED to 5

#6 Updated by daviddavis over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#7 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF