Project

Profile

Help

Issue #3882

closed

pulp_resource_manager is unable to start in pulp 2.16.4 beta due to celery issue 3620.

Added by dustinmcnabb over 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.16.4
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 40
Quarter:

Description

Pulp 2.16.4 has this issue https://github.com/celery/celery/issues/3620. It was fixed in celery 4.1. You need multiple tasks in the queue upon initial startup to reproduce this issue. You can also see more discussion about this issue here: https://github.com/celery/celery/pull/3752 and here https://github.com/celery/celery/issues/4109#issuecomment-314367686. I implemented this fix in my lab environment and it resolved my issue.

  1. Problem with pulp_resource_manager service
    Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1486, in apply_async
    Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) self._quick_put((TASK, (result._job, None, func, args, kwds)))
    Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) TypeError: 'NoneType' object is not callable
  1. Here is the fix that was ultimately implemented which moves the consumption step below the object registration in worker/loops.py.
    - consumer.consume()
    - obj.on_ready()
    obj.controller.register_with_event_loop(hub)
    obj.register_with_event_loop(hub)
    + consumer.consume()
    + obj.on_ready()

Also available in: Atom PDF