Actions
Issue #3882
closedpulp_resource_manager is unable to start in pulp 2.16.4 beta due to celery issue 3620.
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.16.4
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 40
Quarter:
Description
Pulp 2.16.4 has this issue https://github.com/celery/celery/issues/3620. It was fixed in celery 4.1. You need multiple tasks in the queue upon initial startup to reproduce this issue. You can also see more discussion about this issue here: https://github.com/celery/celery/pull/3752 and here https://github.com/celery/celery/issues/4109#issuecomment-314367686. I implemented this fix in my lab environment and it resolved my issue.
- Problem with pulp_resource_manager service
Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1486, in apply_async
Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) self._quick_put((TASK, (result._job, None, func, args, kwds)))
Jul 25 06:10:03 pulp-lab pulp[17698]: celery.worker:CRITICAL: (17698-58784) TypeError: 'NoneType' object is not callable
- Here is the fix that was ultimately implemented which moves the consumption step below the object registration in worker/loops.py.
- consumer.consume()
- obj.on_ready()
obj.controller.register_with_event_loop(hub)
obj.register_with_event_loop(hub)
+ consumer.consume()
+ obj.on_ready()
Updated by mihai.ibanescu@gmail.com over 6 years ago
I have also observed this while testing the debian changes.
pulp-admin tasks list shows nothing, yet the resource manager fails to start.
Updated by daviddavis over 6 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 40
Added by dalley over 6 years ago
Added by dalley over 6 years ago
Revision c79fe114 | View on GitHub
Add patch for celery worker crashing issue
Updated by dalley over 6 years ago
- Status changed from NEW to MODIFIED
- Assignee set to dalley
Updated packages should be released soon.
Updated by daviddavis over 6 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Add patch for celery worker crashing issue
closes #3882 https://pulp.plan.io/issues/3882