Pulp

We've had bugs before where task status timestamps and updates were not working in strange ways. At that time we learned that the issue was the TaskStatus was using an unexpected task_id and updating the wrong TaskStatus. The unexpected case occurred when we call a celery task synchronously as a function from within another celery task. I filed this issue at that time in Celery upstream[0], and there isn't a clear resolution. For our purposes, I had intended that we wouldn't perform that call pattern anymore, but I found at least one case in our code that does it. Here is a traceback which shows that:

celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[5056dd3d-3657-404a-a058-a97056cc1df9] succeeded in 0.0640279349973s: None
pulp.server.controllers.repository:ERROR: (14502-62656) Error received removing distributor [56c620fb266b9c36e8190bf1] from repo [433cafcf-66e5-4f4c-aa1b-53f739fdd4de]
pulp.server.controllers.repository:ERROR: (14502-62656) Traceback (most recent call last):
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 515, in delete
pulp.server.controllers.repository:ERROR: (14502-62656)     dist_controller.delete(distributor.repo_id, distributor.distributor_id)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/celery/local.py", line 167, in <lambda>
pulp.server.controllers.repository:ERROR: (14502-62656)     __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 473, in __call__
pulp.server.controllers.repository:ERROR: (14502-62656)     return super(Task, self).__call__(*args, **kwargs)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 103, in __call__
pulp.server.controllers.repository:ERROR: (14502-62656)     return super(PulpTask, self).__call__(*args, **kwargs)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 438, in __protected_call__
pulp.server.controllers.repository:ERROR: (14502-62656)     return orig(self, *args, **kwargs)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 420, in __call__
pulp.server.controllers.repository:ERROR: (14502-62656)     return self.run(*args, **kwargs)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/server/controllers/distributor.py", line 133, in delete
pulp.server.controllers.repository:ERROR: (14502-62656)     dist_instance.distributor_removed(repo.to_transfer_repo(), call_config)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/plugins/file/distributor.py", line 55, in distributor_removed
pulp.server.controllers.repository:ERROR: (14502-62656)     self.unpublish_repo(repo, config)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/iso_distributor/distributor.py", line 76, in unpublish_repo
pulp.server.controllers.repository:ERROR: (14502-62656)     publish.remove_repository_protection(transfer_repo.repo_obj)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/iso_distributor/publish.py", line 78, in remove_repository_protection
pulp.server.controllers.repository:ERROR: (14502-62656)     protected_repo_utils.delete_protected_repo(relative_path)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/repoauth/protected_repo_utils.py", line 64, in delete_protected_repo
pulp.server.controllers.repository:ERROR: (14502-62656)     f.remove_protected_repo_path(repo_relative_path)
pulp.server.controllers.repository:ERROR: (14502-62656)   File "/usr/lib/python2.6/site-packages/pulp/repoauth/protected_repo_utils.py", line 179, in remove_protected_repo_path

[0]: https://github.com/celery/celery/issues/2494