Pulp

When stopping the worker, the running task state is marked as error (or any completed state) but the message of the task remains in the worker queue. After restarting the worker, the task will be re-run. Same thing happen again, if you try to restart the worker over and over.

Steps to Reproduce:

1. Sync any repository and copy the task id (the initial 8 characters), eg. "b3bd24be"
2. While syncing the repository, stop the workers

systemctl stop pulp_workers

3. Check the /va/log/messages. you should see:

pulp: pulp.server.async.tasks:INFO: [30a62ff1] Task failed : [b3bd24be-b805-42b9-9f81-69eb4ddaad55] : Worker terminated abnormally while processing task b3bd24be-b805-42b9-9f81-69eb4ddaad55. Check the logs for details pulp: celery.app.trace:ERROR: [30a62ff1] (15661-20736) Task pulp.server.async.tasks._release_resource[30a62ff1-7d72-497f-ad25-661119917bbb] raised unexpected: AttributeError("'NoneType' object has no attribute 'top'",)

4. Then start the pulp_workers

systemctl start pulp_workers

5. Check the /var/log/messages again. You should see to same task (b3bd24be) runs again:

pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[b3bd24be-b805-42b9-9f81-69eb4ddaad55]
pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[30a62ff1-7d72-497f-ad25-661119917bbb]
pulp: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
...
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/.
pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [b3bd24be] Parsing metadata.

6. Repeat the steps again, the task will re-run again and again.