Pulp

The nightly builds of Pulp 2.13 fail to start on RHEL 7. To reproduce, provision a RHEL 7 system and install a nightly build of Pulp 2.13 with pulp packaging. One may need to restart the system after installation - I haven't checked whether this issue is present immediately after installing Pulp.

Here's the relevant packages installed on my test system:

[root@rhel-7-3-pulp-2-13 ~]# rpm -qa | grep -i celery | sort
python-celery-3.1.11-1.el7.noarch
[root@rhel-7-3-pulp-2-13 ~]# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
pulp-docker-admin-extensions-2.4.0-0.1.alpha.git.236.897f4ec.el7.noarch
pulp-docker-plugins-2.4.0-0.1.alpha.git.236.897f4ec.el7.noarch
pulp-ostree-admin-extensions-1.3.0-0.1.alpha.git.140.8c5e220.el7.noarch
pulp-ostree-plugins-1.3.0-0.1.alpha.git.140.8c5e220.el7.noarch
pulp-puppet-admin-extensions-2.13.0-0.1.alpha.git.310.e081257.el7.noarch
pulp-puppet-plugins-2.13.0-0.1.alpha.git.310.e081257.el7.noarch
pulp-python-admin-extensions-2.1.0-0.1.alpha.git.137.69c5a80.el7.noarch
pulp-python-plugins-2.1.0-0.1.alpha.git.137.69c5a80.el7.noarch
pulp-rpm-admin-extensions-2.13.0-0.1.alpha.git.704.8ceb59a.el7.noarch
pulp-rpm-plugins-2.13.0-0.1.alpha.git.704.8ceb59a.el7.noarch
pulp-selinux-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
pulp-server-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
python-pulp-client-lib-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
python-pulp-common-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
python-pulp-docker-common-2.4.0-0.1.alpha.git.236.897f4ec.el7.noarch
python-pulp-oid_validation-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
python-pulp-ostree-common-1.3.0-0.1.alpha.git.140.8c5e220.el7.noarch
python-pulp-puppet-common-2.13.0-0.1.alpha.git.310.e081257.el7.noarch
python-pulp-python-common-2.1.0-0.1.alpha.git.137.69c5a80.el7.noarch
python-pulp-repoauth-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch
python-pulp-rpm-common-2.13.0-0.1.alpha.git.704.8ceb59a.el7.noarch
python-pulp-streamer-2.13.0-0.1.alpha.git.18666.4d00bd6.el7.noarch

These are the services that have failed:

[root@rhel-7-3-pulp-2-13 ~]# systemctl --state failed
  UNIT                          LOAD   ACTIVE SUB    DESCRIPTION
● pulp_resource_manager.service loaded failed failed Pulp Resource Manager
● pulp_worker-0.service         loaded failed failed Pulp Worker #0

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

journalctl shows that the root cause is an issue with celery:

Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: Traceback (most recent call last):
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/bin/celery", line 9, in <module>
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: main()
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: cmd.execute_from_commandline(argv)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: super(CeleryCommand, self).execute_from_commandline(argv)))
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: return self.handle_argv(self.prog_name, argv[1:])
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: return self.execute(command, argv)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: return self(*args, **options)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 269, in __call__
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: ret = self.run(*args, **kwargs)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: state_db=self.node_format(state_db, hostname), **kwargs
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 100, in __init__
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: self.setup_instance(**self.prepare_args(**kwargs))
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 141, in setup_instance
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: self.blueprint.apply(self, **kwargs)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 214, in apply
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: for S in self._finalize_steps(steps):
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 257, in _finalize_steps
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: raise KeyError('unknown bootstep: %s' % exc)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[1050]: KeyError: u"unknown bootstep: 'celery.worker.components:Timer'"
Feb 15 15:06:00 rhel-7-3-pulp-2-13 pulp[1224]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 15 15:06:00 rhel-7-3-pulp-2-13 pulp[1226]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: Traceback (most recent call last):
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/bin/celery", line 9, in <module>
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: main()
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: cmd.execute_from_commandline(argv)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: super(CeleryCommand, self).execute_from_commandline(argv)))
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: return self.handle_argv(self.prog_name, argv[1:])
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: return self.execute(command, argv)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: return self(*args, **options)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 269, in __call__
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: ret = self.run(*args, **kwargs)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: state_db=self.node_format(state_db, hostname), **kwargs
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 100, in __init__
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: self.setup_instance(**self.prepare_args(**kwargs))
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 141, in setup_instance
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: self.blueprint.apply(self, **kwargs)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 214, in apply
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: for S in self._finalize_steps(steps):
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 257, in _finalize_steps
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: raise KeyError('unknown bootstep: %s' % exc)
Feb 15 15:06:00 rhel-7-3-pulp-2-13 celery[976]: KeyError: u"unknown bootstep: 'celery.worker.components:Timer'"
Feb 15 15:06:00 rhel-7-3-pulp-2-13 qpidd[963]: 2017-02-15 15:06:00 [Store] notice Linear Store: Store module initialized; store-dir=/var/lib/qpidd/.qpidd
Feb 15 15:06:01 rhel-7-3-pulp-2-13 qpidd[963]: 2017-02-15 15:06:01 [Network] notice Listening on TCP/TCP6 port 5672
Feb 15 15:06:01 rhel-7-3-pulp-2-13 systemd[1]: pulp_worker-0.service: main process exited, code=exited, status=1/FAILURE
Feb 15 15:06:01 rhel-7-3-pulp-2-13 systemd[1]: Unit pulp_worker-0.service entered failed state.
Feb 15 15:06:01 rhel-7-3-pulp-2-13 systemd[1]: pulp_worker-0.service failed.
Feb 15 15:06:01 rhel-7-3-pulp-2-13 systemd[1]: pulp_resource_manager.service: main process exited, code=exited, status=1/FAILURE
Feb 15 15:06:01 rhel-7-3-pulp-2-13 systemd[1]: Unit pulp_resource_manager.service entered failed state.
Feb 15 15:06:01 rhel-7-3-pulp-2-13 systemd[1]: pulp_resource_manager.service failed.