Pulp

The Pulp 2.15 nightlies are broken on Fedora 27 due to a pulp_streamer import error. To reproduce, just install Pulp 2.15 nightly on Fedora 27, using the pulp-ci pulp_server.yaml Ansible playbook. That's it. Pulp will fail to start. Even simple tasks like HTTP POST calls to /pulp/api/v2/actions/login/ will fail with an HTTP 500 error.

Only pulp_streamer obviously fails:

[root@fedora-27-pulp-2-15-nightly ~]# systemctl --state failed
  UNIT                  LOAD   ACTIVE SUB    DESCRIPTION                           
● mcelog.service        loaded failed failed Machine Check Exception Logging Daemon
● pulp_streamer.service loaded failed failed The Pulp lazy content loading streamer

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Disabling SELinux and restarting pulp_streamer has no effect. There don't appear to be any SELinux denials in /var/log/audit/audit.log.

Here's a snippet from the output of journalctl -p 0..3 --boot=-0. The full output is attached.

Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: /usr/lib64/python2.7/lib-dynload/gdbmmodule.so: undefined symbol: gdbm_errno
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1202]: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: /usr/lib64/python2.7/lib-dynload/gdbmmodule.so: undefined symbol: gdbm_errno
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176) Signal handler <function initialize_worker at 0x7fe24c365668> raised: ImportError('/usr/lib64/python2.7/lib-dynload/gdbmmodule.so: undefined symbol: gdbm_errno',)
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176) Traceback (most recent call last):
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 227, in send
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     response = receiver(signal=self, sender=sender, **named)
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 168, in initialize_worker
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     initialization.initialize()
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 36, in initialize
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     plugin_api.initialize()
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 55, in initialize
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     loading.load_plugins_from_entry_point(*entry_point)
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/loading.py", line 60, in load_plugins_from_entry_point
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     cls, cfg = entry_point.load()()
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2402, in load
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     return self.resolve()
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2408, in resolve
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     module = __import__(self.module_name, fromlist=['__name__'], level=0)
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/distributors/distributor.py", line 7, in <module>
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     from pulp_puppet.plugins.distributors import configuration, publish
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)   File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/distributors/publish.py", line 1, in <module>
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176)     import gdbm
Jan 16 15:24:28 fedora-27-pulp-2-15-nightly pulp[1179]: celery.utils.dispatch.signal:ERROR: (1179-34176) ImportError: /usr/lib64/python2.7/lib-dynload/gdbmmodule.so: undefined symbol: gdbm_errno

Here's the relevant packages on the test system:

[root@fedora-27-pulp-2-15-nightly ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
pulp-deb-admin-extensions-1.6.0-0.1.alpha.201801160537gite5c6871.fc27.noarch
pulp-docker-admin-extensions-3.1.0-0.1.alpha.201801160522gitd3b6055.fc27.noarch
pulp-docker-plugins-3.1.0-0.1.alpha.201801160522gitd3b6055.fc27.noarch
pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201801160523git5e12dac.fc27.noarch
pulp-ostree-plugins-1.4.0-0.1.alpha.201801160523git5e12dac.fc27.noarch
pulp-puppet-admin-extensions-2.15.0-0.1.alpha.201801160525gitc026b1a.fc27.noarch
pulp-puppet-plugins-2.15.0-0.1.alpha.201801160525gitc026b1a.fc27.noarch
pulp-puppet-tools-2.15.0-0.1.alpha.201801160525gitc026b1a.fc27.noarch
pulp-python-admin-extensions-2.1.0-0.1.alpha.201801160520git5fd2751.fc27.noarch
pulp-python-plugins-2.1.0-0.1.alpha.201801160520git5fd2751.fc27.noarch
pulp-rpm-admin-extensions-2.15.0-0.1.alpha.201801160528git4933d92.fc27.noarch
pulp-rpm-plugins-2.15.0-0.1.alpha.201801160528git4933d92.fc27.noarch
pulp-selinux-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
pulp-server-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
python-pulp-bindings-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
python-pulp-client-lib-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
python-pulp-common-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
python-pulp-deb-common-1.6.0-0.1.alpha.201801160537gite5c6871.fc27.noarch
python-pulp-docker-common-3.1.0-0.1.alpha.201801160522gitd3b6055.fc27.noarch
python-pulp-oid_validation-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
python-pulp-ostree-common-1.4.0-0.1.alpha.201801160523git5e12dac.fc27.noarch
python-pulp-puppet-common-2.15.0-0.1.alpha.201801160525gitc026b1a.fc27.noarch
python-pulp-python-common-2.1.0-0.1.alpha.201801160520git5fd2751.fc27.noarch
python-pulp-repoauth-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch
python-pulp-rpm-common-2.15.0-0.1.alpha.201801160528git4933d92.fc27.noarch
python-pulp-streamer-2.15.0-0.1.alpha.201801160530git5711121.fc27.noarch