Issue #2684
Updated by Ichimonji10 over 7 years ago
<code>pulp-manage-db</code> should refuse to run when any of pulp_celerybeat, pulp_resource_manager or pulp_workers are running, or when mongod is not running. It has built-in checks that enforce this behaviour, and the work on that is documented in https://pulp.plan.io/issues/2186.
I've written some tests that verify this behaviour. The negative tests are as follows:
1. Do one of the following:
* Stop mongod.
* Let all Pulp services run.
* Stop pulp_resource_manager and pulp_workers, but not pulp_celerybeat.
* Stop pulp_celerybeat and pulp_resource_manager, but not pulp_workers.
* Stop pulp_celerybeat and pulp_workers, but no pulp_pulp_resource_manager.
2. Run <code>pulp-manage-db</code>. Assert a non-zero exit code is returned. (A non-zero exit code indicates that <code>pulp-manage-db</code> refused to run.)
In my manual testing, I've found that the tests where pulp_resource_manager and pulp_workers are left running are especially likely to fail. However, the failures aren't completely consistent. Sometimes, all of the tests will pass.
Here's a summary of results from a particularly bad run. For context, <code>test_required_stopped</code> stops mongod and <code>test_conflicting_running</code> lets all Pulp services run. You can intuit the meaning of the remaining test names.
<pre>============================= ======== ======== ======== ======== ==========
Test F24-2.12 F24-2.13 F25-2.12 F25-2.13 RHEL7-2.12
============================= ======== ======== ======== ======== ==========
test_required_stopped ✓ ✓ ✓ ✓ ✓
test_conflicting_running ✓ ✓ ✓ ✓ ✓
test_celerybeat_running ✓ ✓ ✓ ✓ ✓
test_resource_manager_running ✗ ✗ ✗ ✗ ✗
test_workers_running ✗ ✓ ✗ ✓ ✗
============================= ======== ======== ======== ======== ==========
</pre>
In an attempt to figure out why this is so, I've set the tests to capture the responses from <code>pulp-manage-db</code> and <code>systemctl status</code>. Funnily, <code>pulp-manage-db</code> sometimes prints to stdout, and always prints to stderr.
A concrete example would be helpful. The following is the result of the <code>test_workers_running</code> test from the RHEL 7 2.12 system, above. Here stdout:
<pre>The following processes might still be running:
reserved_resource_worker-0@rhel-7-3-pulp-2-12
Please wait 1 seconds while Pulp confirms this.
</pre>
Note that Pulp *did* correctly count down for 90 seconds, above. It just doesn't show up in the prettified output as interpreted by a terminal or by IDLE. Here's stderr:
<pre>Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch
Loading unit model: python_package = pulp_python.plugins.models:Package
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, ostree, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, python_package, srpm, rpm, distribution, docker_image]
Updating the database with types [puppet_module, docker_tag, ostree, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, python_package, distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 27
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_python.plugins.migrations is up to date at version 1
Migration package pulp_rpm.plugins.migrations is up to date at version 39
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model:rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch
Loading unit model: python_package = pulp_python.plugins.models:Package
Database migrations complete.
</pre>
So far, everything looks totally normal. What about <code>systemctl status</code> Is there any funny business in there? Yes! <code>pulp_worker-0.service</code> is active. That's not good.
<pre>● rhel-7-3-pulp-2-12
State: running
Jobs: 0 queued
Failed: 0 units
Since: Thu 2017-03-30 23:44:46 EDT; 7h left
CGroup: /
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
├─user.slice
│ └─user-0.slice
│ ├─session-25.scope
│ │ ├─5563 sshd: root@notty
│ │ └─5566 -bash
│ ├─session-21.scope
│ │ ├─5145 sshd: root@notty
│ │ └─5149 -bash
│ └─session-17.scope
│ ├─4247 sshd: root@notty
│ ├─5160 -bash
│ ├─5614 bash -c cd /root && /usr/bin/systemctl status
│ └─5619 /usr/bin/systemctl status
└─system.slice
├─pulp_worker-0.service
│ ├─5469 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=5
│ └─5547 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=5
├─mongod.service
│ └─5291 /usr/bin/mongod --quiet -f /etc/mongod.conf run
├─httpd.service
│ ├─4443 /usr/sbin/httpd -DFOREGROUND
│ ├─4456 (wsgi:pulp) -DFOREGROUND
│ ├─4457 (wsgi:pulp) -DFOREGROUND
│ ├─4458 (wsgi:pulp) -DFOREGROUND
│ ├─4459 (wsgi:pulp-cont -DFOREGROUND
│ ├─4460 (wsgi:pulp-cont -DFOREGROUND
│ ├─4461 (wsgi:pulp-cont -DFOREGROUND
│ ├─4462 (wsgi:pulp_forg -DFOREGROUND
│ ├─4463 (wsgi:pulp_forg -DFOREGROUND
│ ├─4464 (wsgi:pulp_forg -DFOREGROUND
│ ├─4465 /usr/sbin/httpd -DFOREGROUND
│ ├─4467 /usr/sbin/httpd -DFOREGROUND
│ ├─4468 /usr/sbin/httpd -DFOREGROUND
│ ├─4469 /usr/sbin/httpd -DFOREGROUND
│ └─4470 /usr/sbin/httpd -DFOREGROUND
├─rhnsd.service
│ └─1011 rhnsd
├─pulp_streamer.service
│ └─982 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac
├─tuned.service
│ └─978 /usr/bin/python -Es /usr/sbin/tuned -l -P
├─rhsmcertd.service
│ └─981 /usr/bin/rhsmcertd
├─rsyslog.service
│ └─974 /usr/sbin/rsyslogd -n
├─qpidd.service
│ └─972 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
├─squid.service
│ ├─1053 /usr/sbin/squid -f /etc/squid/squid.conf
│ ├─1058 (squid-1) -f /etc/squid/squid.conf
│ └─1111 (logfile-daemon) /var/log/squid/access.log
├─postfix.service
│ ├─1579 /usr/libexec/postfix/master -w
│ ├─1584 pickup -l -t unix -u
│ └─1585 qmgr -l -t unix -u
├─sshd.service
│ └─1000 /usr/sbin/sshd
├─NetworkManager.service
│ ├─663 /usr/sbin/NetworkManager --no-daemon
│ └─767 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-3ec3ad7d-e0ef-4a81-911f-d7a77852abbe-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
├─firewalld.service
│ └─662 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
├─crond.service
│ └─645 /usr/sbin/crond -n
├─polkit.service
│ └─628 /usr/lib/polkit-1/polkitd --no-debug
├─chronyd.service
│ └─631 /usr/sbin/chronyd
├─systemd-logind.service
│ └─624 /usr/lib/systemd/systemd-logind
├─dbus.service
│ └─615 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
├─auditd.service
│ └─596 /sbin/auditd -n
├─systemd-udevd.service
│ └─492 /usr/lib/systemd/systemd-udevd
├─lvm2-lvmetad.service
│ └─483 /usr/sbin/lvmetad -f
├─system-getty.slice
│ └─getty@tty1.service
│ └─651 /sbin/agetty --noclear tty1 linux
└─systemd-journald.service
└─465 /usr/lib/systemd/systemd-journald
</pre>
Here's the packages installed on this particular system:
<pre># rpm -qa | grep -i pulp | sort
pulp-admin-client-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-docker-plugins-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-ostree-admin-extensions-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-ostree-plugins-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-puppet-admin-extensions-2.12.3-0.1.alpha.git.1.6953897.el7.noarch
pulp-puppet-plugins-2.12.3-0.1.alpha.git.1.6953897.el7.noarch
pulp-python-admin-extensions-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-python-plugins-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-rpm-admin-extensions-2.12.3-0.1.alpha.git.11.d8adbfa.el7.noarch
pulp-rpm-plugins-2.12.3-0.1.alpha.git.11.d8adbfa.el7.noarch
pulp-selinux-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
pulp-server-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
python-pulp-client-lib-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
python-pulp-common-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
python-pulp-docker-common-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
python-pulp-oid_validation-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
python-pulp-ostree-common-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
python-pulp-puppet-common-2.12.3-0.1.alpha.git.1.6953897.el7.noarch
python-pulp-python-common-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
python-pulp-repoauth-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
python-pulp-rpm-common-2.12.3-0.1.alpha.git.11.d8adbfa.el7.noarch
python-pulp-streamer-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch
# rpm -qa | grep -i mongo | sort
mongodb-2.6.12-4.el7.x86_64
mongodb-server-2.6.12-4.el7.x86_64
python-mongoengine-0.10.5-1.el7.noarch
python-pymongo-3.2-1.el7.x86_64
python-pymongo-gridfs-3.2-1.el7.x86_64
</pre>
.