Project

Profile

Help

Issue #2684

Updated by Ichimonji10 almost 7 years ago

<code>pulp-manage-db</code> should refuse to run when any of pulp_celerybeat, pulp_resource_manager or pulp_workers are running, or when mongod is not running. It has built-in checks that enforce this behaviour, and the work on that is documented in https://pulp.plan.io/issues/2186. 

 I've written some tests that verify this behaviour. The negative tests are as follows: 

 1. Do one of the following: 

 * Stop mongod. 
 * Let all Pulp services run. 
 * Stop pulp_resource_manager and pulp_workers, but not pulp_celerybeat. 
 * Stop pulp_celerybeat and pulp_resource_manager, but not pulp_workers. 
 * Stop pulp_celerybeat and pulp_workers, but no pulp_pulp_resource_manager. 

 2. Run <code>pulp-manage-db</code>. Assert a non-zero exit code is returned. (A non-zero exit code indicates that <code>pulp-manage-db</code> refused to run.) 

 In my manual testing, I've found that the tests where pulp_resource_manager and pulp_workers are left running are especially likely to fail. However, the failures aren't completely consistent. Sometimes, all of the tests will pass. 

 Here's a summary of results from a particularly bad run. For context, <code>test_required_stopped</code> stops mongod and <code>test_conflicting_running</code> lets all Pulp services run. You can intuit the meaning of the remaining test names. 

 <pre>=============================    ========    ========    ========    ========    ========== 
 Test                             F24-2.12    F24-2.13    F25-2.12    F25-2.13    RHEL7-2.12 
 =============================    ========    ========    ========    ========    ========== 
 test_required_stopped            ✓           ✓           ✓           ✓           ✓ 
 test_conflicting_running         ✓           ✓           ✓           ✓           ✓ 
 test_celerybeat_running          ✓           ✓           ✓           ✓           ✓ 
 test_resource_manager_running    ✗           ✗           ✗           ✗           ✗ 
 test_workers_running             ✗           ✓           ✗           ✓           ✗ 
 =============================    ========    ========    ========    ========    ========== 
 </pre> 


 In an attempt to figure out why this is so, I've set the tests to capture the responses from <code>pulp-manage-db</code> and <code>systemctl status</code>. Funnily, <code>pulp-manage-db</code> sometimes prints to stdout, and always prints to stderr. 

 A concrete example would be helpful. The following is the result of the <code>test_workers_running</code> test from the RHEL 7 2.12 system, above. Here stdout: 

 <pre>The following processes might still be running: 
	 reserved_resource_worker-0@rhel-7-3-pulp-2-12 
 Please wait 1 seconds while Pulp confirms this. 
 </pre> 

 Note that Pulp *did* correctly count down for 90 seconds, above. It just doesn't show up in the prettified output as interpreted by a terminal or by IDLE. Here's stderr: 

 <pre>Attempting to connect to localhost:27017 
 Attempting to connect to localhost:27017 
 Write concern for Mongo connection: {} 
 Loading content types. 
 Loading type descriptors [] 
 Parsing type descriptors 
 Validating type descriptor syntactic integrity 
 Validating type descriptor semantic integrity 
 Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata 
 Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution 
 Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM 
 Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup 
 Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory 
 Loading unit model: iso = pulp_rpm.plugins.db.models:ISO 
 Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment 
 Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM 
 Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks 
 Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM 
 Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile 
 Loading unit model: docker_blob = pulp_docker.plugins.models:Blob 
 Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest 
 Loading unit model: docker_image = pulp_docker.plugins.models:Image 
 Loading unit model: docker_tag = pulp_docker.plugins.models:Tag 
 Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module 
 Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch 
 Loading unit model: python_package = pulp_python.plugins.models:Package 
 Updating the database with types [] 
 Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, ostree, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, python_package, srpm, rpm, distribution, docker_image] 
 Updating the database with types [puppet_module, docker_tag, ostree, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, python_package, distribution, rpm, srpm, docker_image] 
 Content types loaded. 
 Ensuring the admin role and user are in place. 
 Admin role and user are in place. 
 Beginning database migrations. 
 Migration package pulp.server.db.migrations is up to date at version 27 
 Migration package pulp_docker.plugins.migrations is up to date at version 2 
 Migration package pulp_puppet.plugins.migrations is up to date at version 5 
 Migration package pulp_python.plugins.migrations is up to date at version 1 
 Migration package pulp_rpm.plugins.migrations is up to date at version 39 
 Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata 
 Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution 
 Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM 
 Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup 
 Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory 
 Loading unit model: iso = pulp_rpm.plugins.db.models:ISO 
 Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment 
 Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM 
 Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks 
 Loading unit model:rpm = pulp_rpm.plugins.db.models:RPM 
 Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile 
 Loading unit model: docker_blob = pulp_docker.plugins.models:Blob 
 Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest 
 Loading unit model: docker_image = pulp_docker.plugins.models:Image 
 Loading unit model: docker_tag = pulp_docker.plugins.models:Tag 
 Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module 
 Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch 
 Loading unit model: python_package = pulp_python.plugins.models:Package 
 Database migrations complete. 
 </pre> 

 So far, everything looks totally normal. What about <code>systemctl status</code> Is there any funny business in there? Yes! <code>pulp_worker-0.service</code> is active. That's not good. 

 <pre>● rhel-7-3-pulp-2-12 
     State: running 
      Jobs: 0 queued 
    Failed: 0 units 
    Since: Thu 2017-03-30 23:44:46 EDT; 7h left 
    CGroup: / 
            ├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 
            ├─user.slice 
            │ └─user-0.slice 
            │     ├─session-25.scope 
            │     │ ├─5563 sshd: root@notty  
            │     │ └─5566 -bash 
            │     ├─session-21.scope 
            │     │ ├─5145 sshd: root@notty  
            │     │ └─5149 -bash 
            │     └─session-17.scope 
            │       ├─4247 sshd: root@notty  
            │       ├─5160 -bash 
            │       ├─5614 bash -c cd /root && /usr/bin/systemctl status 
            │       └─5619 /usr/bin/systemctl status 
            └─system.slice 
              ├─pulp_worker-0.service 
              │ ├─5469 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=5 
              │ └─5547 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=5 
              ├─mongod.service 
              │ └─5291 /usr/bin/mongod --quiet -f /etc/mongod.conf run 
              ├─httpd.service 
              │ ├─4443 /usr/sbin/httpd -DFOREGROUND 
              │ ├─4456 (wsgi:pulp)       -DFOREGROUND 
              │ ├─4457 (wsgi:pulp)       -DFOREGROUND 
              │ ├─4458 (wsgi:pulp)       -DFOREGROUND 
              │ ├─4459 (wsgi:pulp-cont -DFOREGROUND 
              │ ├─4460 (wsgi:pulp-cont -DFOREGROUND 
              │ ├─4461 (wsgi:pulp-cont -DFOREGROUND 
              │ ├─4462 (wsgi:pulp_forg -DFOREGROUND 
              │ ├─4463 (wsgi:pulp_forg -DFOREGROUND 
              │ ├─4464 (wsgi:pulp_forg -DFOREGROUND 
              │ ├─4465 /usr/sbin/httpd -DFOREGROUND 
              │ ├─4467 /usr/sbin/httpd -DFOREGROUND 
              │ ├─4468 /usr/sbin/httpd -DFOREGROUND 
              │ ├─4469 /usr/sbin/httpd -DFOREGROUND 
              │ └─4470 /usr/sbin/httpd -DFOREGROUND 
              ├─rhnsd.service 
              │ └─1011 rhnsd 
              ├─pulp_streamer.service 
              │ └─982 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac 
              ├─tuned.service 
              │ └─978 /usr/bin/python -Es /usr/sbin/tuned -l -P 
              ├─rhsmcertd.service 
              │ └─981 /usr/bin/rhsmcertd 
              ├─rsyslog.service 
              │ └─974 /usr/sbin/rsyslogd -n 
              ├─qpidd.service 
              │ └─972 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf 
              ├─squid.service 
              │ ├─1053 /usr/sbin/squid -f /etc/squid/squid.conf 
              │ ├─1058 (squid-1) -f /etc/squid/squid.conf 
              │ └─1111 (logfile-daemon) /var/log/squid/access.log 
              ├─postfix.service 
              │ ├─1579 /usr/libexec/postfix/master -w 
              │ ├─1584 pickup -l -t unix -u 
              │ └─1585 qmgr -l -t unix -u 
              ├─sshd.service 
              │ └─1000 /usr/sbin/sshd 
              ├─NetworkManager.service 
              │ ├─663 /usr/sbin/NetworkManager --no-daemon 
              │ └─767 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-3ec3ad7d-e0ef-4a81-911f-d7a77852abbe-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 
              ├─firewalld.service 
              │ └─662 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid 
              ├─crond.service 
              │ └─645 /usr/sbin/crond -n 
              ├─polkit.service 
              │ └─628 /usr/lib/polkit-1/polkitd --no-debug 
              ├─chronyd.service 
              │ └─631 /usr/sbin/chronyd 
              ├─systemd-logind.service 
              │ └─624 /usr/lib/systemd/systemd-logind 
              ├─dbus.service 
              │ └─615 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 
              ├─auditd.service 
              │ └─596 /sbin/auditd -n 
              ├─systemd-udevd.service 
              │ └─492 /usr/lib/systemd/systemd-udevd 
              ├─lvm2-lvmetad.service 
              │ └─483 /usr/sbin/lvmetad -f 
              ├─system-getty.slice 
              │ └─getty@tty1.service 
              │     └─651 /sbin/agetty --noclear tty1 linux 
              └─systemd-journald.service 
                └─465 /usr/lib/systemd/systemd-journald 
 </pre> 

 Here's the packages installed on this particular system: 

 <pre># rpm    -qa | grep -i pulp | sort 
 pulp-admin-client-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.5.052c506.el7.noarch 
 pulp-docker-plugins-2.3.1-0.1.alpha.git.5.052c506.el7.noarch 
 pulp-ostree-admin-extensions-1.2.2-0.1.alpha.git.3.809be44.el7.noarch 
 pulp-ostree-plugins-1.2.2-0.1.alpha.git.3.809be44.el7.noarch 
 pulp-puppet-admin-extensions-2.12.3-0.1.alpha.git.1.6953897.el7.noarch 
 pulp-puppet-plugins-2.12.3-0.1.alpha.git.1.6953897.el7.noarch 
 pulp-python-admin-extensions-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch 
 pulp-python-plugins-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch 
 pulp-rpm-admin-extensions-2.12.3-0.1.alpha.git.11.d8adbfa.el7.noarch 
 pulp-rpm-plugins-2.12.3-0.1.alpha.git.11.d8adbfa.el7.noarch 
 pulp-selinux-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 pulp-server-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 python-isodate-0.5.0-4.pulp.el7.noarch 
 python-kombu-3.0.33-6.pulp.el7.noarch 
 python-pulp-bindings-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 python-pulp-client-lib-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 python-pulp-common-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 python-pulp-docker-common-2.3.1-0.1.alpha.git.5.052c506.el7.noarch 
 python-pulp-oid_validation-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 python-pulp-ostree-common-1.2.2-0.1.alpha.git.3.809be44.el7.noarch 
 python-pulp-puppet-common-2.12.3-0.1.alpha.git.1.6953897.el7.noarch 
 python-pulp-python-common-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch 
 python-pulp-repoauth-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 python-pulp-rpm-common-2.12.3-0.1.alpha.git.11.d8adbfa.el7.noarch 
 python-pulp-streamer-2.12.3-0.1.alpha.git.1198.db2de38.el7.noarch 
 # rpm    -qa | grep -i mongo | sort 
 mongodb-2.6.12-4.el7.x86_64 
 mongodb-server-2.6.12-4.el7.x86_64 
 python-mongoengine-0.10.5-1.el7.noarch 
 python-pymongo-3.2-1.el7.x86_64 
 python-pymongo-gridfs-3.2-1.el7.x86_64 
 </pre>

Back