Pulp

The feature where pulp-manage-db waits to run until other processes are exited uses the records that back the status API. In 2.12, for the pulp_resource_manager and pulp_workers to have their presence recorded as status API records the pulp_celerybeat process must be running. In 2.13+ each process records its own status API records so it shouldn't be dependant on other processes to be running.

All ^ is to ask, when pulp-manage-db continues when it shouldn't what do the status API records show? Since 2.13 is the current GA maybe this could be rerun w/ only 2.13 and include the output of the status API in cases where it fails to wait even though it should?

All ^ is to ask, when pulp-manage-db continues when it shouldn't what do the status API records show?

Is executing pulp-admin status an appropriate way to discover this information? Or is there a better way?

Ichimonji10 wrote:

All ^ is to ask, when pulp-manage-db continues when it shouldn't what do the status API records show?

Is executing pulp-admin status an appropriate way to discover this information? Or is there a better way?

pulp-admin status is good but it also does some filtering. Really getting the db records with the mongo query db.workers.find().pretty() would be the best. For example I get this:

> db.workers.find().pretty()
{
        "_id" : "scheduler@dev",
        "last_heartbeat" : ISODate("2017-05-12T16:12:39.183Z")
}
{
        "_id" : "reserved_resource_worker-3@dev",
        "last_heartbeat" : ISODate("2017-05-12T16:12:39.188Z")
}
{
        "_id" : "reserved_resource_worker-0@dev",
        "last_heartbeat" : ISODate("2017-05-12T16:12:39.357Z")
}
{
        "_id" : "reserved_resource_worker-2@dev",
        "last_heartbeat" : ISODate("2017-05-12T16:12:39.196Z")
}
{
        "_id" : "reserved_resource_worker-1@dev",
        "last_heartbeat" : ISODate("2017-05-12T16:12:39.202Z")
}
{
        "_id" : "resource_manager@dev",
        "last_heartbeat" : ISODate("2017-05-12T16:12:41.573Z")
}

Neato. That makes more sense. So is this an appropriate command to run?

mongo pulp_database --eval db.workers.find().pretty()

(/me just tries it and sees what happens.)

To reproduce this bug and get some additional debugging output, did the following:

1. Modify the Pulp Smash source code.
2. Run the modified test.

The modifications consisted of the following several changes to pulp_smash.tests.platform.cli.test_pulp_manage_db.NegativeTestCase, like the following (this is not a full diff):

diff --git a/pulp_smash/tests/platform/cli/test_pulp_manage_db.py b/pulp_smash/tests/platform/cli/test_pulp_manage_db.py
index 10b180b..fb02726 100644
--- a/pulp_smash/tests/platform/cli/test_pulp_manage_db.py
+++ b/pulp_smash/tests/platform/cli/test_pulp_manage_db.py
@@ -106,11 +121,16 @@ class NegativeTestCase(BaseTestCase):

         This test targets `Pulp #2684 <https://pulp.plan.io/issues/2684>`_.
         """
-        if selectors.bug_is_untestable(2684, self.cfg.version):
-            self.skipTest('https://pulp.plan.io/issues/2684')
         cli.GlobalServiceManager(config.get_config()).stop((
             CONFLICTING_SERVICES.difference(('pulp_workers',))
         ))
+        print('test_workers_running')
+        print(cli.Client(self.cfg).run((
+            'mongo', 'pulp_database', '--eval',
+            'db.workers.find().pretty()'
+        )).stdout)
         self._do_test()

     def _do_test(self):

The modified tests generally do the following:

1. Stop some number of services.
2. Print the newly inserted debugging output.
3. Execute runuser --shell /bin/sh --command pulp-manage-db - apache, and assert that the command fails. If the command succeeds, print some additional debugging output.

I ran these modified tests against a matrix of nine systems, where one axis is F24, F25 and RHEL 7, and the other axis is Pulp 2.13 (stable), Pulp 2.13 (nightly), and Pulp 2.14 (nightly). I was able to reproduce test failures with test_resource_manager_running on at least the following. Note that it may be possible to reproduce these test failures on the platforms marked with an ✗, too. I'm just showing what I was able to reproduce with a limited number (~3) of test runs.

===================  ===  ===  =====
Pulp Version         F24  F25  RHEL7
===================  ===  ===  =====
Pulp 2.13 (stable)   ✓    ✓    ✗
Pulp 2.13 (nightly)  ✓    ✗    ✗
Pulp 2.14 (nightly)  ✓    ✓    ✗
===================  ===  ===  =====

I was unable to reproduce the test_workers_running test failure on any platform. The issue may still be there. Again, this is just what I was able to reproduce with a limited number of test runs.

Here's the output of mongo pulp_database --eval 'db.workers.find().pretty()', right before pulp-manage-db executes. The output is similar for all test failures, with the only difference being timestamps. This output is taken from Fedora 24, with Pulp 2.13 stable.

test_resource_manager_running
MongoDB shell version: 3.2.12
connecting to: pulp_database
{
        "_id" : "resource_manager@fedora-24-pulp-2-13-final",
        "last_heartbeat" : ISODate("2017-05-12T17:15:55.451Z")
}

Here's the contents of stderr, from when the pulp-manage-db command ran. This output is taken from the same system.

Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch
Loading unit model: python_package = pulp_python.plugins.models:Package
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, ostree, package_langpacks, erratum, docker_blob, docker_tag, distribution, package_group, package_category, iso, package_environment, drpm, python_package, srpm, rpm, y
um_repo_metadata_file, docker_image, docker_manifest]
Updating the database with types [puppet_module, ostree, package_langpacks, erratum, docker_blob, docker_tag, distribution, package_group, package_category, iso, package_environment, drpm, python_package, srpm, rpm, yum_repo_metadata_file, docker_image, docker_manifes
t]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 28
Migration package pulp_docker.plugins.migrations is up to date at version 3
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_python.plugins.migrations is up to date at version 2
Migration package pulp_rpm.plugins.migrations is up to date at version 39
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: ostree = pulp_ostree.plugins.db.model:Branch
Loading unit model: python_package = pulp_python.plugins.models:Package
Database migrations complete.

Here's the output of stdout, from when systemctl status ran:

● fedora-24-pulp-2-13-final
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Fri 2017-05-12 20:46:39 EDT; 7h left
   CGroup: /
           ├─user.slice
           │ └─user-0.slice
           │   ├─session-44.scope
           │   │ ├─8687 sshd: root [priv]
           │   │ ├─8698 sshd: root@notty 
           │   │ └─8701 -bash
           │   ├─user@0.service
           │   │ └─init.scope
           │   │   ├─1282 /usr/lib/systemd/systemd --user
           │   │   └─1297 (sd-pam)                                                          
           │   ├─session-43.scope
           │   │ ├─8645 sshd: root [priv]
           │   │ ├─8664 sshd: root@notty 
           │   │ └─8669 -bash
           │   └─session-30.scope
           │     ├─5458 sshd: root [priv]
           │     ├─5461 sshd: root@notty 
           │     ├─8133 -bash
           │     ├─8764 -bash
           │     ├─8826 -bash
           │     ├─8841 -bash
           │     ├─8905 -bash
           │     ├─8920 -bash
           │     ├─8979 -bash
           │     ├─9012 -bash
           │     ├─9068 bash -c cd /root && /usr/bin/systemctl status
           │     └─9077 /usr/bin/systemctl status
           ├─init.scope
           │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
           └─system.slice
             ├─lvm2-lvmetad.service
             │ └─562 /usr/sbin/lvmetad -f -t 3600
             ├─rngd.service
             │ └─686 /sbin/rngd -f
             ├─firewalld.service
             │ └─727 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid
             ├─pulp_streamer.service
             │ └─1030 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac
             ├─gssproxy.service
             │ └─697 /usr/sbin/gssproxy -D
             ├─mcelog.service
             │ └─685 /usr/sbin/mcelog --ignorenodev --daemon --foreground
             ├─NetworkManager.service
             │ ├─752 /usr/sbin/NetworkManager --no-daemon
             │ └─955 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-ens3.pid -lf /var/lib/NetworkManager/dhclient-917ba805-2398-3b85-a40d-a22af7988efd-ens3.lease -cf /var/lib/NetworkManager/dhclient-ens3.conf ens3
             ├─dbus.service
             │ └─687 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─httpd.service
             │ ├─7769 /usr/sbin/httpd -DFOREGROUND
             │ ├─7776 (wsgi:pulp)     -DFOREGROUND
             │ ├─7777 (wsgi:pulp)     -DFOREGROUND
             │ ├─7778 (wsgi:pulp)     -DFOREGROUND
             │ ├─7779 (wsgi:pulp-cont -DFOREGROUND
             │ ├─7780 (wsgi:pulp-cont -DFOREGROUND
             │ ├─7781 (wsgi:pulp-cont -DFOREGROUND
             │ ├─7782 (wsgi:pulp_forg -DFOREGROUND
             │ ├─7783 (wsgi:pulp_forg -DFOREGROUND
             │ ├─7784 (wsgi:pulp_forg -DFOREGROUND
             │ ├─7785 /usr/sbin/httpd -DFOREGROUND
             │ ├─7786 /usr/sbin/httpd -DFOREGROUND
             │ ├─7789 /usr/sbin/httpd -DFOREGROUND
             │ ├─7810 /usr/sbin/httpd -DFOREGROUND
             │ └─7811 /usr/sbin/httpd -DFOREGROUND
             ├─sshd.service
             │ └─785 /usr/sbin/sshd
             ├─abrt-xorg.service
             │ └─741 /usr/bin/abrt-dump-journal-xorg -fxtD
             ├─squid.service
             │ ├─ 805 /usr/sbin/squid -f /etc/squid/squid.conf
             │ ├─ 807 (squid-1) -f /etc/squid/squid.conf
             │ └─1000 (logfile-daemon) /var/log/squid/access.log
             ├─abrt-oops.service
             │ └─742 /usr/bin/abrt-dump-journal-oops -fxtD
             ├─system-getty.slice
             │ └─getty@tty1.service
             │   └─743 /sbin/agetty --noclear tty1 linux
             ├─smartd.service
             │ └─692 /usr/sbin/smartd -n -q never
             ├─abrtd.service
             │ └─703 /usr/sbin/abrtd -d -s
             ├─systemd-logind.service
             │ └─696 /usr/lib/systemd/systemd-logind
             ├─crond.service
             │ └─732 /usr/sbin/crond -n
             ├─polkit.service
             │ └─694 /usr/lib/polkit-1/polkitd --no-debug
             ├─mongod.service
             │ └─8888 /usr/bin/mongod -f /etc/mongod.conf run
             ├─pulp_resource_manager.service
             │ ├─8392 /usr/bin/python2 /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid
             │ └─8502 /usr/bin/python2 /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid
             ├─systemd-udevd.service
             │ └─581 /usr/lib/systemd/systemd-udevd
             ├─rsyslog.service
             │ └─684 /usr/sbin/rsyslogd -n
             ├─atd.service
             │ └─731 /usr/sbin/atd -f
             ├─systemd-journald.service
             │ └─543 /usr/lib/systemd/systemd-journald
             ├─auditd.service
             │ └─663 /sbin/auditd
             ├─qpidd.service
             │ └─777 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
             └─chronyd.service
               └─707 /usr/sbin/chronyd

Thanks a lot Ichimonji10!

It sounds like the check here[0] is having still_active_workers = [] when it should have (still_active_workers) > 0.

[0]: https://github.com/pulp/pulp/blob/c9ff2a1fcbe86afaef006329fbbc363d1813a06d/server/pulp/server/db/manage.py#L216

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.