Pulp

I tried to upgrade from 2.4->2.6

1. Without stopping services I did a yum update
2. Tried to restart service
3. Tried to stop service
got this
[root@mgmt12 ~]# service pulp_workers restart && service pulp_celerybeat restart && service pulp_resource_manager restart
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
Traceback (most recent call last):
File "/usr/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 761, in handle_argv
return self.execute(command, argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 693, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 97, in run_from_argv
[command] + argv, prog_name,
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 206, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 380, in restart
self._stop_nodes(p, cmd, retry=2, callback=on_node_shutdown)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 362, in stop_nodes
self.shutdown_nodes(self.getpids(p, cmd, callback=callback),
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 336, in getpids
pid = Pidfile(pidfile).read_pid()
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 179, in read_pid
'pidfile {0.path} contents invalid.'.format(self))
File "/usr/lib64/python2.6/contextlib.py", line 34, inexit_
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 732, in ignore_errno
yield
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 169, in read_pid
with open(self.path, 'r') as fh:
IOError: [Errno 13] Permission denied: u'/var/run/pulp/reserved_resource_worker-0.pid'
[root@mgmt12 ~]#
[root@mgmt12 ~]#
[root@mgmt12 ~]#
[root@mgmt12 ~]# service pulp_workers restart
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
Traceback (most recent call last):
File "/usr/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 761, in handle_argv
return self.execute(command, argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 693, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 97, in run_from_argv
[command] + argv, prog_name,
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 206, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 380, in restart
self._stop_nodes(p, cmd, retry=2, callback=on_node_shutdown)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 362, in stop_nodes
self.shutdown_nodes(self.getpids(p, cmd, callback=callback),
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 336, in getpids
pid = Pidfile(pidfile).read_pid()
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 179, in read_pid
'pidfile {0.path} contents invalid.'.format(self))
File "/usr/lib64/python2.6/contextlib.py", line 34, inexit_
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 732, in ignore_errno
yield
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 169, in read_pid
with open(self.path, 'r') as fh:
IOError: [Errno 13] Permission denied: u'/var/run/pulp/reserved_resource_worker-0.pid'
[root@mgmt12 ~]# service pulp_workers stop
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
Traceback (most recent call last):
File "/usr/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 761, in handle_argv
return self.execute(command, argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 693, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 97, in run_from_argv
[command] + argv, prog_name,
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 206, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 387, in stopwait
return self._stop_nodes(p, cmd, retry=2)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 362, in _stop_nodes
self.shutdown_nodes(self.getpids(p, cmd, callback=callback),
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 336, in getpids
pid = Pidfile(pidfile).read_pid()
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 179, in read_pid
'pidfile {0.path} contents invalid.'.format(self))
File "/usr/lib64/python2.6/contextlib.py", line 34, in exit
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 732, in ignore_errno
yield
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 169, in read_pid
with open(self.path, 'r') as fh:
IOError: [Errno 13] Permission denied: u'/var/run/pulp/reserved_resource_worker-0.pid'
[root@mgmt12 ~]#
[root@mgmt12 ~]#
[root@mgmt12 ~]# ls laZ /var/run/pulp
drwxr-sr-x. apache root unconfined_u:object_r:var_run_t:s0 .
drwxr-xr-x. root root system_u:object_r:var_run_t:s0 ..
-rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 celerybeat.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-0.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-1.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-2.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-3.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-4.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-5.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-6.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-7.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 resource_manager.pid

Pulp 2.4 had a fairly lax SELinux policy. The new SELinux policy introduced in 2.5 significantly restricts what celery process running as pulp_workers, pulp_celerybeat, and pulp_resource_manager can do. As a result, if a process was started before an upgrade, it is running in a different context than the process started after the new SELinux policy is applied. As a result the new process is not able to read the PID file of the old process in order to stop it. This is an issue when upgrading from anything before 2.5.

In order to avoid this problem, pulp_workers, pulp_celerybeat, and pulp_resource_manager need to be stopped before the upgrade.

I will add release notes about this.

Decided that we should fix this. On upgrade, we will stop pulp_workers, pulp_celerybeat, pulp_resource_manager. The same will happen when uninstalling completely.

https://github.com/pulp/pulp/pull/1721/