Project

Profile

Help

Issue #740

service pulp_resource_manager restart doesn't stop resource manager after upgrade

Added by dkliban@redhat.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.6 Beta
Platform Release:
2.6.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

If resource manager is not stopped before Pulp is updated, a call to 'systemctl restart pulp_resource_manager' will not stop the service that was started before the update. A new pulp_resource_manager service is started though.

History

#1 Updated by dkliban@redhat.com over 6 years ago

  • Subject changed from systemctl restart pulp_resource_manager doesn't stop resource manager after upgrade to service pulp_resource_manager restart doesn't stop resource manager after upgrade

#2 Updated by pthomas@redhat.com over 6 years ago

I tried to upgrade from 2.4->2.6

1. Without stopping services I did a yum update
2. Tried to restart service
3. Tried to stop service
got this
[root@mgmt12 ~]# service pulp_workers restart && service pulp_celerybeat restart && service pulp_resource_manager restart
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
Traceback (most recent call last):
File "/usr/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 761, in handle_argv
return self.execute(command, argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 693, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 97, in run_from_argv
[command] + argv, prog_name,
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 206, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 380, in restart
self._stop_nodes(p, cmd, retry=2, callback=on_node_shutdown)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 362, in stop_nodes
self.shutdown_nodes(self.getpids(p, cmd, callback=callback),
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 336, in getpids
pid = Pidfile(pidfile).read_pid()
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 179, in read_pid
'pidfile {0.path} contents invalid.'.format(self))
File "/usr/lib64/python2.6/contextlib.py", line 34, inexit
_
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 732, in ignore_errno
yield
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 169, in read_pid
with open(self.path, 'r') as fh:
IOError: [Errno 13] Permission denied: u'/var/run/pulp/reserved_resource_worker-0.pid'
[root@mgmt12 ~]#
[root@mgmt12 ~]#
[root@mgmt12 ~]#
[root@mgmt12 ~]# service pulp_workers restart
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
Traceback (most recent call last):
File "/usr/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 761, in handle_argv
return self.execute(command, argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 693, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 97, in run_from_argv
[command] + argv, prog_name,
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 206, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 380, in restart
self._stop_nodes(p, cmd, retry=2, callback=on_node_shutdown)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 362, in stop_nodes
self.shutdown_nodes(self.getpids(p, cmd, callback=callback),
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 336, in getpids
pid = Pidfile(pidfile).read_pid()
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 179, in read_pid
'pidfile {0.path} contents invalid.'.format(self))
File "/usr/lib64/python2.6/contextlib.py", line 34, inexit
_
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 732, in ignore_errno
yield
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 169, in read_pid
with open(self.path, 'r') as fh:
IOError: [Errno 13] Permission denied: u'/var/run/pulp/reserved_resource_worker-0.pid'
[root@mgmt12 ~]# service pulp_workers stop
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
Traceback (most recent call last):
File "/usr/bin/celery", line 9, in <module>
load_entry_point('celery==3.1.11', 'console_scripts', 'celery')()
File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
main()
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 306, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 761, in handle_argv
return self.execute(command, argv)
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 693, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 97, in run_from_argv
[command] + argv, prog_name,
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 206, in execute_from_commandline
self.commands[argv[0]](argv[1:], cmd)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 387, in stopwait
return self._stop_nodes(p, cmd, retry=2)
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 362, in _stop_nodes
self.shutdown_nodes(self.getpids(p, cmd, callback=callback),
File "/usr/lib/python2.6/site-packages/celery/bin/multi.py", line 336, in getpids
pid = Pidfile(pidfile).read_pid()
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 179, in read_pid
'pidfile {0.path} contents invalid.'.format(self))
File "/usr/lib64/python2.6/contextlib.py", line 34, in exit
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 732, in ignore_errno
yield
File "/usr/lib/python2.6/site-packages/celery/platforms.py", line 169, in read_pid
with open(self.path, 'r') as fh:
IOError: [Errno 13] Permission denied: u'/var/run/pulp/reserved_resource_worker-0.pid'
[root@mgmt12 ~]#
[root@mgmt12 ~]#
[root@mgmt12 ~]# ls laZ /var/run/pulp
drwxr-sr-x. apache root unconfined_u:object_r:var_run_t:s0 .
drwxr-xr-x. root root system_u:object_r:var_run_t:s0 ..
-rw-r--r-
. apache root unconfined_u:object_r:initrc_var_run_t:s0 celerybeat.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-0.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-1.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-2.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-3.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-4.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-5.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-6.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 reserved_resource_worker-7.pid
rw-r--r-. apache root unconfined_u:object_r:initrc_var_run_t:s0 resource_manager.pid

#4 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Priority changed from Normal to High
  • Severity set to Medium
  • Triaged changed from No to Yes
  • Platform Release set to 2.6.1

#5 Updated by dkliban@redhat.com over 6 years ago

Pulp 2.4 had a fairly lax SELinux policy. The new SELinux policy introduced in 2.5 significantly restricts what celery process running as pulp_workers, pulp_celerybeat, and pulp_resource_manager can do. As a result, if a process was started before an upgrade, it is running in a different context than the process started after the new SELinux policy is applied. As a result the new process is not able to read the PID file of the old process in order to stop it. This is an issue when upgrading from anything before 2.5.

In order to avoid this problem, pulp_workers, pulp_celerybeat, and pulp_resource_manager need to be stopped before the upgrade.

I will add release notes about this.

#6 Updated by dkliban@redhat.com over 6 years ago

  • Platform Release changed from 2.6.1 to 2.6.0

#7 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from ASSIGNED to POST
  • Platform Release changed from 2.6.0 to 2.6.1

#8 Updated by dkliban@redhat.com over 6 years ago

Decided that we should fix this. On upgrade, we will stop pulp_workers, pulp_celerybeat, pulp_resource_manager. The same will happen when uninstalling completely.

https://github.com/pulp/pulp/pull/1721/

#9 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED

#10 Updated by bmbouter over 6 years ago

  • Severity changed from Medium to 2. Medium

#11 Updated by bcourt over 6 years ago

  • Status changed from MODIFIED to 5

#13 Updated by pthomas@redhat.com over 6 years ago

  • Status changed from 5 to 6

verified

Upgrade from 2.4 ->2.6.1 without stopping services worked fine.

Restarted the service without any issues

root@qe-blade-10 ~]#  service pulp_workers restart
celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
> reserved_resource_worker-0@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-1@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-2@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-3@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-4@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-5@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-6@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
> reserved_resource_worker-7@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
celery multi v3.1.11 (Cipater)
> Starting nodes...
    > reserved_resource_worker-0@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-1@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-2@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-3@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-4@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-5@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-6@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
    > reserved_resource_worker-7@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK

[root@qe-blade-10 ~]#  service pulp_celerybeat restart
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... NOT RUNNING
Starting pulp_celerybeat...
[root@qe-blade-10 ~]#  service pulp_resource_manager restart
celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
celery multi v3.1.11 (Cipater)
> resource_manager@qe-blade-10.idmqe.lab.eng.bos.redhat.com: DOWN
celery multi v3.1.11 (Cipater)
> Starting nodes...
    > resource_manager@qe-blade-10.idmqe.lab.eng.bos.redhat.com: OK
[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# service httpd restart
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]

<\pre>

#14 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#17 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF