Project

Profile

Help

Issue #719

Pulp workers won't start

Added by dkliban@redhat.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.7.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

When I build RPMs from master and install, the /var/cache/pulp directory is not created.
As a result when I start pulp_workers, they fail to start because their working directories can't be created.

You can see that a merge from 2.6-dev [0] changed the pulp.spec file incorrectly. This was most likely a result of a merge using ours strategy.

[0] https://github.com/pulp/pulp/commit/f4d2de05d58f71e9baccd18e0ce8020c3e80d6fb#diff-d478c192eb97a80291504e709a4172b3L359

note from triage: this only affects 2.7, not 2.6.x

Associated revisions

Revision f89780d2 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Added persistence of /var/run/pulp

This patch provides a fix for both systemd and upstart. A configuration file for systemd-tmpfiles helps ensure that /var/run/pulp is recreated at boot time. The upstart script already created the directory if it was missing. The same script now restores the context on the newly created directory. The relabel.sh script now contains a statement to restore context of /var/run/pulp also.

Fixes #719

Revision f89780d2 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Added persistence of /var/run/pulp

This patch provides a fix for both systemd and upstart. A configuration file for systemd-tmpfiles helps ensure that /var/run/pulp is recreated at boot time. The upstart script already created the directory if it was missing. The same script now restores the context on the newly created directory. The relabel.sh script now contains a statement to restore context of /var/run/pulp also.

Fixes #719

History

#1 Updated by ipanova@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Priority changed from Normal to High
  • Triaged changed from No to Yes
  • Severity set to Medium

#2 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from ASSIGNED to POST

#3 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED

#4 Updated by bmbouter over 6 years ago

  • Severity changed from Medium to 2. Medium

#5 Updated by bmbouter over 6 years ago

  • Description updated (diff)
  • Status changed from MODIFIED to NEW
  • Assignee deleted (dkliban@redhat.com)
  • Triaged changed from Yes to No
  • Platform Release set to 2.7.0

A user upgraded from pulp-server-2.6.0-1 to the 2.7 alpha and experienced this issue again. They were on centos7, and after upgrading the /var/run/pulp directory was not created. They attempted to start their workers with sudo service pulp_workers start, but they received this error message:

Failed at step CHDIR spawning /usr/libexec/pulp-manage-workers: No such file or directory

To resolve it they manually ran:

sudo mkdir /var/run/pulp
sudo chown apache:apache /var/run/pulp

After that they could start their workers successfully. I suspect this line in the spec file is not behaving like we think it is. That is especially concerning given that many lines in the spec file have that form, perhaps more than this is broken?

#7 Updated by bmbouter over 6 years ago

Using a permalink is a good idea. I updated this one, and I'll do that in the future too.

#8 Updated by paji@redhat.com over 6 years ago

Also noticed that the /var/run/pulp directory got deleted or removed when I did a "Force off" of the vm and rebooted.

[~]$ uname -a
Linux <fqdn> 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[~]$ cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core)
[~]$ rpm -qa|grep pulp
python-pulp-docker-common-1.0.1-0.1.alpha.el7.noarch
pulp-nodes-parent-2.7.0-0.1.alpha.el7.noarch
python-kombu-3.0.24-6.pulp.el7.noarch
python-pulp-rpm-common-2.7.0-0.1.alpha.el7.noarch
pulp-nodes-common-2.7.0-0.1.alpha.el7.noarch
pulp-selinux-2.7.0-0.1.alpha.el7.noarch
pulp-docker-admin-extensions-1.0.1-0.1.alpha.el7.noarch
python-pulp-bindings-2.7.0-0.1.alpha.el7.noarch
python-pulp-repoauth-2.7.0-0.1.alpha.el7.noarch
pulp-docker-plugins-1.0.1-0.1.alpha.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-admin-client-2.7.0-0.1.alpha.el7.noarch
pulp-katello-0.3-4.el7.noarch
python-pulp-common-2.7.0-0.1.alpha.el7.noarch
python-pulp-puppet-common-2.7.0-0.1.alpha.el7.noarch
pulp-rpm-plugins-2.7.0-0.1.alpha.el7.noarch
python-pulp-ostree-common-1.0.0-0.1.alpha.el7.noarch
pulp-ostree-plugins-1.0.0-0.1.alpha.el7.noarch
pulp-server-2.7.0-0.1.alpha.el7.noarch
pulp-ostree-admin-extensions-1.0.0-0.1.alpha.el7.noarch
pulp-puppet-plugins-2.7.0-0.1.alpha.el7.noarch
python-pulp-client-lib-2.7.0-0.1.alpha.el7.noarch

#9 Updated by bmbouter over 6 years ago

When this was implemented we expected the /var/run directory be created by the spec file and expect it to exist for all time after that. I think it doesn't survive a reboot by design in Linux, and /var/run/* is cleared on restart! linfo.org says that the /var/run directory is only valid until system restart.

I recommend taking the creation of /var/run/pulp/ out of the spec file and letting the systemd units and upstart scripts handle the creation of /var/run/pulp.

#10 Updated by paji@redhat.com over 6 years ago

I needed to add the following to get /var/run/pulp persist across reboots

[~]$ cat /usr/lib/tmpfiles.d/pulp.conf 
d /run/pulp 0755 apache apache

#11 Updated by cduryee over 6 years ago

  • Description updated (diff)
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#12 Updated by dkliban@redhat.com over 6 years ago

  • Assignee set to dkliban@redhat.com

#13 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from NEW to MODIFIED
  • % Done changed from 0 to 100

#14 Updated by dkliban@redhat.com over 6 years ago

When verifying this issue, please make sure to have SELinux Enforcing. Since the implementation is different for systemd and upstart systems, this issue should be verified on both types of machines. Here are the instructions for both:

On EL6 start mongod, qpidd, and pulp_workers and verify that pid files are created:

sudo service mongod start
sudo service qpidd start
sudo service pulp_workers start
ls -laZ /var/run/pulp

The output should look like this:

drwxr-sr-x. apache root unconfined_u:object_r:pulp_var_run_t:s0 .
drwxr-xr-x. root   root system_u:object_r:var_run_t:s0   ..
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-0.pid
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-1.pid

Then you should stop the workers:

sudo service pulp_workers stop

Then restart the server:

sudo init 6

After the system reboots:

sudo service mongod start
sudo service qpidd start
sudo service pulp_workers start
ls -laZ /var/run/pulp

The output should look same as above.


On EL7 start mongod, qpidd, and pulp_workers and verify that pid files are created:

sudo systemctl start mongod
sudo systemctl start qpidd
sudo systemctl start pulp_workers
ls -laZ /var/run/pulp

The output should look like this:

drwxr-sr-x. apache root unconfined_u:object_r:pulp_var_run_t:s0 .
drwxr-xr-x. root   root system_u:object_r:var_run_t:s0   ..
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-0.pid
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-1.pid

Then you should stop the workers:

sudo systemctl stop pulp_workers

Then restart the server:

sudo init 6

After the system reboots:

sudo systemctl start mongod
sudo systemctl start qpidd
sudo systemctl start pulp_workers
ls -laZ /var/run/pulp

The output should look same as above.

#15 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from MODIFIED to 5

#16 Updated by pthomas@redhat.com about 6 years ago

  • Status changed from 5 to 6

verified
pulp-workers are started without any issues in el6 & el7.

#17 Updated by amacdona@redhat.com over 5 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#19 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF