Issue #719
closed
Pulp workers won't start
Description
When I build RPMs from master and install, the /var/cache/pulp directory is not created.
As a result when I start pulp_workers, they fail to start because their working directories can't be created.
You can see that a merge from 2.6-dev [0] changed the pulp.spec file incorrectly. This was most likely a result of a merge using ours strategy.
note from triage: this only affects 2.7, not 2.6.x
Updated by ipanova@redhat.com almost 9 years ago
- Status changed from NEW to ASSIGNED
- Priority changed from Normal to High
- Triaged changed from No to Yes
- Severity set to Medium
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from ASSIGNED to POST
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from POST to MODIFIED
Updated by bmbouter over 8 years ago
- Description updated (diff)
- Status changed from MODIFIED to NEW
- Assignee deleted (
dkliban@redhat.com) - Triaged changed from Yes to No
- Platform Release set to 2.7.0
A user upgraded from pulp-server-2.6.0-1 to the 2.7 alpha and experienced this issue again. They were on centos7, and after upgrading the /var/run/pulp directory was not created. They attempted to start their workers with sudo service pulp_workers start, but they received this error message:
Failed at step CHDIR spawning /usr/libexec/pulp-manage-workers: No such file or directory
To resolve it they manually ran:
sudo mkdir /var/run/pulp
sudo chown apache:apache /var/run/pulp
After that they could start their workers successfully. I suspect this line in the spec file is not behaving like we think it is. That is especially concerning given that many lines in the spec file have that form, perhaps more than this is broken?
Updated by mhrivnak over 8 years ago
Permanent link to the line mentioned above:
https://github.com/pulp/pulp/blob/6f4a29f7a7eb664c5e75e691c797280a96841562/pulp.spec#L394
Updated by bmbouter over 8 years ago
Using a permalink is a good idea. I updated this one, and I'll do that in the future too.
Updated by paji@redhat.com over 8 years ago
Also noticed that the /var/run/pulp directory got deleted or removed when I did a "Force off" of the vm and rebooted.
[~]$ uname -a
Linux <fqdn> 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[~]$ cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[~]$ rpm -qa|grep pulp
python-pulp-docker-common-1.0.1-0.1.alpha.el7.noarch
pulp-nodes-parent-2.7.0-0.1.alpha.el7.noarch
python-kombu-3.0.24-6.pulp.el7.noarch
python-pulp-rpm-common-2.7.0-0.1.alpha.el7.noarch
pulp-nodes-common-2.7.0-0.1.alpha.el7.noarch
pulp-selinux-2.7.0-0.1.alpha.el7.noarch
pulp-docker-admin-extensions-1.0.1-0.1.alpha.el7.noarch
python-pulp-bindings-2.7.0-0.1.alpha.el7.noarch
python-pulp-repoauth-2.7.0-0.1.alpha.el7.noarch
pulp-docker-plugins-1.0.1-0.1.alpha.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-admin-client-2.7.0-0.1.alpha.el7.noarch
pulp-katello-0.3-4.el7.noarch
python-pulp-common-2.7.0-0.1.alpha.el7.noarch
python-pulp-puppet-common-2.7.0-0.1.alpha.el7.noarch
pulp-rpm-plugins-2.7.0-0.1.alpha.el7.noarch
python-pulp-ostree-common-1.0.0-0.1.alpha.el7.noarch
pulp-ostree-plugins-1.0.0-0.1.alpha.el7.noarch
pulp-server-2.7.0-0.1.alpha.el7.noarch
pulp-ostree-admin-extensions-1.0.0-0.1.alpha.el7.noarch
pulp-puppet-plugins-2.7.0-0.1.alpha.el7.noarch
python-pulp-client-lib-2.7.0-0.1.alpha.el7.noarch
Updated by bmbouter over 8 years ago
When this was implemented we expected the /var/run directory be created by the spec file and expect it to exist for all time after that. I think it doesn't survive a reboot by design in Linux, and /var/run/* is cleared on restart! linfo.org says that the /var/run directory is only valid until system restart.
I recommend taking the creation of /var/run/pulp/ out of the spec file and letting the systemd units and upstart scripts handle the creation of /var/run/pulp.
Updated by paji@redhat.com over 8 years ago
I needed to add the following to get /var/run/pulp persist across reboots
[~]$ cat /usr/lib/tmpfiles.d/pulp.conf
d /run/pulp 0755 apache apache
Updated by cduryee over 8 years ago
- Description updated (diff)
- Severity changed from 2. Medium to 3. High
- Triaged changed from No to Yes
Updated by dkliban@redhat.com over 8 years ago
- Assignee set to dkliban@redhat.com
Added by dkliban@redhat.com over 8 years ago
Added by dkliban@redhat.com over 8 years ago
Revision f89780d2 | View on GitHub
Added persistence of /var/run/pulp
This patch provides a fix for both systemd and upstart. A configuration file for systemd-tmpfiles helps ensure that /var/run/pulp is recreated at boot time. The upstart script already created the directory if it was missing. The same script now restores the context on the newly created directory. The relabel.sh script now contains a statement to restore context of /var/run/pulp also.
Fixes #719
Updated by dkliban@redhat.com over 8 years ago
- Status changed from NEW to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|f89780d24e72904bc09b2eb466e2be153b7993d1.
Updated by dkliban@redhat.com over 8 years ago
When verifying this issue, please make sure to have SELinux Enforcing. Since the implementation is different for systemd and upstart systems, this issue should be verified on both types of machines. Here are the instructions for both:
On EL6 start mongod, qpidd, and pulp_workers and verify that pid files are created:
sudo service mongod start
sudo service qpidd start
sudo service pulp_workers start
ls -laZ /var/run/pulp
The output should look like this:
drwxr-sr-x. apache root unconfined_u:object_r:pulp_var_run_t:s0 .
drwxr-xr-x. root root system_u:object_r:var_run_t:s0 ..
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-0.pid
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-1.pid
Then you should stop the workers:
sudo service pulp_workers stop
Then restart the server:
sudo init 6
After the system reboots:
sudo service mongod start
sudo service qpidd start
sudo service pulp_workers start
ls -laZ /var/run/pulp
The output should look same as above.
On EL7 start mongod, qpidd, and pulp_workers and verify that pid files are created:
sudo systemctl start mongod
sudo systemctl start qpidd
sudo systemctl start pulp_workers
ls -laZ /var/run/pulp
The output should look like this:
drwxr-sr-x. apache root unconfined_u:object_r:pulp_var_run_t:s0 .
drwxr-xr-x. root root system_u:object_r:var_run_t:s0 ..
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-0.pid
-rw-r--r--. apache root unconfined_u:object_r:pulp_var_run_t:s0 reserved_resource_worker-1.pid
Then you should stop the workers:
sudo systemctl stop pulp_workers
Then restart the server:
sudo init 6
After the system reboots:
sudo systemctl start mongod
sudo systemctl start qpidd
sudo systemctl start pulp_workers
ls -laZ /var/run/pulp
The output should look same as above.
Updated by dkliban@redhat.com over 8 years ago
- Status changed from MODIFIED to 5
Updated by pthomas@redhat.com over 8 years ago
- Status changed from 5 to 6
verified
pulp-workers are started without any issues in el6 & el7.
Updated by amacdona@redhat.com about 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
.
Added persistence of /var/run/pulp
This patch provides a fix for both systemd and upstart. A configuration file for systemd-tmpfiles helps ensure that /var/run/pulp is recreated at boot time. The upstart script already created the directory if it was missing. The same script now restores the context on the newly created directory. The relabel.sh script now contains a statement to restore context of /var/run/pulp also.
Fixes #719