pulp_celerybeat,pulp_resource_manager fail to start on boot
I am running a reasonably stock Katello 2.3 on C7 and when I reboot the system celerybeat and resource_maager do not start.
They can be restarted via systemctl and run just fine.
Aug 17 09:23:49 foreman systemd: pulp_celerybeat.service: main process exited, code=exited, status=200/CHDIR
Aug 17 09:23:49 foreman systemd: Unit pulp_celerybeat.service entered failed state.
Aug 17 09:23:49 foreman systemd: pulp_resource_manager.service: main process exited, code=exited, status=200/CHDIR
Aug 17 09:23:49 foreman systemd: Unit pulp_resource_manager.service entered failed state.
It occurs to me that the issue may be that I have a /var/lib/pulp as a mounted NFS partition.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos_foreman-root 50577276 10911124 39666152 22% /
devtmpfs 3987812 0 3987812 0% /dev
tmpfs 3865924 0 3865924 0% /dev/shm
tmpfs 3865924 8680 3857244 1% /run
tmpfs 3865924 0 3865924 0% /sys/fs/cgroup
/dev/mapper/centos_foreman-home 24695008 32928 24662080 1% /home
/dev/xvda1 508588 166600 341988 33% /boot
xen:/var/storage/foreman/pulp 780709888 27501568 753208320 4% /var/lib/pulp
Not sure if that is the issue or not.
The version I am running is:
Name : pulp-server
Arch : noarch
Version : 2.6.2
Release : 1.el7
Size : 2.8 M
Repo : installed
From repo : katello-pulp
Summary : The pulp platform server
URL : https://fedorahosted.org/pulp/
License : GPLv2
Description : Pulp provides replication, access, and accounting for software
#2 Updated by alvinstarr over 5 years ago
Adding "After=remote-fs.target" to the unit secion seems to help.
I am by no means a systemd expert so there may be other/better solutions.
From my reading remote-fs.target is activated after the remote file systems are all mounted.
I made the change and celerybeat/resource_manager now start on boot where they did not before.
#3 Updated by bmbouter over 5 years ago
A newer version of Pulp (iirc 2.7.0 beta+) will keep these pids on a local folder and not in the /var/lib/pulp. Even still, I think it would be good to have all Pulp services wait until the network is started. I think we should propogate the units change to all of our units files.
@rbarlow, what do you think?
@alvinstarr, do you have any interest in making a PR for that change?
#5 Updated by rbarlow over 5 years ago
On 08/21/2015 03:53 PM, Pulp wrote:
A newer version of Pulp (iirc 2.7.0 beta+) will keep these pids on a
local folder and not in the /var/lib/pulp. Even still, I think it would
be good to have all Pulp services wait until the network is started. I
think we should propogate the units change to all of our units files.
@rbarlow, what do you think?
For the network services I think this is a reasonable suggestion, though
I don't believe it will solve any issues that people might be
experiencing since our Celery code will keep attempting to connect to
its network dependencies until it is able. Thus, even without doing this
everything will work. I would say it's the "right thing", though I also
don't think it'll change anyone's experiences with Pulp in a noticeable way.
For the remote-fs, there's a way to add a symlink under /etc/systemd to
indicate that you want a service to start after another one. Since the
remote-fs is system-dependent (i.e., not everybody is hosting
/var/lib/pulp in NFS), I think this would be the way to go. I don't
recall the exact way to do this, but I believe you can learn about it here:
#8 Updated by alvinstarr over 5 years ago
remote_fs.target is a bit of a blunt instrument.
It should not be true until all the remote filesystems(NFS,GFS,Gluster...) have been mounted.
I believe that remote_fs.target gets set to true when there are no(-more) remote filesystems to mount.
I have not tested this though.
On one hand pulp should not need to wait for /home/xxx to be mounted.
On the other hand pulp is the kind of application where it will burn up lots of space and be the kind of thing someone will want to mount on some form of remote storage.
There is a RequiresMountsFor feature in systemd and it looks like it may work and that would need to have a list of directories that would reasonably used.
So /var/lib/pulp/* should cover it.
That way pulp could start faster than having to wait for all the remote filesystems to mount.
#10 Updated by rbarlow over 5 years ago
- Triaged changed from Yes to No
- Tags deleted (
RequiresMountsFor is documented here:
#13 Updated by bmbouter about 2 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
Please register to edit this issue