Project

Profile

Help

Issue #1202

pulp_celerybeat,pulp_resource_manager fail to start on boot

Added by alvinstarr about 5 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.6.2 Beta
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix, Pulp 2
Sprint:
Quarter:

Description

I am running a reasonably stock Katello 2.3 on C7 and when I reboot the system celerybeat and resource_maager do not start.
They can be restarted via systemctl and run just fine.

Aug 17 09:23:49 foreman systemd: pulp_celerybeat.service: main process exited, code=exited, status=200/CHDIR
Aug 17 09:23:49 foreman systemd: Unit pulp_celerybeat.service entered failed state.
Aug 17 09:23:49 foreman systemd: pulp_resource_manager.service: main process exited, code=exited, status=200/CHDIR
Aug 17 09:23:49 foreman systemd: Unit pulp_resource_manager.service entered failed state.

It occurs to me that the issue may be that I have a /var/lib/pulp as a mounted NFS partition.

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos_foreman-root 50577276 10911124 39666152 22% /
devtmpfs 3987812 0 3987812 0% /dev
tmpfs 3865924 0 3865924 0% /dev/shm
tmpfs 3865924 8680 3857244 1% /run
tmpfs 3865924 0 3865924 0% /sys/fs/cgroup
/dev/mapper/centos_foreman-home 24695008 32928 24662080 1% /home
/dev/xvda1 508588 166600 341988 33% /boot
xen:/var/storage/foreman/pulp 780709888 27501568 753208320 4% /var/lib/pulp

Not sure if that is the issue or not.
The version I am running is:

Installed Packages
Name : pulp-server
Arch : noarch
Version : 2.6.2
Release : 1.el7
Size : 2.8 M
Repo : installed
From repo : katello-pulp
Summary : The pulp platform server
URL : https://fedorahosted.org/pulp/
License : GPLv2
Description : Pulp provides replication, access, and accounting for software
: repositories.

History

#1 Updated by bcourt about 5 years ago

Do you have any modifications that you would suggest for the pulp systemd service definitions to ensure that pulp starts after NFS?

#2 Updated by alvinstarr about 5 years ago

Adding "After=remote-fs.target" to the unit secion seems to help.
I am by no means a systemd expert so there may be other/better solutions.

From my reading remote-fs.target is activated after the remote file systems are all mounted.
I made the change and celerybeat/resource_manager now start on boot where they did not before.

[Unit]
Description=Pulp's Celerybeat
After=network.target
After=remote-fs.target

#3 Updated by bmbouter about 5 years ago

A newer version of Pulp (iirc 2.7.0 beta+) will keep these pids on a local folder and not in the /var/lib/pulp. Even still, I think it would be good to have all Pulp services wait until the network is started. I think we should propogate the units change to all of our units files.

@rbarlow, what do you think?

@alvinstarr, do you have any interest in making a PR for that change?

#4 Updated by alvinstarr about 5 years ago

I am new to this development process so at the risk of sounding stupid I
am happy to make a PR but what is a PR?

On 08/21/2015 03:53 PM, Pulp wrote:

#5 Updated by rbarlow about 5 years ago

On 08/21/2015 03:53 PM, Pulp wrote:

A newer version of Pulp (iirc 2.7.0 beta+) will keep these pids on a
local folder and not in the /var/lib/pulp. Even still, I think it would
be good to have all Pulp services wait until the network is started. I
think we should propogate the units change to all of our units files.

@rbarlow, what do you think?

For the network services I think this is a reasonable suggestion, though
I don't believe it will solve any issues that people might be
experiencing since our Celery code will keep attempting to connect to
its network dependencies until it is able. Thus, even without doing this
everything will work. I would say it's the "right thing", though I also
don't think it'll change anyone's experiences with Pulp in a noticeable way.

For the remote-fs, there's a way to add a symlink under /etc/systemd to
indicate that you want a service to start after another one. Since the
remote-fs is system-dependent (i.e., not everybody is hosting
/var/lib/pulp in NFS), I think this would be the way to go. I don't
recall the exact way to do this, but I believe you can learn about it here:

http://www.freedesktop.org/software/systemd/man/systemd.unit.html

--
Randy Barlow

#6 Updated by rbarlow about 5 years ago

On 08/23/2015 09:35 AM, Pulp wrote:

I am new to this development process so at the risk of sounding stupid I
am happy to make a PR but what is a PR?

A PR is a Pull Request on GitHub. Thanks for helping out!

--
Randy Barlow

#7 Updated by bmbouter about 5 years ago

+1 to doing it in a way that generalizes for more than just NFS.

We should make a change (or doc it) because the services will wait until the network becomes available, but if they can't start because the shared filesystem is down then they won't wait.

#8 Updated by alvinstarr about 5 years ago

remote_fs.target is a bit of a blunt instrument.
It should not be true until all the remote filesystems(NFS,GFS,Gluster...) have been mounted.
I believe that remote_fs.target gets set to true when there are no(-more) remote filesystems to mount.
I have not tested this though.

On one hand pulp should not need to wait for /home/xxx to be mounted.
On the other hand pulp is the kind of application where it will burn up lots of space and be the kind of thing someone will want to mount on some form of remote storage.

There is a RequiresMountsFor feature in systemd and it looks like it may work and that would need to have a list of directories that would reasonably used.
So /var/lib/pulp/* should cover it.
That way pulp could start faster than having to wait for all the remote filesystems to mount.

RequiresMountsFor

#9 Updated by mhrivnak about 5 years ago

  • Triaged changed from No to Yes
  • Tags Easy Fix added

The RequiresMountsFor option sounds like the way to go. Thanks for finding that!

#10 Updated by rbarlow about 5 years ago

  • Triaged changed from Yes to No
  • Tags deleted (Easy Fix)

#11 Updated by rbarlow about 5 years ago

  • Triaged changed from No to Yes
  • Tags Easy Fix added

#12 Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

#13 Updated by bmbouter over 1 year ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#14 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF