Pulp

Is selinux enabled on this system? You can tell with the output of:

getenforce

If it is, we can confirm that SELinux is causing the denial by tailing audit.log. Please paste the relevant denial output from audit.log here.

[stanb@bumblebee ~]$ getenforce
Disabled

I did some experimenting outside pulp, and managed to reproduce the problem outside python.

Attempt to create the export directory as the user:

[stanb@bumblebee ~]$ mkdir /var/run/media/stanb/Linux_Mirrors/test_dir_1
[stanb@bumblebee ~]$ rmdir /var/run/media/stanb/Linux_Mirrors/test_dir_1

Attempt to create the export directory as root:

[root@bumblebee ~]# mkdir /var/run/media/stanb/Linux_Mirrors/test_dir_1
[root@bumblebee ~]# ls /var/run/media/stanb/Linux_Mirrors
test_dir_1
[root@bumblebee ~]# rmdir /var/run/media/stanb/Linux_Mirrors/test_dir_1

Attempt to create the export directory as apache:

[root@bumblebee ~]# sudo -u apache mkdir /var/run/media/stanb/Linux_Mirrors/test_dir_1
mkdir: cannot create directory ‘/var/run/media/stanb/Linux_Mirrors/test_dir_1’: Permission denied
[root@bumblebee ~]# echo $?
1

Why does it fail? Check the permissions up the path:

[root@bumblebee ~]# ls -al /var/run/media/stanb/Linux_Mirrors/
drwxrwxrwx   7 root  root      4096 Jun 15 11:27 .
drwxr-x---+  3 root  root        60 Jun 10 15:27 ..
[root@bumblebee ~]# ls -al /var/run/media/stanb
drwxr-x---+ 3 root root   60 Jun 10 15:27 .
drwxr-xr-x  3 root root   60 Jun 10 15:27 ..

Can we make it work by hook or by crook?

[root@bumblebee ~]# chmod o+rwx /var/run/media/stanb/
[root@bumblebee ~]# ls -al /var/run/media/stanb
drwxr-xrwx+ 3 root root   60 Jun 10 15:27 .
drwxr-xr-x  3 root root   60 Jun 10 15:27 ..
[root@bumblebee ~]# sudo -u apache mkdir /var/run/media/stanb/Linux_Mirrors/test_dir_1
[root@bumblebee ~]# chmod o-rwx /var/run/media/stanb

Yes, it can be worked around.

Testing it with pulp repo export:

pulp-admin rpm repo export run \
    --repo-id=foreman \
    --export-dir=/var/run/media/stanb/Linux_Mirrors/pulp_repos/foreman
+----------------------------------------------------------------------+
                    Publishing Repository [foreman]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Export Dir:  /var/run/media/stanb/Linux_Mirrors/pulp_repos/foreman

This command may be exited via ctrl+c without affecting the request.

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
98 of 98 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Copying files
[\]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[stanb@bumblebee ~]$ 

In summary:

1. user can access the external disk export path
2. root can access the external disk export path
3. apache cannot access the external disk export path, unless o+rwx permissions are granted to /var/run/media/${USER} .

If this use case is to work as expected in RHEL7, the process writing the data to the export disk must belong to the user or root, not to apache. This is a significant architectural change.

When I export the repo, I observe the following:

1. Everything is owned by apache.
2. Files have permissions rw-r--r-
3. repodata directory has permissions drwxr-x---

It would seem to be more useful if the exported files were owned by the user who executed the export command. As it is, the user cannot access the repodata directory!

I have started exporting my 613 repos to an external drive.

I can't believe how long it takes to publish a repo.

When pulp is exporting, the following steps are performed:

1. Initializing repo metadata
2. Publishing Distribution files
3. Publishing RPMs <---- publishes to an internal location, seems to take a long time. Is it checking the checksums, or just placing a symlink?
4. Publishing Delta RPMs
5. Publishing Errata
6. Publishing Comps file
7. Publishing Metadata.
8. Closing repo metadata
9. Generating sqlite files
10. Copying files <---- copies to export path
11. Writing Listings File
12. Task Succeeded

If the files are being copied bit-for-bit twice, could the second copy be eliminated to gain efficiency?

I might have to break out systemtap [0] and produce a flame graph [1] to figure out what it is really spending its time on.

[0] [[https://en.wikipedia.org/wiki/SystemTap]]
[1] [[http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html ]]

I have now tried this a few different ways, and it looks like pulp-admin export is the worst of them.

In summary, there are several practical problems to exporting a collection of repos to another machine:

1. Writing the files to an HDD media (apache trying to write to /var/run/media/${USER}/${FS_LABEL}, owned by ${USER}
2. The large data size presented when viewed as a collection of repos (without sharing identical files)
3. The problem of re-creating the repos on the destination machine.

Due to the slowness of using pulp-admin export, I attempted to rsync /var/lib/pulp/published/yum/http/repos directly to my HDD. I found I had to use --copy-links, which de-references the links while copying, otherwise it copied links to the files on the main system instead of copying the RPMs over to the external HDD. This ended up badly, as RHEL 5.10 alone consumed 400Gb. The 2Tb sync HDD will not contain even the RHEL5 series by itself, let alone everything else I have.

After some poking around, I considered rsyncing the following two directories:

/var/lib/pulp
/var/lib/mongodb

Consider this as transfer by backup/restore.

Some care is required to preserve permissions, ownership and links. Also, the target machine must have the same numeric UIDs/GIDs for mongodb, qpid and apache (and maybe others?).

However, it is possible to fit these directories completely on to a 2Tb external HDD to perform the sync.

This method has the advantage that it also transfers the repo information, although the sync feed is wrong. However, I don't plan to sync the disconnected pulp server. I think I will maintain it by rsync.

The downside of this technique is that I must stop pulp and mongod before performing the final sync, and while restoring on the other end.

I have also found that attempting to load a page on the secondary pulp server causes the qpidd server on the primary machine to go crazy (the two machines are temporarily connected to the same network for setup purposes). So something is referencing the primary machine. If I can find out what that is and change it to reference the secondary machine instead, then I hope it will all work.

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.