RPM Support

I started attempting to reproduce this by syncing the stage CDN equivalent:

pulp-admin rpm repo create --repo-id el7-ks --feed http://cdn.stage.redhat.com/content/dist/rhel/server/7/7Server/x86_64/kickstart/ --download-policy on_demand 

This failed initially with the following traceback:

Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776) 'NoneType' object has no attribute 'lower'
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776) Traceback (most recent call last):
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 235, in run
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)     dist_sync.run()
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 125, in run
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)     self._run(tmp_dir)
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 165, in _run
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)     self.update_unit_files(unit, files)
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 242, in update_unit_
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)     CHECKSUM_TYPE: verification.sanitize_checksum_type(_file[CHECKSUM_TYPE])})
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/verification.py", line 67, in sanitize_checksum_type
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776)     lowercase_checksum_type = checksum_type.lower()
Jan 15 21:53:11 dev pulp[14792]: pulp_rpm.plugins.importers.yum.sync:ERROR: (14792-07776) AttributeError: 'NoneType' object has no attribute 'lower'

This occurs because some files come from the http://cdn.stage.redhat.com/content/dist/rhel/server/7/7Server/x86_64/kickstart/PULP_DISTRIBUTION.xml which does not have a checksum or checksum type. Nor does it include an XML schema (I will file a separate issue about this). The schema is located at https://github.com/pulp/pulp_rpm/blob/1ebbd75e8b076fd5d00c95b91c1c1ad26006ed52/plugins/usr/share/pulp-rpm/pulp_distribution.xsd. To handle this problem I added the following:

diff --git a/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py b/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py
index aafc292..b2afe7f 100644
--- a/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py
+++ b/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py
@@ -236,10 +237,12 @@ class DistSync(object):
         if not isinstance(unit.files, list):
             _list = list(unit.files)
         for _file in files:
+            if _file[CHECKSUM_TYPE] is not None:
+                _file[CHECKSUM_TYPE] = verification.sanitize_checksum_type(_file[CHECKSUM_TYPE])
             _list.append({
                 RELATIVE_PATH: _file[RELATIVE_PATH],
                 CHECKSUM: _file[CHECKSUM],
-                CHECKSUM_TYPE: verification.sanitize_checksum_type(_file[CHECKSUM_TYPE])})
+                CHECKSUM_TYPE: _file[CHECKSUM_TYPE]})
         unit.files = _list

     def download_files(self, tmp_dir, files):

This allowed me to complete a sync, but I was unable to kickstart from the repository. The following is the httpd request log while attempting to kickstart from http://dev.example.com/pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/

192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "GET /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/.treeinfo HTTP/1.1" 404 89 "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "GET /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/content HTTP/1.1" 404 87 "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/Fedora HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/current/images/MANIFEST HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/daily/MANIFEST HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/current/images/MANIFEST HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/install/netboot/version.info HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/SL HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/CentOS HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/VERSION HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/.disk/info HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/Server HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/Client HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/RedHat HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/images/pxeboot/vmlinuz HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/images/boot.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/boot/boot.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/current/images/netboot/mini.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/install/images/boot.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:45 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/ HTTP/1.1" 500 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "GET /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/.treeinfo HTTP/1.1" 404 89 "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "GET /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/content HTTP/1.1" 404 87 "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/Fedora HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/current/images/MANIFEST HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/daily/MANIFEST HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/current/images/MANIFEST HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/install/netboot/version.info HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/SL HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/CentOS HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/VERSION HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/.disk/info HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/Server HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/Client HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/RedHat HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/images/pxeboot/vmlinuz HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/images/boot.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/boot/boot.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/current/images/netboot/mini.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/install/images/boot.iso HTTP/1.1" 404 - "-" "python-requests/2.9.1"
192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/ HTTP/1.1" 500 - "-" "python-requests/2.9.1"

That 500 is interesting (I will file a separate issue), but the surprising thing is this entry:

192.168.121.1 - - [15/Jan/2016:21:28:52 +0000] "HEAD /pulp/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/images/pxeboot/vmlinuz HTTP/1.1" 404

This is in the published repo (note that these were all broken links):

[root@dev pxeboot]# pwd
/var/www/pub/yum/http/repos/content/dist/rhel/server/7/7Server/x86_64/kickstart/images/pxeboot

[root@dev pxeboot]# ls -lah
total 20K
drwxr-sr-x. 2 apache apache 4.0K Jan 15 22:10 .
drwxr-sr-x. 3 apache apache 4.0K Jan 15 22:10 ..
lrwxrwxrwx. 1 apache apache  108 Jan 15 22:10 initrd.img -> /var/lib/pulp/content/units/distribution/3e82/3e82b040-8b09-434f-ae1f-a076a825cbc6/images/pxeboot/initrd.img
lrwxrwxrwx. 1 apache apache  109 Jan 15 22:10 upgrade.img -> /var/lib/pulp/content/units/distribution/3e82/3e82b040-8b09-434f-ae1f-a076a825cbc6/images/pxeboot/upgrade.img
lrwxrwxrwx. 1 apache apache  105 Jan 15 22:10 vmlinuz -> /var/lib/pulp/content/units/distribution/3e82/3e82b040-8b09-434f-ae1f-a076a825cbc6/images/pxeboot/vmlinuz

[root@dev pxeboot]# ls -lah /var/lib/pulp/content/units/distribution/3e82/3e82b040-8b09-434f-ae1f-a076a825cbc6/images/pxeboot/
ls: cannot access /var/lib/pulp/content/units/distribution/3e82/3e82b040-8b09-434f-ae1f-a076a825cbc6/images/pxeboot/: No such file or directory

But not in the catalog:

> db.lazy_content_catalog.find({"path": "/var/lib/pulp/content/units/rpm/48b4/48b40206-5723-420e-9a9b-2ec69e6ad5eb/zlib-devel-1.2.7-13.el7.x86_64.rpm"}).pretty()
{
        "_id" : ObjectId("56996a44e7798939c82be7b3"),
        "_ns" : "lazy_content_catalog",
        "path" : "/var/lib/pulp/content/units/rpm/48b4/48b40206-5723-420e-9a9b-2ec69e6ad5eb/zlib-devel-1.2.7-13.el7.x86_64.rpm",
        "importer_id" : "569969b5e77989373afc9dd0",
        "unit_id" : "48b40206-5723-420e-9a9b-2ec69e6ad5eb",
        "unit_type_id" : "rpm",
        "url" : "http://cdn.stage.redhat.com/content/dist/rhel/server/7/7Server/x86_64/kickstart/Packages/zlib-devel-1.2.7-13.el7.x86_64.rpm",
        "revision" : 1,
        "data" : {

        }
}
> db.lazy_content_catalog.find({"path": "/var/lib/pulp/content/units/distribution/3e82/3e82b040-8b09-434f-ae1f-a076a825cbc6/images/pxeboot/vmlinuz"}).pretty()

https://github.com/pulp/pulp_rpm/pull/773
https://github.com/pulp/pulp/pull/2333