Move Pulp to a Software Collection on EL6
This story is the result of https://pulp.plan.io/issues/931. Currently, all Pulp code must be Python 2.6 compatible. This makes transitioning to Python 3 compatible code difficult, as many features introduced in 2.7 are specifically for writing code that runs on 2.7 and 3.x. Furthermore, we are stuck with a very old version of Django (1.4). Therefore, it would be helpful if we could raise our minimum to requirements to Python 2.7 and Django 1.6 (for now). Both these ship with RHEL 7 so in order to support RHEL6, we can use Software Collections.
Software Collections install into /opt/ or /var/opt, so they don't interfere with the versions that ship with the OS. We need to extend the existing Python 2.7 SCL with all our Python dependencies. For each Python dependency we have that isn't provided by the Python 2.7 SCL, we must create an RPM package that uses the SCL. This can be done by acquiring the existing spec files for these dependencies and tweaking them (potentially with spec2scl). Note that this applies not just to our direct dependencies, but also the dependencies of those dependencies, all the way to the packages that have no dependencies beyond those already provided by the Python 2.7 SCL or the platform itself.
Once all the dependencies are packaged for the SCL, Pulp itself needs its spec file modified to use the SCL. It also needs the init scripts and commands installed into /usr/bin to use the SCL paths. All these changes can occur as patches applied by the spec file (or even as install steps in the spec itself).
All these spec files (and their associated source tarballs) need to be tracked. The Foreman has a SCL and they use Git with git-annex. We could probably do the same. My recommendation is that we move our spec files out of the pulp and pulp_<plugin> repos and into a package repository. Within that repository we should have branches for each platform, much like Fedora does (so an "el6", "el7", "f23", and "f24" branch). For the "el6" branch, we check in all the SCL-related files. This has two advantages: we get rid of all our nasty if/else blocks for various platforms and we don't have to add more for the SCL work.
Here is a potentially incomplete dependency tree for pulp-server on EL6:
pulp-server-2.8.2-1.el6.noarch [cmd line] \_ Django14-1.4.21-1.el6.noarch [1: Django14] \_ m2crypto-0.20.2-9.el6.x86_64 [1: m2crypto] \_ mod_wsgi-3.4-2.pulp.el6.x86_64 [1: mod_wsgi >= 3.4-1.pulp] \_ python-blinker-1.1-1.el6.noarch [1: python-blinker] \_ python-celery-3.1.11-1.el6.noarch [2: python-celery < 3.2.0, python-celery >= 3.1.0] | \_ pyparsing-1.5.6-1.el6.noarch [1: pyparsing] | \_ python-amqp-1.4.9-1.el6.noarch [1: python-amqp] | \_ 1:python-billiard-188.8.131.52-2.el6.x86_64 [1: python-billiard >= 184.108.40.206] | \_ python-dateutil-1.4.1-6.el6.noarch [1: python-dateutil] | | \_ tzdata-2016c-1.el6.noarch [1: tzdata] | \_ python-importlib-1.0.2-1.el6.noarch [1: python-importlib] | \_ 1:python-kombu-3.0.33-4.pulp.el6.noarch [1: python-kombu >= 3.0.15] | | \_ python-amqp-1.4.9-1.el6.noarch [2: python-amqp < 2.0, python-amqp >= 1.4.9] | | \_ python-libs-2.6.6-64.el6.i686 [1: python-ordereddict] | | \_ python-libs-2.6.6-64.el6.x86_64 [1: python-ordereddict] | | \_ python-ordereddict-1.1-2.el6.noarch [1: python-ordereddict] | \_ python-setuptools-0.6.10-3.el6.noarch [1: python-setuptools] | \_ pytz-2010h-2.el6.noarch [1: pytz] \_ python-gofer-2.7.5-1.el6.noarch [1: python-gofer >= 2.5] \_ python-httplib2-0.7.7-1.el6.noarch [1: python-httplib2] \_ python-ldap-2.3.10-1.el6.x86_64 [1: python-ldap] \_ python-mongoengine-0.10.5-1.el6.noarch [1: python-mongoengine >= 0.10.0] | \_ python-blinker-1.1-1.el6.noarch [1: python-blinker] | \_ python-pymongo-3.2-1.el6.x86_64 [1: python-pymongo >= 3.2] | \_ python-pymongo-gridfs-3.2-1.el6.x86_64 [1: python-pymongo-gridfs >= 3.2] | | \_ python-pymongo-3.2-1.el6.x86_64 [1: python-pymongo(x86-64) = 3.2-1.el6] \_ python-nectar-1.5.1-1.el6.noarch [1: python-nectar >= 1.5.0] | \_ python-requests-2.6.0-3.el6.noarch [1: python-requests >= 2.4.3] | | \_ python-chardet-2.2.1-1.el6.noarch [1: python-chardet >= 2.2.1-1] | | \_ python-libs-2.6.6-64.el6.i686 [1: python-ordereddict >= 1.1] | | \_ python-libs-2.6.6-64.el6.x86_64 [1: python-ordereddict >= 1.1] | | \_ python-ordereddict-1.1-2.el6.noarch [1: python-ordereddict >= 1.1] | | \_ python-urllib3-1.10.2-1.el6.noarch [1: python-urllib3 >= 1.10.2-1] | | | \_ python-backports-ssl_match_hostname-220.127.116.11-2.el6.noarch [1: python-backports-ssl_match_hostname] | | | | \_ python-backports-1.0-5.el6.x86_64 [1: python-backports] | | | \_ python-libs-2.6.6-64.el6.i686 [1: python-ordereddict] | | | \_ python-libs-2.6.6-64.el6.x86_64 [1: python-ordereddict] | | | \_ python-ordereddict-1.1-2.el6.noarch [1: python-ordereddict] | | | \_ python-six-1.9.0-2.el6.noarch [1: python-six] \_ python-oauth2-1.5.211-8.el6.noarch [1: python-oauth2 >= 1.5.211] | \_ python-httplib2-0.7.7-1.el6.noarch [1: python-httplib2] \_ python-pulp-common-2.8.2-1.el6.noarch [1: python-pulp-common = 2.8.2] | \_ python-iniparse-0.3.1-2.1.el6.noarch [1: python-iniparse] \_ python-pulp-repoauth-2.8.2-1.el6.noarch [1: python-pulp-repoauth = 2.8.2] | \_ mod_wsgi-3.4-2.pulp.el6.x86_64 [1: mod_wsgi >= 3.4-1.pulp] | \_ python-pulp-common-2.8.2-1.el6.noarch [1: python-pulp-common = 2.8.2] | \_ python-rhsm-0.99.12-1.el6.noarch [1: python-rhsm] | | \_ m2crypto-0.20.2-9.el6.x86_64 [1: m2crypto] | | \_ python-iniparse-0.3.1-2.1.el6.noarch [1: python-iniparse] | \_ python-rhsm-1.14.3-1.el6.x86_64 [1: python-rhsm] | | \_ m2crypto-0.20.2-9.el6.x86_64 [1: m2crypto] | | \_ python-dateutil-1.4.1-6.el6.noarch [1: python-dateutil] | | \_ python-iniparse-0.3.1-2.1.el6.noarch [1: python-iniparse] | | \_ rpm-python-4.8.0-47.el6.x86_64 [1: rpm-python] | \_ python-setuptools-0.6.10-3.el6.noarch [1: python-setuptools] \_ python-pymongo-3.2-1.el6.x86_64 [1: python-pymongo >= 3.0.0] \_ python-qpid-0.14-11.el6_3.noarch [1: python-qpid] | \_ python-saslwrapper-0.14-1.el6.x86_64 [1: python-saslwrapper >= 0.10] \_ python-semantic_version-2.2.0-6.el6.noarch [1: python-semantic_version >= 2.2.0] \_ python-setuptools-0.6.10-3.el6.noarch [1: python-setuptools]
In the above list I stripped out all dependencies to non-Python packages. It could be I accidentally got rid of a Python library I simply didn't recognize, so the tree should not be treated as an authoritative reference.
Getting Software Collections¶
To get started with Software Collections, there are a few additional steps after a fresh installation of CentOS 6 or RHEL 6:
With a brand new RHEL6 installation, ensure the following repositories are enabled:
Install the SCL utilities:
$ sudo yum install scl-utils scl-utils-build
According to https://wiki.centos.org/AdditionalResources/Repositories/SCL you need to have the CentOS Extras repository enabled, from which you can install the SCL release package:
$ sudo yum install centos-release-SCL
Install the SCL utilities:
$ sudo yum install scl-utils scl-utils-build
Updated by bmbouter over 7 years ago
@jcline, this plan is really great and I think we should do just what you suggest. I'm willing to help port my dependencies to SCL on el6.
The one additional observation to add in is that we will only need branches for ["el6", "el7", "f23"] since f22 will be dropped before we finish the SCL work and all f24 software deps should be built directly in Fedora. Long term we will only need to check in deps for el6 and el7.
Updated by email@example.com over 7 years ago
I considered adding a checklist, but thought it might not be very helpful. It would require looking at every dependency of Pulp, deciding if it's a Python package*, deciding on the version we're going to package, looking at its dependencies, deciding if it's a Python package, deciding on what version to package... At that point, I think I might as well do the rest of the work (grabbing the spec file and tarball, checking it into a repo, etc).
- One scenario that I hope we don't run into is this: suppose a library is too old on EL6 to work with a Python package that depends on it. The one I'm most concerned about is openssl and m2crypto/python-cryptography. I really don't want us carrying our own version. The good news is m2crypto is pretty dead as a project so we're probably okay there.
Updated by mhrivnak over 7 years ago
Dropping support for deployment on EL6 is under active discussion right now on the foreman-dev list, and it has a lot of backing both on-list and off-list. If we can get all the katello-related projects to do so, that would eliminate the need (for now) to use SCLs. It's probably worth delaying this SCL work until we get a commitment from all the impacted teams one way or the other.