Project

Profile

Help

Story #1852

closed

Move Pulp to a Software Collection on EL6

Added by jcline@redhat.com about 8 years ago. Updated about 4 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

This story is the result of https://pulp.plan.io/issues/931. Currently, all Pulp code must be Python 2.6 compatible. This makes transitioning to Python 3 compatible code difficult, as many features introduced in 2.7 are specifically for writing code that runs on 2.7 and 3.x. Furthermore, we are stuck with a very old version of Django (1.4). Therefore, it would be helpful if we could raise our minimum to requirements to Python 2.7 and Django 1.6 (for now). Both these ship with RHEL 7 so in order to support RHEL6, we can use Software Collections.

Software Collections[0] install into /opt/ or /var/opt, so they don't interfere with the versions that ship with the OS. We need to extend the existing Python 2.7 SCL[1] with all our Python dependencies. For each Python dependency we have that isn't provided by the Python 2.7 SCL, we must create an RPM package that uses the SCL. This can be done by acquiring the existing spec files for these dependencies and tweaking them (potentially with spec2scl[2]). Note that this applies not just to our direct dependencies, but also the dependencies of those dependencies, all the way to the packages that have no dependencies beyond those already provided by the Python 2.7 SCL or the platform itself.

Once all the dependencies are packaged for the SCL, Pulp itself needs its spec file modified to use the SCL. It also needs the init scripts and commands installed into /usr/bin to use the SCL paths. All these changes can occur as patches applied by the spec file (or even as install steps in the spec itself).

All these spec files (and their associated source tarballs) need to be tracked. The Foreman has a SCL and they use Git with git-annex[3]. We could probably do the same. My recommendation is that we move our spec files out of the pulp and pulp_<plugin> repos and into a package repository. Within that repository we should have branches for each platform, much like Fedora does (so an "el6", "el7", "f23", and "f24" branch). For the "el6" branch, we check in all the SCL-related files. This has two advantages: we get rid of all our nasty if/else blocks for various platforms and we don't have to add more for the SCL work.

Here is a potentially incomplete dependency tree for pulp-server on EL6:

pulp-server-2.8.2-1.el6.noarch [cmd line]
 \_  Django14-1.4.21-1.el6.noarch [1: Django14]
 \_  m2crypto-0.20.2-9.el6.x86_64 [1: m2crypto]
 \_  mod_wsgi-3.4-2.pulp.el6.x86_64 [1: mod_wsgi >= 3.4-1.pulp]
 \_  python-blinker-1.1-1.el6.noarch [1: python-blinker]
 \_  python-celery-3.1.11-1.el6.noarch [2: python-celery < 3.2.0, python-celery >= 3.1.0]
 |   \_  pyparsing-1.5.6-1.el6.noarch [1: pyparsing]
 |   \_  python-amqp-1.4.9-1.el6.noarch [1: python-amqp]
 |   \_  1:python-billiard-3.3.0.17-2.el6.x86_64 [1: python-billiard >= 3.3.0.17]
 |   \_  python-dateutil-1.4.1-6.el6.noarch [1: python-dateutil]
 |   |   \_  tzdata-2016c-1.el6.noarch [1: tzdata]
 |   \_  python-importlib-1.0.2-1.el6.noarch [1: python-importlib]
 |   \_  1:python-kombu-3.0.33-4.pulp.el6.noarch [1: python-kombu >= 3.0.15]
 |   |   \_  python-amqp-1.4.9-1.el6.noarch [2: python-amqp < 2.0, python-amqp >= 1.4.9]
 |   |   \_  python-libs-2.6.6-64.el6.i686 [1: python-ordereddict]
 |   |   \_  python-libs-2.6.6-64.el6.x86_64 [1: python-ordereddict]
 |   |   \_  python-ordereddict-1.1-2.el6.noarch [1: python-ordereddict]
 |   \_  python-setuptools-0.6.10-3.el6.noarch [1: python-setuptools]
 |   \_  pytz-2010h-2.el6.noarch [1: pytz]
 \_  python-gofer-2.7.5-1.el6.noarch [1: python-gofer >= 2.5]
 \_  python-httplib2-0.7.7-1.el6.noarch [1: python-httplib2]
 \_  python-ldap-2.3.10-1.el6.x86_64 [1: python-ldap]
 \_  python-mongoengine-0.10.5-1.el6.noarch [1: python-mongoengine >= 0.10.0]
 |   \_  python-blinker-1.1-1.el6.noarch [1: python-blinker]
 |   \_  python-pymongo-3.2-1.el6.x86_64 [1: python-pymongo >= 3.2]
 |   \_  python-pymongo-gridfs-3.2-1.el6.x86_64 [1: python-pymongo-gridfs >= 3.2]
 |   |   \_  python-pymongo-3.2-1.el6.x86_64 [1: python-pymongo(x86-64) = 3.2-1.el6]
 \_  python-nectar-1.5.1-1.el6.noarch [1: python-nectar >= 1.5.0]
 |   \_  python-requests-2.6.0-3.el6.noarch [1: python-requests >= 2.4.3]
 |   |   \_  python-chardet-2.2.1-1.el6.noarch [1: python-chardet >= 2.2.1-1]
 |   |   \_  python-libs-2.6.6-64.el6.i686 [1: python-ordereddict >= 1.1]
 |   |   \_  python-libs-2.6.6-64.el6.x86_64 [1: python-ordereddict >= 1.1]
 |   |   \_  python-ordereddict-1.1-2.el6.noarch [1: python-ordereddict >= 1.1]
 |   |   \_  python-urllib3-1.10.2-1.el6.noarch [1: python-urllib3 >= 1.10.2-1]
 |   |   |   \_  python-backports-ssl_match_hostname-3.4.0.2-2.el6.noarch [1: python-backports-ssl_match_hostname]
 |   |   |   |   \_  python-backports-1.0-5.el6.x86_64 [1: python-backports]
 |   |   |   \_  python-libs-2.6.6-64.el6.i686 [1: python-ordereddict]
 |   |   |   \_  python-libs-2.6.6-64.el6.x86_64 [1: python-ordereddict]
 |   |   |   \_  python-ordereddict-1.1-2.el6.noarch [1: python-ordereddict]
 |   |   |   \_  python-six-1.9.0-2.el6.noarch [1: python-six]
 \_  python-oauth2-1.5.211-8.el6.noarch [1: python-oauth2 >= 1.5.211]
 |   \_  python-httplib2-0.7.7-1.el6.noarch [1: python-httplib2]
 \_  python-pulp-common-2.8.2-1.el6.noarch [1: python-pulp-common = 2.8.2]
 |   \_  python-iniparse-0.3.1-2.1.el6.noarch [1: python-iniparse]
 \_  python-pulp-repoauth-2.8.2-1.el6.noarch [1: python-pulp-repoauth = 2.8.2]
 |   \_  mod_wsgi-3.4-2.pulp.el6.x86_64 [1: mod_wsgi >= 3.4-1.pulp]
 |   \_  python-pulp-common-2.8.2-1.el6.noarch [1: python-pulp-common = 2.8.2]
 |   \_  python-rhsm-0.99.12-1.el6.noarch [1: python-rhsm]
 |   |   \_  m2crypto-0.20.2-9.el6.x86_64 [1: m2crypto]
 |   |   \_  python-iniparse-0.3.1-2.1.el6.noarch [1: python-iniparse]
 |   \_  python-rhsm-1.14.3-1.el6.x86_64 [1: python-rhsm]
 |   |   \_  m2crypto-0.20.2-9.el6.x86_64 [1: m2crypto]
 |   |   \_  python-dateutil-1.4.1-6.el6.noarch [1: python-dateutil]
 |   |   \_  python-iniparse-0.3.1-2.1.el6.noarch [1: python-iniparse]
 |   |   \_  rpm-python-4.8.0-47.el6.x86_64 [1: rpm-python]
 |   \_  python-setuptools-0.6.10-3.el6.noarch [1: python-setuptools]
 \_  python-pymongo-3.2-1.el6.x86_64 [1: python-pymongo >= 3.0.0]
 \_  python-qpid-0.14-11.el6_3.noarch [1: python-qpid]
 |   \_  python-saslwrapper-0.14-1.el6.x86_64 [1: python-saslwrapper >= 0.10]
 \_  python-semantic_version-2.2.0-6.el6.noarch [1: python-semantic_version >= 2.2.0]
 \_  python-setuptools-0.6.10-3.el6.noarch [1: python-setuptools]

In the above list I stripped out all dependencies to non-Python packages. It could be I accidentally got rid of a Python library I simply didn't recognize, so the tree should not be treated as an authoritative reference.

[0] https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/2/html-single/Packaging_Guide/index.html
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/2/html-single/Packaging_Guide/index.html#sect-Extending_the_python27_and_rh-python34_Software_Collections
[2] https://pypi.python.org/pypi/spec2scl
[3] https://github.com/theforeman/foreman-packaging/tree/rpm/1.11

Getting Software Collections

To get started with Software Collections, there are a few additional steps after a fresh installation of CentOS 6 or RHEL 6:

RHEL6

With a brand new RHEL6 installation, ensure the following repositories are enabled:

  • rhel-server-rhscl-6-rpms
  • rhel-6-server-rpms
  • rhel-6-server-optional-rpms

Install the SCL utilities:

$ sudo yum install scl-utils scl-utils-build

CentOS 6

According to https://wiki.centos.org/AdditionalResources/Repositories/SCL you need to have the CentOS Extras repository enabled, from which you can install the SCL release package:

$ sudo yum install centos-release-SCL

Install the SCL utilities:

$ sudo yum install scl-utils scl-utils-build

Files

pulp-server-deps-el6 (179 KB) pulp-server-deps-el6 A full dependency tree for Pulp 2.8 on EL6, generated with the Python script found in http://www.redhat.com/archives/fedora-devel-list/2008-September/msg00001.html jcline@redhat.com, 04/20/2016 03:22 PM

Related issues

Blocked by Pulp - Task #1993: Clean up our distribution packaging by moving it to a dedicated repository and building with coprCLOSED - CURRENTRELEASEsemyers

Actions
Blocks Pulp - Task #2107: Unpin flake8 version from testing requirementsCLOSED - WONTFIX

Actions
Actions #1

Updated by bmbouter about 8 years ago

@jcline, this plan is really great and I think we should do just what you suggest. I'm willing to help port my dependencies to SCL on el6.

The one additional observation to add in is that we will only need branches for ["el6", "el7", "f23"] since f22 will be dropped before we finish the SCL work and all f24 software deps should be built directly in Fedora. Long term we will only need to check in deps for el6 and el7.

Actions #2

Updated by bmbouter about 8 years ago

Maybe add some checklist items to this, one for each dependency?

Actions #3

Updated by jcline@redhat.com about 8 years ago

I considered adding a checklist, but thought it might not be very helpful. It would require looking at every dependency of Pulp, deciding if it's a Python package*, deciding on the version we're going to package, looking at its dependencies, deciding if it's a Python package, deciding on what version to package... At that point, I think I might as well do the rest of the work (grabbing the spec file and tarball, checking it into a repo, etc).

  • One scenario that I hope we don't run into is this: suppose a library is too old on EL6 to work with a Python package that depends on it. The one I'm most concerned about is openssl and m2crypto/python-cryptography. I really don't want us carrying our own version. The good news is m2crypto is pretty dead as a project so we're probably okay there.
Actions #4

Updated by jcline@redhat.com about 8 years ago

  • Description updated (diff)
Actions #6

Updated by jcline@redhat.com about 8 years ago

  • Description updated (diff)
Actions #7

Updated by mhrivnak about 8 years ago

  • Sprint/Milestone set to 20
Actions #8

Updated by mhrivnak about 8 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #9

Updated by rbarlow about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to rbarlow
Actions #10

Updated by rbarlow about 8 years ago

This is going to be EPIC.

Actions #11

Updated by mhrivnak about 8 years ago

Dropping support for deployment on EL6 is under active discussion right now on the foreman-dev list, and it has a lot of backing both on-list and off-list. If we can get all the katello-related projects to do so, that would eliminate the need (for now) to use SCLs. It's probably worth delaying this SCL work until we get a commitment from all the impacted teams one way or the other.

Actions #13

Updated by mhrivnak about 8 years ago

  • Sprint/Milestone changed from 20 to 21
Actions #14

Updated by mhrivnak about 8 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (rbarlow)
Actions #15

Updated by jcline@redhat.com about 8 years ago

  • Blocked by Task #1993: Clean up our distribution packaging by moving it to a dedicated repository and building with copr added
Actions #16

Updated by jcline@redhat.com about 8 years ago

  • Sprint/Milestone deleted (21)
Actions #17

Updated by mhrivnak almost 8 years ago

  • Sprint Candidate changed from Yes to No
Actions #18

Updated by bmbouter almost 8 years ago

  • Sprint Candidate changed from No to Yes

Adding back for consideration on Sprint 5

Actions #19

Updated by amacdona@redhat.com almost 8 years ago

  • Blocks Task #2107: Unpin flake8 version from testing requirements added
Actions #20

Updated by mhrivnak almost 8 years ago

  • Sprint Candidate changed from Yes to No

Having conferred with bmbouter, this is no longer a sprint candidate.

Actions #21

Updated by mhrivnak over 7 years ago

  • Status changed from NEW to CLOSED - WONTFIX

EL6 deployment support is being dropped after 2.11.

Actions #22

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF