Project

Profile

Help

Issue #5696

dependency pinning makes packaging pulp3 more complicated than necessary

Added by evgeni 5 months ago. Updated 4 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
3. High
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 61

Description

pulpcore (today) defines the following requirements:

requirements = [
    'aiohttp',
    'aiofiles',
    'backoff',
    'coreapi~=2.3.3',
    'Django~=2.2.3',  # LTS version, switch only if we have a compelling reason to
    'django-filter~=2.2.0',
    'djangorestframework~=3.10.2',
    'djangorestframework-queryfields~=1.0.0',
    'drf-nested-routers~=0.91.0',
    'drf-yasg~=1.17.0',
    'gunicorn~=19.9.0',
    'psycopg2-binary',
    'PyYAML~=5.1.1',
    'rq~=1.1.0',
    'redis~=3.1.0',
    'setuptools>=41.0.1,<41.7.0',
    'dynaconf>=2.2,<2.3',
    'whitenoise~=4.1.3',
]

I understand that the idea behind this is to ensure that it can only be ever installed with tested dependencies (see https://pulp.plan.io/issues/5196 for the original discussion and decision).

However, this raises the following "issues" when deploying pulpcore via (RPM) packages:

As pulpcore uses pkg_resources.get_distribution at runtime, it will error out when psycopg2-binary or a sufficiently new version of setuptools is not found and Pulp won't work.

While I can easily patch both things during package building, I think this warrants a broader discussion how production grade setups shall be handled.

Associated revisions

Revision 3dfd0674 View on GitHub
Added by daviddavis 5 months ago

Fixing package requirements for CentOS 7

Also bump gunicorn version requirement.

fixes #5696

History

#1 Updated by daviddavis 5 months ago

I was thinking that perhaps we could use Pipenv[0] to solve this problem. We would set a loose set of package requirements in our Pipfile that would work with most OSes. But we could also ship a Pipfile.lock that would include a set of specific package versions that we have tested against. Pulp installations from pypi could use the Pipfile.lock while the RPM builds could just rely on Pipfile.

[0] https://github.com/pypa/pipenv

#2 Updated by evgeni 5 months ago

#3 Updated by bmbouter 5 months ago

Regarding maximum versions

I want to share some information about why this is done. It's not because "the idea behind this is to ensure that it can only be ever installed with tested dependencies". The reason is that a breaking change from any dependency released any day could break 100% of pulp environments. The restriction of newer dependencies guards against this risk. For example django has a 2-release removal policy (not semver) so we their is a real breakage risk by not pinning to X.Y for Django for example.

With ^ in mind, it's debatable about which ones should be have newer version limited, and to which level X or X.Y.

Regarding minimum versions

Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.

#4 Updated by daviddavis 5 months ago

evgeni wrote:

IMHO pulpcore is a "library" (c.f. https://pipenv.kennethreitz.org/en/latest/advanced/#pipfile-vs-setuppy), so pipenv is not strictly a solution.

also, pipfile authors also say Pipfiles are not meant to be used for user installation of a package, they are a development tool.

The problem is that Pulp is both an application and a library. And we use one set of package requirements in pulpcore for both. This is probably not optimal as the optimal solution would allow a user (when installing Pulp) to get a set of dependencies that Pulp has been tested against. Pipfile or pinning to y-releases might not be great solutions for that but I'm beginning to think something external to Pulp would be better suited.

#5 Updated by daviddavis 5 months ago

Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.

This hasn't been our policy. We've been pinning to y-releases but a dependabot bug[0] actually doesn't move the minimum version. That said, I am fine with not moving the minimum version and supporting old versions of dependencies. I guess my one concern would be how do we know future changes in Pulp are compatible with old versions of deps? I suppose when we get a user-reported bug, we can bump the minimum?

[0] https://github.com/dependabot/dependabot-core/issues/1440

#6 Updated by dkliban@redhat.com 5 months ago

The subject and the description of this issue seem to be about 2 different issues. The first is about how versions of requirements are expressed in setup.py and the latter is about two specific packages in that list of requirements: psycopg2-binary and setuptools>=41.0.1,<41.7.0. I am interested in resolving the latter.

What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.

What version of setuptools comes with CentOS 7?

#7 Updated by bmbouter 5 months ago

daviddavis wrote:

Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.

This hasn't been our policy. We've been pinning to y-releases but a dependabot bug[0] actually doesn't move the minimum version. That said, I am fine with not moving the minimum version and supporting old versions of dependencies. I guess my one concern would be how do we know future changes in Pulp are compatible with old versions of deps? I suppose when we get a user-reported bug, we can bump the minimum?

Then I'm double glad we're talking about it. The policy that makes sense to me is to allow the oldest working dep, but be future compatible with the newest as long as it's safe (prevent unexpected breaking changes). My understanding with dependabot was that it never moved the minimum, only the maximum, but my expectation from dependabot could be incorrect.

I think we could iterate back the deps until the functional tests fail.

[0] https://github.com/dependabot/dependabot-core/issues/1440

#8 Updated by daviddavis 5 months ago

wrote:

What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.

At the end it of the page calls out that psycopg2-binary is a precompiled binary and it recommends that users actually compile psycopg2 for anything that's not dev/test.

What version of setuptools comes with CentOS 7?

Looks like centos 7 has 39.2.0 of python36-setuptools.

#9 Updated by evgeni 5 months ago

wrote:

The subject and the description of this issue seem to be about 2 different issues. The first is about how versions of requirements are expressed in setup.py and the latter is about two specific packages in that list of requirements: psycopg2-binary and setuptools>=41.0.1,<41.7.0. I am interested in resolving the latter.

So, uh, let's retitle to "dependency definition …"? :)
I don't care about the title when the two mentioned issues are fixed ;)

What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.

The linked page states at the bottom "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources."
My understanding is that this package contains a binary module and the libraries required to load that (libpq, libselinux, etc). This means that besides pulling a new version from PyPI there is no way to update these libraries (which might contain bugs, security issues, etc at some point).
And I would expect it to fail against PostgreSQL versions that are newer than the embedded libpq.

What version of setuptools comes with CentOS 7?

39.2.0, same in CentOS8.

#10 Updated by daviddavis 5 months ago

I opened up a separate issue to continue the discussion about what our dependency policy should be:

https://pulp.plan.io/issues/5697

And opened a PR that I hope will address Zhenech's immediate needs:

https://github.com/pulp/pulpcore/pull/378

#11 Updated by daviddavis 5 months ago

  • Status changed from NEW to POST
  • Assignee set to daviddavis

#12 Updated by fabricio.aguiar 5 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 61

#13 Updated by daviddavis 5 months ago

  • Status changed from POST to MODIFIED

#14 Updated by bmbouter 4 months ago

  • Sprint/Milestone set to 3.0.0

#15 Updated by bmbouter 4 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF