Issue #5696
closeddependency pinning makes packaging pulp3 more complicated than necessary
Description
pulpcore (today) defines the following requirements:
requirements = [
'aiohttp',
'aiofiles',
'backoff',
'coreapi~=2.3.3',
'Django~=2.2.3', # LTS version, switch only if we have a compelling reason to
'django-filter~=2.2.0',
'djangorestframework~=3.10.2',
'djangorestframework-queryfields~=1.0.0',
'drf-nested-routers~=0.91.0',
'drf-yasg~=1.17.0',
'gunicorn~=19.9.0',
'psycopg2-binary',
'PyYAML~=5.1.1',
'rq~=1.1.0',
'redis~=3.1.0',
'setuptools>=41.0.1,<41.7.0',
'dynaconf>=2.2,<2.3',
'whitenoise~=4.1.3',
]
I understand that the idea behind this is to ensure that it can only be ever installed with tested dependencies (see https://pulp.plan.io/issues/5196 for the original discussion and decision).
However, this raises the following "issues" when deploying pulpcore via (RPM) packages:
- psycopg2-binary is not meant for production setups and people shall use psycopg2, because the binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.
- setuptools is usually already provided by the OS, but often an older version (albeit one that should still work fine for the pkg_resources calls that pulpcore does).
As pulpcore uses pkg_resources.get_distribution at runtime, it will error out when psycopg2-binary or a sufficiently new version of setuptools is not found and Pulp won't work.
While I can easily patch both things during package building, I think this warrants a broader discussion how production grade setups shall be handled.
Updated by daviddavis almost 5 years ago
I was thinking that perhaps we could use Pipenv[0] to solve this problem. We would set a loose set of package requirements in our Pipfile that would work with most OSes. But we could also ship a Pipfile.lock that would include a set of specific package versions that we have tested against. Pulp installations from pypi could use the Pipfile.lock while the RPM builds could just rely on Pipfile.
Updated by evgeni almost 5 years ago
IMHO pulpcore is a "library" (c.f. https://pipenv.kennethreitz.org/en/latest/advanced/#pipfile-vs-setuppy), so pipenv is not strictly a solution.
also, pipfile authors also say Pipfiles are not meant to be used for user installation of a package, they are a development tool.
Updated by bmbouter almost 5 years ago
Regarding maximum versions¶
I want to share some information about why this is done. It's not because "the idea behind this is to ensure that it can only be ever installed with tested dependencies". The reason is that a breaking change from any dependency released any day could break 100% of pulp environments. The restriction of newer dependencies guards against this risk. For example django has a 2-release removal policy (not semver) so we their is a real breakage risk by not pinning to X.Y for Django for example.
With ^ in mind, it's debatable about which ones should be have newer version limited, and to which level X or X.Y.
Regarding minimum versions¶
Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.
Updated by daviddavis almost 5 years ago
evgeni wrote:
IMHO pulpcore is a "library" (c.f. https://pipenv.kennethreitz.org/en/latest/advanced/#pipfile-vs-setuppy), so pipenv is not strictly a solution.
also, pipfile authors also say Pipfiles are not meant to be used for user installation of a package, they are a development tool.
The problem is that Pulp is both an application and a library. And we use one set of package requirements in pulpcore for both. This is probably not optimal as the optimal solution would allow a user (when installing Pulp) to get a set of dependencies that Pulp has been tested against. Pipfile or pinning to y-releases might not be great solutions for that but I'm beginning to think something external to Pulp would be better suited.
Updated by daviddavis almost 5 years ago
Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.
This hasn't been our policy. We've been pinning to y-releases but a dependabot bug[0] actually doesn't move the minimum version. That said, I am fine with not moving the minimum version and supporting old versions of dependencies. I guess my one concern would be how do we know future changes in Pulp are compatible with old versions of deps? I suppose when we get a user-reported bug, we can bump the minimum?
[0] https://github.com/dependabot/dependabot-core/issues/1440
Updated by dkliban@redhat.com almost 5 years ago
The subject and the description of this issue seem to be about 2 different issues. The first is about how versions of requirements are expressed in setup.py and the latter is about two specific packages in that list of requirements: psycopg2-binary and setuptools>=41.0.1,<41.7.0. I am interested in resolving the latter.
What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.
What version of setuptools comes with CentOS 7?
Updated by bmbouter almost 5 years ago
daviddavis wrote:
Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.
This hasn't been our policy. We've been pinning to y-releases but a dependabot bug[0] actually doesn't move the minimum version. That said, I am fine with not moving the minimum version and supporting old versions of dependencies. I guess my one concern would be how do we know future changes in Pulp are compatible with old versions of deps? I suppose when we get a user-reported bug, we can bump the minimum?
Then I'm double glad we're talking about it. The policy that makes sense to me is to allow the oldest working dep, but be future compatible with the newest as long as it's safe (prevent unexpected breaking changes). My understanding with dependabot was that it never moved the minimum, only the maximum, but my expectation from dependabot could be incorrect.
I think we could iterate back the deps until the functional tests fail.
[0] https://github.com/dependabot/dependabot-core/issues/1440
Updated by daviddavis almost 5 years ago
dkliban@redhat.com wrote:
What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.
At the end it of the page calls out that psycopg2-binary is a precompiled binary and it recommends that users actually compile psycopg2 for anything that's not dev/test.
What version of setuptools comes with CentOS 7?
Looks like centos 7 has 39.2.0 of python36-setuptools.
Updated by evgeni almost 5 years ago
dkliban@redhat.com wrote:
The subject and the description of this issue seem to be about 2 different issues. The first is about how versions of requirements are expressed in setup.py and the latter is about two specific packages in that list of requirements: psycopg2-binary and setuptools>=41.0.1,<41.7.0. I am interested in resolving the latter.
So, uh, let's retitle to "dependency definition …"? :)
I don't care about the title when the two mentioned issues are fixed ;)
What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.
The linked page states at the bottom "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources."
My understanding is that this package contains a binary module and the libraries required to load that (libpq, libselinux, etc). This means that besides pulling a new version from PyPI there is no way to update these libraries (which might contain bugs, security issues, etc at some point).
And I would expect it to fail against PostgreSQL versions that are newer than the embedded libpq.
What version of setuptools comes with CentOS 7?
39.2.0, same in CentOS8.
Updated by daviddavis almost 5 years ago
I opened up a separate issue to continue the discussion about what our dependency policy should be:
https://pulp.plan.io/issues/5697
And opened a PR that I hope will address Zhenech's immediate needs:
Updated by daviddavis almost 5 years ago
- Status changed from NEW to POST
- Assignee set to daviddavis
Updated by fao89 almost 5 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 61
Added by daviddavis almost 5 years ago
Updated by daviddavis almost 5 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulpcore|3dfd06745da718f529cf8a31b16281a5b3c04de3.
Updated by bmbouter over 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Fixing package requirements for CentOS 7
Also bump gunicorn version requirement.
fixes #5696