Pulp

I was thinking that perhaps we could use Pipenv[0] to solve this problem. We would set a loose set of package requirements in our Pipfile that would work with most OSes. But we could also ship a Pipfile.lock that would include a set of specific package versions that we have tested against. Pulp installations from pypi could use the Pipfile.lock while the RPM builds could just rely on Pipfile.

[0] https://github.com/pypa/pipenv

IMHO pulpcore is a "library" (c.f. https://pipenv.kennethreitz.org/en/latest/advanced/#pipfile-vs-setuppy), so pipenv is not strictly a solution.

also, pipfile authors also say Pipfiles are not meant to be used for user installation of a package, they are a development tool.

Regarding maximum versions¶

I want to share some information about why this is done. It's not because "the idea behind this is to ensure that it can only be ever installed with tested dependencies". The reason is that a breaking change from any dependency released any day could break 100% of pulp environments. The restriction of newer dependencies guards against this risk. For example django has a 2-release removal policy (not semver) so we their is a real breakage risk by not pinning to X.Y for Django for example.

With ^ in mind, it's debatable about which ones should be have newer version limited, and to which level X or X.Y.

Regarding minimum versions¶

Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.

evgeni wrote:

IMHO pulpcore is a "library" (c.f. https://pipenv.kennethreitz.org/en/latest/advanced/#pipfile-vs-setuppy), so pipenv is not strictly a solution.

also, pipfile authors also say Pipfiles are not meant to be used for user installation of a package, they are a development tool.

The problem is that Pulp is both an application and a library. And we use one set of package requirements in pulpcore for both. This is probably not optimal as the optimal solution would allow a user (when installing Pulp) to get a set of dependencies that Pulp has been tested against. Pipfile or pinning to y-releases might not be great solutions for that but I'm beginning to think something external to Pulp would be better suited.

Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.

This hasn't been our policy. We've been pinning to y-releases but a dependabot bug[0] actually doesn't move the minimum version. That said, I am fine with not moving the minimum version and supporting old versions of dependencies. I guess my one concern would be how do we know future changes in Pulp are compatible with old versions of deps? I suppose when we get a user-reported bug, we can bump the minimum?

[0] https://github.com/dependabot/dependabot-core/issues/1440

The subject and the description of this issue seem to be about 2 different issues. The first is about how versions of requirements are expressed in setup.py and the latter is about two specific packages in that list of requirements: psycopg2-binary and setuptools>=41.0.1,<41.7.0. I am interested in resolving the latter.

What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.

What version of setuptools comes with CentOS 7?

daviddavis wrote:

Pulp has minimum version requirements, so once set I expect the minimum version to never move unless we are specifically upgrading for some reason. For example our RQ minimum contains a feature we need specifically or Pulp won't run.

This hasn't been our policy. We've been pinning to y-releases but a dependabot bug[0] actually doesn't move the minimum version. That said, I am fine with not moving the minimum version and supporting old versions of dependencies. I guess my one concern would be how do we know future changes in Pulp are compatible with old versions of deps? I suppose when we get a user-reported bug, we can bump the minimum?

Then I'm double glad we're talking about it. The policy that makes sense to me is to allow the oldest working dep, but be future compatible with the newest as long as it's safe (prevent unexpected breaking changes). My understanding with dependabot was that it never moved the minimum, only the maximum, but my expectation from dependabot could be incorrect.

I think we could iterate back the deps until the functional tests fail.

[0] https://github.com/dependabot/dependabot-core/issues/1440

dkliban@redhat.com wrote:

What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.

At the end it of the page calls out that psycopg2-binary is a precompiled binary and it recommends that users actually compile psycopg2 for anything that's not dev/test.

What version of setuptools comes with CentOS 7?

Looks like centos 7 has 39.2.0 of python36-setuptools.

dkliban@redhat.com wrote:

The subject and the description of this issue seem to be about 2 different issues. The first is about how versions of requirements are expressed in setup.py and the latter is about two specific packages in that list of requirements: psycopg2-binary and setuptools>=41.0.1,<41.7.0. I am interested in resolving the latter.

So, uh, let's retitle to "dependency definition …"? :)
I don't care about the title when the two mentioned issues are fixed ;)

What makes psycopg2-binary not safe for production use? I did not see anything in the PyPI page you linked.

The linked page states at the bottom "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources."
My understanding is that this package contains a binary module and the libraries required to load that (libpq, libselinux, etc). This means that besides pulling a new version from PyPI there is no way to update these libraries (which might contain bugs, security issues, etc at some point).
And I would expect it to fail against PostgreSQL versions that are newer than the embedded libpq.

What version of setuptools comes with CentOS 7?

39.2.0, same in CentOS8.

I opened up a separate issue to continue the discussion about what our dependency policy should be:

https://pulp.plan.io/issues/5697

And opened a PR that I hope will address Zhenech's immediate needs:

https://github.com/pulp/pulpcore/pull/378