Issue #6189
closed
Story #5890: [Epic] The ansible-pulp installer needs to handle multiple versions of Pulp
ansible-pulp should not upgrade pulpcore if any plugins can't work with the newer version
Added by dkliban@redhat.com about 4 years ago. Updated almost 4 years ago.
Description
The pulp_version variable in ansible-pulp specifies which version of pulpcore should be installed. Before performing an installation of that version of pulpcore and installing any plugins, the installer should use pip-compile to detect version conflicts. If pip-compile reports conflicts between packages, the user should be notified that one of the plugins is not compatible with the version of pulpcore requested. If no conflicts are found, the installer should continue with the install.
Updated by mdepaulo@redhat.com about 4 years ago
I have some issues with the temporary virtualenv approach:
- It will require us to install all the C dependencies first, and thus will modify the system.
- It will take a while.
Is it possible to use any higher level package managers for pip to determine if it is safe? Those I know of (and tested for a way to determine & install the newest version of each plugin compatible with the fixed pulpcore version, but was unsuccessful at that higher goal) are:
- pip-tools
- poetry
- pipenv
Updated by dkliban@redhat.com about 4 years ago
If pip uses a cache directory when doing the pre-install, that same cache directory can be reused by the actual install.
Updated by mdepaulo@redhat.com about 4 years ago
Here's a test of pip-tools (pip-check, pip-compile & pip-sync) doing this successfully for 4 test cases.
Note that the 3rd pip-compile command succeeds because it has the previous requirements.txt, where as the 4th pip-compile fails because it only looks at the latest pulp_rpm
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ printf "pulpcore==3.0\npulp-rpm==3.0\n" > requirements.in
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ cat requirements.in
pulpcore==3.0
pulp-rpm==3.0
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile
#
aiofiles==0.4.0 # via pulpcore
aiohttp==3.6.2 # via pulpcore
async-timeout==3.0.1 # via aiohttp
attrs==19.3.0 # via aiohttp
backoff==1.10.0 # via pulpcore
certifi==2019.11.28 # via requests
chardet==3.0.4 # via aiohttp, requests
click==7.0 # via dynaconf, rq
coreapi==2.3.3 # via drf-yasg, pulpcore
coreschema==0.0.4 # via coreapi, drf-yasg
createrepo-c==0.15.5 # via pulp-rpm
django-filter==2.2.0 # via pulpcore
django==2.2.10 # via django-filter, drf-nested-routers, drf-yasg, pulpcore
djangorestframework-queryfields==1.0.0 # via pulpcore
djangorestframework==3.10.3 # via drf-nested-routers, drf-yasg, pulpcore
drf-nested-routers==0.91 # via pulpcore
drf-yasg==1.17.1 # via pulpcore
dynaconf==2.2.2 # via pulpcore
gunicorn==20.0.4 # via pulpcore
idna==2.9 # via requests, yarl
inflection==0.3.1 # via drf-yasg
itypes==1.1.0 # via coreapi
jinja2==2.11.1 # via coreschema
libcomps==0.1.14.post1 # via pulp-rpm
markupsafe==1.1.1 # via jinja2
multidict==4.7.4 # via aiohttp, yarl
packaging==20.1 # via drf-yasg
productmd==1.24 # via pulp-rpm
psycopg2==2.8.4 # via pulpcore
pulp-rpm==3.0
pulpcore==3.0.0
pycairo==1.19.1 # via pygobject
pygobject==3.34.0 # via pulp-rpm
pygtrie==2.3.2 # via pulpcore
pyparsing==2.4.6 # via packaging
python-box==3.4.6 # via dynaconf
python-dotenv==0.10.3 # via dynaconf
pytz==2019.3 # via django
pyyaml==5.2 # via pulpcore
redis==3.1.0 # via pulpcore, rq
requests==2.23.0 # via coreapi
rq==1.1.0 # via pulpcore
ruamel.yaml.clib==0.2.0 # via ruamel.yaml
ruamel.yaml==0.16.10 # via drf-yasg
six==1.14.0 # via drf-yasg, packaging, productmd
sqlparse==0.3.0 # via django
toml==0.10.0 # via dynaconf
uritemplate==3.0.1 # via coreapi, drf-yasg
urllib3==1.25.8 # via requests
whitenoise==4.1.4 # via pulpcore
yarl==1.4.2 # via aiohttp
# The following packages are considered to be unsafe in a requirements file:
# setuptools
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ printf "pulpcore==3.0\npulp-rpm==3.1\n" > requirements.in
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ cat requirements.in
pulpcore==3.0
pulp-rpm==3.1
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ pip-compile
Could not find a version that matches pulpcore<3.2,==3.0,>=3.1 (from -r requirements.in (line 1))
Tried: 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1
Skipped pre-versions: 3.0.0a1.dev3, 3.0.0a1.dev3, 3.0.0a1.dev4, 3.0.0a1.dev4, 3.0.0a1.dev5, 3.0.0a1.dev5, 3.0.0a1, 3.0.0a1, 3.0.0a2, 3.0.0a2, 3.0.0a3, 3.0.0a3, 3.0.0a4, 3.0.0a4, 3.0.0a5, 3.0.0a5, 3.0.0a6, 3.0.0a6, 3.0.0a7, 3.0.0a7, 3.0.0a8, 3.0.0a8, 3.0.0a9, 3.0.0a9, 3.0.0a10, 3.0.0a10, 3.0.0a11, 3.0.0a11, 3.0.0a12, 3.0.0a12, 3.0.0a13, 3.0.0a13, 3.0.0a14, 3.0.0a14, 3.0.0a15, 3.0.0a15, 3.0.0a16, 3.0.0a16, 3.0.0a17, 3.0.0a17, 3.0.0a18, 3.0.0a18, 3.0.0a19, 3.0.0a19, 3.0.0a20, 3.0.0a20, 3.0.0a21, 3.0.0a21, 3.0.0a22, 3.0.0a22, 3.0.0a23, 3.0.0a23, 3.0.0a24, 3.0.0a24, 3.0.0a25, 3.0.0a25, 3.0.0a26, 3.0.0a26, 3.0.0a27, 3.0.0a27, 3.0.0a28, 3.0.0a28, 3.0.0a29, 3.0.0a29, 3.0.0b1, 3.0.0b1, 3.0.0b2, 3.0.0b2, 3.0.0b3, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.0.0b5, 3.0.0b5, 3.0.0b6, 3.0.0b6, 3.0.0b7, 3.0.0b7, 3.0.0b8, 3.0.0b8, 3.0.0b9, 3.0.0b9, 3.0.0b10, 3.0.0b10, 3.0.0b11, 3.0.0b11, 3.0.0b12, 3.0.0b12, 3.0.0b13, 3.0.0b13, 3.0.0b14, 3.0.0b14, 3.0.0b15, 3.0.0b15, 3.0.0b16, 3.0.0b16, 3.0.0b17, 3.0.0b18, 3.0.0b18, 3.0.0b19, 3.0.0b19, 3.0.0b20, 3.0.0b20, 3.0.0b21, 3.0.0b21, 3.0.0b22, 3.0.0b22, 3.0.0b23, 3.0.0b23, 3.0.0rc1, 3.0.0rc1, 3.0.0rc2, 3.0.0rc2, 3.0.0rc3, 3.0.0rc3, 3.0.0rc4, 3.0.0rc4, 3.0.0rc5, 3.0.0rc5, 3.0.0rc6, 3.0.0rc6, 3.0.0rc7, 3.0.0rc7, 3.0.0rc8, 3.0.0rc8, 3.0.0rc9, 3.0.0rc9
There are incompatible versions in the resolved dependencies:
pulpcore==3.0 (from -r requirements.in (line 1))
pulpcore<3.2,>=3.1 (from pulp-rpm==3.1->-r requirements.in (line 2))
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ printf "pulpcore==3.0\npulp-rpm\n" > requirements.in
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ cat requirements.in
pulpcore==3.0
pulp-rpm
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile
#
aiofiles==0.4.0 # via pulpcore
aiohttp==3.6.2 # via pulpcore
async-timeout==3.0.1 # via aiohttp
attrs==19.3.0 # via aiohttp
backoff==1.10.0 # via pulpcore
certifi==2019.11.28 # via requests
chardet==3.0.4 # via aiohttp, requests
click==7.0 # via dynaconf, rq
coreapi==2.3.3 # via drf-yasg, pulpcore
coreschema==0.0.4 # via coreapi, drf-yasg
createrepo-c==0.15.5 # via pulp-rpm
django-filter==2.2.0 # via pulpcore
django==2.2.10 # via django-filter, drf-nested-routers, drf-yasg, pulpcore
djangorestframework-queryfields==1.0.0 # via pulpcore
djangorestframework==3.10.3 # via drf-nested-routers, drf-yasg, pulpcore
drf-nested-routers==0.91 # via pulpcore
drf-yasg==1.17.1 # via pulpcore
dynaconf==2.2.2 # via pulpcore
gunicorn==20.0.4 # via pulpcore
idna==2.9 # via requests, yarl
inflection==0.3.1 # via drf-yasg
itypes==1.1.0 # via coreapi
jinja2==2.11.1 # via coreschema
libcomps==0.1.14.post1 # via pulp-rpm
markupsafe==1.1.1 # via jinja2
multidict==4.7.4 # via aiohttp, yarl
packaging==20.1 # via drf-yasg
productmd==1.24 # via pulp-rpm
psycopg2==2.8.4 # via pulpcore
pulp-rpm==3.0
pulpcore==3.0.0
pycairo==1.19.1 # via pygobject
pygobject==3.34.0 # via pulp-rpm
pygtrie==2.3.2 # via pulpcore
pyparsing==2.4.6 # via packaging
python-box==3.4.6 # via dynaconf
python-dotenv==0.10.3 # via dynaconf
pytz==2019.3 # via django
pyyaml==5.2 # via pulpcore
redis==3.1.0 # via pulpcore, rq
requests==2.23.0 # via coreapi
rq==1.1.0 # via pulpcore
ruamel.yaml.clib==0.2.0 # via ruamel.yaml
ruamel.yaml==0.16.10 # via drf-yasg
six==1.14.0 # via drf-yasg, packaging, productmd
sqlparse==0.3.0 # via django
toml==0.10.0 # via dynaconf
uritemplate==3.0.1 # via coreapi, drf-yasg
urllib3==1.25.8 # via requests
whitenoise==4.1.4 # via pulpcore
yarl==1.4.2 # via aiohttp
# The following packages are considered to be unsafe in a requirements file:
# setuptools
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ rm requirements.txt
(pip-tools) [mdepaulo@mdepaulo pip-tools]$ pip-compile
Could not find a version that matches pulpcore<3.2,==3.0,>=3.1 (from -r requirements.in (line 1))
Tried: 3.0.0, 3.0.0, 3.0.1, 3.0.1, 3.1.0, 3.1.0, 3.1.1, 3.1.1
Skipped pre-versions: 3.0.0a1.dev3, 3.0.0a1.dev3, 3.0.0a1.dev4, 3.0.0a1.dev4, 3.0.0a1.dev5, 3.0.0a1.dev5, 3.0.0a1, 3.0.0a1, 3.0.0a2, 3.0.0a2, 3.0.0a3, 3.0.0a3, 3.0.0a4, 3.0.0a4, 3.0.0a5, 3.0.0a5, 3.0.0a6, 3.0.0a6, 3.0.0a7, 3.0.0a7, 3.0.0a8, 3.0.0a8, 3.0.0a9, 3.0.0a9, 3.0.0a10, 3.0.0a10, 3.0.0a11, 3.0.0a11, 3.0.0a12, 3.0.0a12, 3.0.0a13, 3.0.0a13, 3.0.0a14, 3.0.0a14, 3.0.0a15, 3.0.0a15, 3.0.0a16, 3.0.0a16, 3.0.0a17, 3.0.0a17, 3.0.0a18, 3.0.0a18, 3.0.0a19, 3.0.0a19, 3.0.0a20, 3.0.0a20, 3.0.0a21, 3.0.0a21, 3.0.0a22, 3.0.0a22, 3.0.0a23, 3.0.0a23, 3.0.0a24, 3.0.0a24, 3.0.0a25, 3.0.0a25, 3.0.0a26, 3.0.0a26, 3.0.0a27, 3.0.0a27, 3.0.0a28, 3.0.0a28, 3.0.0a29, 3.0.0a29, 3.0.0b1, 3.0.0b1, 3.0.0b2, 3.0.0b2, 3.0.0b3, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.0.0b5, 3.0.0b5, 3.0.0b6, 3.0.0b6, 3.0.0b7, 3.0.0b7, 3.0.0b8, 3.0.0b8, 3.0.0b9, 3.0.0b9, 3.0.0b10, 3.0.0b10, 3.0.0b11, 3.0.0b11, 3.0.0b12, 3.0.0b12, 3.0.0b13, 3.0.0b13, 3.0.0b14, 3.0.0b14, 3.0.0b15, 3.0.0b15, 3.0.0b16, 3.0.0b16, 3.0.0b17, 3.0.0b18, 3.0.0b18, 3.0.0b19, 3.0.0b19, 3.0.0b20, 3.0.0b20, 3.0.0b21, 3.0.0b21, 3.0.0b22, 3.0.0b22, 3.0.0b23, 3.0.0b23, 3.0.0rc1, 3.0.0rc1, 3.0.0rc2, 3.0.0rc2, 3.0.0rc3, 3.0.0rc3, 3.0.0rc4, 3.0.0rc4, 3.0.0rc5, 3.0.0rc5, 3.0.0rc6, 3.0.0rc6, 3.0.0rc7, 3.0.0rc7, 3.0.0rc8, 3.0.0rc8, 3.0.0rc9, 3.0.0rc9
There are incompatible versions in the resolved dependencies:
pulpcore==3.0 (from -r requirements.in (line 1))
pulpcore<3.2,>=3.1 (from pulp-rpm==3.1.0->-r requirements.in (line 2))
Added by dkliban@redhat.com about 4 years ago
Added by dkliban@redhat.com about 4 years ago
Revision 84c3ea5c | View on GitHub
Adds a pulpcore/plugin version compatibility check
This change also reorders the running of pre-requisite roles and the creation of the virtualenv. The version compatibility check is performed before any other packages are installed into the virtualenv.
A requirement.in file is generated using the pulp_version and pulp_install_plugins settings passed in from the playbook. pip-compile inspects all the packages in the requirements.in and returns any conflicts between version requirements.
Some plugins such as pulp_rpm have to have build dependencies installed before the setup.py can be fully interpreted by pip-compile. An error raised in such a case is ignored because it occurs after the versions of direct dependencies of the plugin have been evaluated.
Updated by dkliban@redhat.com about 4 years ago
- Description updated (diff)
- Status changed from NEW to POST
- Assignee set to dkliban@redhat.com
Updated by dkliban@redhat.com about 4 years ago
- Status changed from POST to MODIFIED
Applied in changeset ansible-pulp|84c3ea5cff1b63636bdb5144638793f7132f2aa3.
Updated by dkliban@redhat.com about 4 years ago
Applied in changeset ansible-pulp3|84c3ea5cff1b63636bdb5144638793f7132f2aa3.
Updated by mdepaulo@redhat.com about 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Updated by bmbouter almost 4 years ago
- Category set to Installer - Moved to GitHub issues
- Tags deleted (
Pulp 3 installer)
.
Adds a pulpcore/plugin version compatibility check
This change also reorders the running of pre-requisite roles and the creation of the virtualenv. The version compatibility check is performed before any other packages are installed into the virtualenv.
A requirement.in file is generated using the pulp_version and pulp_install_plugins settings passed in from the playbook. pip-compile inspects all the packages in the requirements.in and returns any conflicts between version requirements.
Some plugins such as pulp_rpm have to have build dependencies installed before the setup.py can be fully interpreted by pip-compile. An error raised in such a case is ignored because it occurs after the versions of direct dependencies of the plugin have been evaluated.
fixes: #6189 https://pulp.plan.io/issues/6189