Project

Profile

Help

Issue #3578

closed

pulp-python wrong path to pypi

Added by Poil about 6 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Python:
Platform Release:
2.16.4
Target Release - Python:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix, Pulp 2
Sprint:
Sprint 36
Quarter:

Description

When creating a repo like this

pulp-admin python repo create --repo-id pypitest --feed https://pypi.org/ --package-names CouchDB --proxy-host http://22.0.53.8 --proxy-port 3128
And syncing
pulp-admin python repo sync run --repo-id pypitest

pulp try to download the metadata from https://pypi.org/pypi/CouchDB/json/
but this is 404, it should be https://pypi.org/pypi/CouchDB/json (without the / at the end)

After removing the trailing slash in /usr/lib/python2.7/site-packages/pulp_python/plugins/importers/sync.py, it's working fine.

pulp-python-plugins-2.0.2-1.el7.noarch

Best regards,

Actions #1

Updated by bizhang about 6 years ago

  • Status changed from NEW to POST
  • Assignee set to bizhang

Thanks for filing this Poil! This is caused by the recent pypi migration to warehouse- the new json api thinks that a trailing slash means that it should redirect.

I've opened a PR for this issue: https://github.com/pulp/pulp_python/pull/144
and we'll get the fix out in the new version.

For future reference we really appreciate PRs for easy fixes like these and are willing to bribe potential contributors with pulp stickers :)

Added by werwty about 6 years ago

Revision 7af05347 | View on GitHub

Remove trailing slash from json api.

This is caused by the warehouse migrations- the json api works differently in warehousem and a trailing slash would cause it to be redirected to a different location.

fixes #3578 https://pulp.plan.io/issues/3578

Actions #2

Updated by Poil about 6 years ago

It seems there are another changes, in the json file

avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: pulp_python.plugins.importers.sync:INFO: Processing metadata retrieved from https://pypi.org/pypi/CouchDB/json.
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) 'path'
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) Traceback (most recent call last):
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     event_listener_callback(*args, **kwargs)
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/sync.py", line 65, in download_succeeded
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     units = models.Package.objects.from_metadata(destination.read())
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/querysets.py", line 55, in from_metadata
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     units.append(self._document.from_json(package, version, metadata['info']))
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/models.py", line 115, in from_json
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     package_attrs['path'] = package_data['path']
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) KeyError: 'path'

https://github.com/pulp/pulp_python/pull/145/files

Actions #3

Updated by Poil about 6 years ago

I've got another bug, I don't think it comes from my fix :

pip install -i pulp.repo trusted-host pulp.repo CouchDB

Is trying to download "/pulp/python/web/ghostdev/simple/couchdb/" but pulp mirrored it to http://pulp.repo/pulp/python/web/ghostdev/simple/CouchDB/

From PEP 503 it seems that it must be lowercases : https://www.python.org/dev/peps/pep-0503/

From PEP 426 it seems that it must be lowercases : https://www.python.org/dev/peps/pep-0426/#source-url

edit By activating mod_speling I have a quick fix

cat /etc/httpd/conf.d/pulp_python.conf 
#
# Apache configuration file for Pulp's Python support
#

# -- HTTPS Repositories ---------

Alias /pulp/python /var/www/pub/python/

<Directory /var/www/pub/python>
    Options FollowSymLinks Indexes
    DirectoryIndex index.html index.json
    CheckCaseOnly on
    CheckSpelling on
</Directory>
Actions #4

Updated by dalley about 6 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 36
Actions #5

Updated by werwty almost 6 years ago

  • Status changed from POST to MODIFIED
Actions #6

Updated by Poil almost 6 years ago

Hi,

Should I open another issue about PEP 503 and my quick fix via mod_speling ?

Actions #7

Updated by bizhang almost 6 years ago

Hey Poil,
I think the right solution would be for us to normalize the names during publication: I've filed a issue for that https://pulp.plan.io/issues/3632

Actions #8

Updated by daviddavis almost 6 years ago

  • Platform Release set to 2.16.4

Added by werwty over 5 years ago

Revision 2a658f3f | View on GitHub

Remove trailing slash from json api.

This is caused by the warehouse migrations- the json api works differently in warehousem and a trailing slash would cause it to be redirected to a different location.

fixes #3578 https://pulp.plan.io/issues/3578

(cherry picked from commit 7af05347bbd1723b65d9f5507a02529dd821a15a)

Actions #9

Updated by werwty over 5 years ago

Actions #10

Updated by daviddavis over 5 years ago

There's another PR associated with this issue:

https://github.com/pulp/pulp_python/pull/145

Which has the following commits:

79b46411a049e63fc24ba97a60b51fe0574b268c
https://github.com/pulp/pulp_python/commit/79b46411a049e63fc24ba97a60b51fe0574b268c

2a2c0c772f1889c1f0a791c082dc969d7119cac4
https://github.com/pulp/pulp_python/commit/2a2c0c772f1889c1f0a791c082dc969d7119cac4

Looks like the commit messages don't properly contain the issue # which is why they weren't associated with this issue.

Actions #11

Updated by daviddavis over 5 years ago

  • Status changed from MODIFIED to 5
Actions #12

Updated by daviddavis over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #13

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF