Project

Profile

Help

Issue #3578

pulp-python wrong path to pypi

Added by Poil over 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Python:
Platform Release:
2.16.4
Target Release - Python:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix, Pulp 2
Sprint:
Sprint 36
Quarter:

Description

When creating a repo like this

pulp-admin python repo create --repo-id pypitest --feed https://pypi.org/ --package-names CouchDB --proxy-host http://22.0.53.8 --proxy-port 3128
And syncing
pulp-admin python repo sync run --repo-id pypitest

pulp try to download the metadata from https://pypi.org/pypi/CouchDB/json/
but this is 404, it should be https://pypi.org/pypi/CouchDB/json (without the / at the end)

After removing the trailing slash in /usr/lib/python2.7/site-packages/pulp_python/plugins/importers/sync.py, it's working fine.

pulp-python-plugins-2.0.2-1.el7.noarch

Best regards,

Associated revisions

Revision 7af05347 View on GitHub
Added by werwty over 3 years ago

Remove trailing slash from json api.

This is caused by the warehouse migrations- the json api works differently in warehousem and a trailing slash would cause it to be redirected to a different location.

fixes #3578 https://pulp.plan.io/issues/3578

Revision 2a658f3f View on GitHub
Added by werwty over 3 years ago

Remove trailing slash from json api.

This is caused by the warehouse migrations- the json api works differently in warehousem and a trailing slash would cause it to be redirected to a different location.

fixes #3578 https://pulp.plan.io/issues/3578

(cherry picked from commit 7af05347bbd1723b65d9f5507a02529dd821a15a)

History

#1 Updated by bizhang over 3 years ago

  • Status changed from NEW to POST
  • Assignee set to bizhang

Thanks for filing this Poil! This is caused by the recent pypi migration to warehouse- the new json api thinks that a trailing slash means that it should redirect.

I've opened a PR for this issue: https://github.com/pulp/pulp_python/pull/144
and we'll get the fix out in the new version.

For future reference we really appreciate PRs for easy fixes like these and are willing to bribe potential contributors with pulp stickers :)

#2 Updated by Poil over 3 years ago

It seems there are another changes, in the json file

avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: pulp_python.plugins.importers.sync:INFO: Processing metadata retrieved from https://pypi.org/pypi/CouchDB/json.
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) 'path'
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) Traceback (most recent call last):
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     event_listener_callback(*args, **kwargs)
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/sync.py", line 65, in download_succeeded
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     units = models.Package.objects.from_metadata(destination.read())
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/querysets.py", line 55, in from_metadata
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     units.append(self._document.from_json(package, version, metadata['info']))
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/models.py", line 115, in from_json
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     package_attrs['path'] = package_data['path']
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) KeyError: 'path'

https://github.com/pulp/pulp_python/pull/145/files

#3 Updated by Poil over 3 years ago

I've got another bug, I don't think it comes from my fix :

pip install -i pulp.repo trusted-host pulp.repo CouchDB

Is trying to download "/pulp/python/web/ghostdev/simple/couchdb/" but pulp mirrored it to http://pulp.repo/pulp/python/web/ghostdev/simple/CouchDB/

From PEP 503 it seems that it must be lowercases : https://www.python.org/dev/peps/pep-0503/

From PEP 426 it seems that it must be lowercases : https://www.python.org/dev/peps/pep-0426/#source-url

edit By activating mod_speling I have a quick fix

cat /etc/httpd/conf.d/pulp_python.conf 
#
# Apache configuration file for Pulp's Python support
#

# -- HTTPS Repositories ---------

Alias /pulp/python /var/www/pub/python/

<Directory /var/www/pub/python>
    Options FollowSymLinks Indexes
    DirectoryIndex index.html index.json
    CheckCaseOnly on
    CheckSpelling on
</Directory>

#4 Updated by dalley over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 36

#5 Updated by werwty over 3 years ago

  • Status changed from POST to MODIFIED

#6 Updated by Poil over 3 years ago

Hi,

Should I open another issue about PEP 503 and my quick fix via mod_speling ?

#7 Updated by bizhang over 3 years ago

Hey Poil,
I think the right solution would be for us to normalize the names during publication: I've filed a issue for that https://pulp.plan.io/issues/3632

#8 Updated by daviddavis over 3 years ago

  • Platform Release set to 2.16.4

#9 Updated by werwty over 3 years ago

#10 Updated by daviddavis over 3 years ago

There's another PR associated with this issue:

https://github.com/pulp/pulp_python/pull/145

Which has the following commits:

79b46411a049e63fc24ba97a60b51fe0574b268c
https://github.com/pulp/pulp_python/commit/79b46411a049e63fc24ba97a60b51fe0574b268c

2a2c0c772f1889c1f0a791c082dc969d7119cac4
https://github.com/pulp/pulp_python/commit/2a2c0c772f1889c1f0a791c082dc969d7119cac4

Looks like the commit messages don't properly contain the issue # which is why they weren't associated with this issue.

#11 Updated by daviddavis over 3 years ago

  • Status changed from MODIFIED to 5

#12 Updated by daviddavis over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#13 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF