Issue #3578: pulp-python wrong path to pypi - Python Support - Pulp

Agile board
Agile charts

Agile boards

Pulp 2 QE Agile Board
Pulp 2 QE Agile Board - Priority View
Pulp 3 QE Agile Board
Pulp 3 RPM Tests Agile Board
Sprint 112 Agile Board

Custom queries

[All Projects] All Test Trackers
2.21.4 - Next Bugfix Release
2.21.4 MODIFIED
2.21.5
3.14.3
Easy Fix
FIPS open
Functional Tests
Galaxy NG
Groomed
Issues for Triage
Katello Integration
Next Docs Day
POST/ASSIGNED - no Sprint - Pulp2
POST/ASSIGNED - no Sprint - Pulp3
Prioritized Bugs
Pulp 3 Docs issues
Pulp2 issues available for next release
SELinux related
Sprint 111
Sprint 112
Sprint Candidates (also groomed)
Un-Triaged Bugs
Verification Required

Actions

Send by e-mail Copy link

Issue #3578

closed

pulp-python wrong path to pypi

Added by Poil over 6 years ago. Updated over 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

bizhang

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version - Python:

Platform Release:

2.16.4

Target Release - Python:

OS:

RHEL 7

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Easy Fix, Pulp 2

Sprint:

Sprint 36

Quarter:

Description

When creating a repo like this

pulp-admin python repo create --repo-id pypitest --feed https://pypi.org/ --package-names CouchDB --proxy-host http://22.0.53.8 --proxy-port 3128
And syncing
pulp-admin python repo sync run --repo-id pypitest

pulp try to download the metadata from https://pypi.org/pypi/CouchDB/json/
but this is 404, it should be https://pypi.org/pypi/CouchDB/json (without the / at the end)

After removing the trailing slash in /usr/lib/python2.7/site-packages/pulp_python/plugins/importers/sync.py, it's working fine.

pulp-python-plugins-2.0.2-1.el7.noarch

Best regards,

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by bizhang over 6 years ago

Status changed from NEW to POST
Assignee set to bizhang

Thanks for filing this Poil! This is caused by the recent pypi migration to warehouse- the new json api thinks that a trailing slash means that it should redirect.

I've opened a PR for this issue: https://github.com/pulp/pulp_python/pull/144
and we'll get the fix out in the new version.

For future reference we really appreciate PRs for easy fixes like these and are willing to bribe potential contributors with pulp stickers :)

Added by werwty over 6 years ago

Revision 7af05347 | View on GitHub

Remove trailing slash from json api.

This is caused by the warehouse migrations- the json api works differently in warehousem and a trailing slash would cause it to be redirected to a different location.

fixes #3578 https://pulp.plan.io/issues/3578

Actions

Copy link

Updated by Poil over 6 years ago

It seems there are another changes, in the json file

avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: pulp_python.plugins.importers.sync:INFO: Processing metadata retrieved from https://pypi.org/pypi/CouchDB/json.
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) 'path'
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) Traceback (most recent call last):
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/base.py", line 145, in _fire_event_to_listener
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     event_listener_callback(*args, **kwargs)
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/sync.py", line 65, in download_succeeded
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     units = models.Package.objects.from_metadata(destination.read())
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/querysets.py", line 55, in from_metadata
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     units.append(self._document.from_json(package, version, metadata['info']))
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/models.py", line 115, in from_json
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376)     package_attrs['path'] = package_data['path']
avril 19 09:25:38 tlspxe-p01.engsec pulp[1598]: nectar.downloaders.base:ERROR: (1598-21376) KeyError: 'path'

https://github.com/pulp/pulp_python/pull/145/files

Actions

Copy link

Updated by Poil over 6 years ago

I've got another bug, I don't think it comes from my fix :

pip install -i pulp.repo trusted-host pulp.repo CouchDB

Is trying to download "/pulp/python/web/ghostdev/simple/couchdb/" but pulp mirrored it to http://pulp.repo/pulp/python/web/ghostdev/simple/CouchDB/

From PEP 503 it seems that it must be lowercases : https://www.python.org/dev/peps/pep-0503/

~~From PEP 426 it seems that it must be lowercases : https://www.python.org/dev/peps/pep-0426/#source-url~~

edit By activating mod_speling I have a quick fix

cat /etc/httpd/conf.d/pulp_python.conf 
#
# Apache configuration file for Pulp's Python support
#

# -- HTTPS Repositories ---------

Alias /pulp/python /var/www/pub/python/

<Directory /var/www/pub/python>
    Options FollowSymLinks Indexes
    DirectoryIndex index.html index.json
    CheckCaseOnly on
    CheckSpelling on
</Directory>

Actions

Copy link

Updated by dalley over 6 years ago

Triaged changed from No to Yes
Sprint set to Sprint 36

Actions

Copy link

Updated by werwty over 6 years ago

Status changed from POST to MODIFIED

Applied in changeset 7af05347bbd1723b65d9f5507a02529dd821a15a.

Actions

Copy link

Updated by Poil over 6 years ago

Hi,

Should I open another issue about PEP 503 and my quick fix via mod_speling ?

Actions

Copy link

Updated by bizhang over 6 years ago

Hey Poil,
I think the right solution would be for us to normalize the names during publication: I've filed a issue for that https://pulp.plan.io/issues/3632

Actions

Copy link

Updated by daviddavis over 6 years ago

Platform Release set to 2.16.4

Added by werwty over 6 years ago

Revision 2a658f3f | View on GitHub

Remove trailing slash from json api.

This is caused by the warehouse migrations- the json api works differently in warehousem and a trailing slash would cause it to be redirected to a different location.

fixes #3578 https://pulp.plan.io/issues/3578

(cherry picked from commit 7af05347bbd1723b65d9f5507a02529dd821a15a)

Actions

Copy link