Project

Profile

Help

Story #2887

As a User, I can upload a python package to a repository from twine

Added by amacdona@redhat.com over 2 years ago. Updated 7 months ago.

Status:
NEW
Priority:
Low
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Target Release - Python:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

In pulp_python for Pulp 2, uploads come in over the standard "uploads" REST API. Because of this, the python plugin uses the twine utility to inspect the uploaded Python Distribution to parse the appropriate metadata. Rather than continue to depend on twine or carry identical code to inspect packages, it makes sense to receive those uploads directly from twine.[0]

This story will be complete when a user can upload a package to pulp_python using `twine upload`. This story does not include the use of the standard Pulp 3 upload API.

[0]: https://packaging.python.org/tutorials/distributing-packages/#upload-your-distributions

This story will be complete when a user can use twine extract metadata from a Python package (all types) and create a ContentUnit via the REST API.

source.txt (9 KB) dalley, 12/18/2017 12:04 AM source.txt
wheel.txt (10.5 KB) dalley, 12/18/2017 12:04 AM wheel.txt

Related issues

Blocked by Python Support - Story #2884: As a user I can sync from PyPI MODIFIED Actions

History

#1 Updated by amacdona@redhat.com over 2 years ago

  • Blocked by Story #2884: As a user I can sync from PyPI added

#2 Updated by amacdona@redhat.com about 2 years ago

  • Subject changed from As a User, I can upload a python package to a repository to As a User, I can upload a python package to a repository from twine
  • Description updated (diff)

#3 Updated by dalley almost 2 years ago

(All of this info applies to legacy PyPI, not Warehouse)

This is the code which twine uses to do uploads [0]

The URL is just the unmodified PyPI endpoint url, e.g. https://upload.pypi.org/legacy/, https://test.pypi.org/legacy/. Nothing special going on there.

At the point where it is shoved into the multipart encoder [1], this is what the raw set of tuples looks like

(sdist)

[('name', 'pulp-python'), ('version', '3.0.0a1.dev0'), ('filetype', 'sdist'), ('pyversion', None), ('metadata_version', '1.0'), ('summary', 'pulp-python plugin for the Pulp Project'), ('home_page', 'http://www.pulpproject.org'), ('author', 'Pulp Project Developers'), ('author_email', 'pulp-list@redhat.com'), ('maintainer', None), ('maintainer_email', None), ('license', 'GPLv2+'), ('description', None), ('keywords', None), ('platform', 'UNKNOWN'), ('download_url', None), ('comment', None), ('md5_digest', '1488f866e0a86455e3a90ed8152167bb'), ('sha256_digest', '41b0233eb20db4324c0285720f3c206b78df3219d6e636c09e85cfa22751d857'), ('blake2_256_digest', '9445dbe404dd962f9af7bd915b7e8bd92bd602fa032ec42c5ac33a9c5f6d4cc2'), ('requires_python', None), (':action', 'file_upload'), ('protcol_version', '1'), ('content', ('pulp-python-3.0.0a1.dev0.tar.gz', <_io.BufferedReader name='dist/pulp-python-3.0.0a1.dev0.tar.gz'>, 'application/octet-stream'))]

(bdist_wheel)

[('name', 'pulp-python'), ('version', '3.0.0a1.dev0'), ('filetype', 'bdist_wheel'), ('pyversion', 'py3'), ('metadata_version', '2.0'), ('summary', 'pulp-python plugin for the Pulp Project'), ('home_page', 'http://www.pulpproject.org'), ('author', 'Pulp Project Developers'), ('author_email', 'pulp-list@redhat.com'), ('maintainer', None), ('maintainer_email', None), ('license', 'GPLv2+'), ('description', 'UNKNOWN\n\n\n'), ('keywords', None), ('platform', 'UNKNOWN'), ('download_url', None), ('comment', None), ('md5_digest', 'e7589d3c306f46003bcbb90107b16421'), ('sha256_digest', '880c97d59ec6a94a5e35ef49cfeb3be7161d503dc7a7a283894b61bb4b5aacc5'), ('blake2_256_digest', '8848709ab5c62da72825b76477483073e2179e1681c41c8fa9545b18bf7ef93d'), ('requires_dist', 'pulpcore-plugin'), ('requires_python', None), (':action', 'file_upload'), ('protcol_version', '1'), ('content', ('pulp_python-3.0.0a1.dev0-py3-none-any.whl', <_io.BufferedReader name='dist/pulp_python-3.0.0a1.dev0-py3-none-any.whl'>, 'application/octet-stream'))]

This is the raw bytes content of the encoder itself are attached as files. One is an upload with an sdist content and one is an upload with bdist_wheel as the content. The request is slightly different between the two. This is the closest I could get to the actual HTTP request, considering PyPI is HTTPS-only which prevents actual interception of the request (i.e. Wireshark). It's good enough for our purposes.

The POST request is created by the requests library with these options [2]. The Content-Type header is

multipart/form-data; boundary=b9762ceb450a48d98e81d94d363782c0

Where "boundary" is a uuid generated here [3]

Looks like there is also a defect in this code... [4] "protocol" is misspelled, and it is spelled correctly in the register code above. I submitted a PR to twine for this.

[0] https://github.com/pypa/twine/blob/a90be8f57f02630c25cbb7e9f3d9a89578122f6c/twine/repository.py#L120-L172
[1] https://github.com/pypa/twine/blob/a90be8f57f02630c25cbb7e9f3d9a89578122f6c/twine/repository.py#L133
[2] https://github.com/pypa/twine/blob/a90be8f57f02630c25cbb7e9f3d9a89578122f6c/twine/repository.py#L145
[3] https://github.com/requests/toolbelt/blob/master/requests_toolbelt/multipart/encoder.py#L83
[4] https://github.com/pypa/twine/blob/master/twine/repository.py#L127

#4 Updated by dalley almost 2 years ago

#5 Updated by amacdona@redhat.com almost 2 years ago

Since the twine uploads go to an "unmodified PyPI endpoint url" this one depends on making a python repository as part of a live API. Once the live API story is written, this one needs to be related to it.

#6 Updated by amacdona@redhat.com over 1 year ago

  • Tags Pulp 3 added

+1, this is a cool issue.

We can't do this just yet, because currently publications are served by an associated distribution, which is a pulpcore feature. To allow this, the python plugin will need to field requests that are namespaced by distribution.base_path. I'm not sure if this can be done while leaving the existing code, or if we would need to alter the workflows for pulp_python to avoid the pulpcore distributions. If this is the way that we have to go, we would need to implement a live API endpoint for upload and somehow also handle the workload of distributions. I imagine this could be done in two ways, 1) we could serve static files (publications) or 2) we could implement a live API for GET requests for the JSON metadata like PyPI does itself.

#7 Updated by bizhang over 1 year ago

  • Sprint/Milestone set to 3.0 GA

#8 Updated by bizhang over 1 year ago

  • Priority changed from Normal to Low

#9 Updated by amacdona@redhat.com 8 months ago

  • Sprint/Milestone deleted (3.0 GA)

#10 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF