Story #2887
closed
As a User, I can upload a python package to a repository from twine
0%
Description
In pulp_python for Pulp 2, uploads come in over the standard "uploads" REST API. Because of this, the python plugin uses the twine utility to inspect the uploaded Python Distribution to parse the appropriate metadata. Rather than continue to depend on twine or carry identical code to inspect packages, it makes sense to receive those uploads directly from twine.[0]
This story will be complete when a user can upload a package to pulp_python using `twine upload`. This story does not include the use of the standard Pulp 3 upload API.
[0]: https://packaging.python.org/tutorials/distributing-packages/#upload-your-distributions
This story will be complete when a user can use twine extract metadata from a Python package (all types) and create a ContentUnit via the REST API.
Files
Related issues
Updated by amacdona@redhat.com almost 7 years ago
- Blocked by Story #2884: As a user I can sync from PyPI added
Updated by amacdona@redhat.com over 6 years ago
- Subject changed from As a User, I can upload a python package to a repository to As a User, I can upload a python package to a repository from twine
- Description updated (diff)
Updated by dalley over 6 years ago
(All of this info applies to legacy PyPI, not Warehouse)
This is the code which twine uses to do uploads [0]
The URL is just the unmodified PyPI endpoint url, e.g. https://upload.pypi.org/legacy/, https://test.pypi.org/legacy/. Nothing special going on there.
At the point where it is shoved into the multipart encoder [1], this is what the raw set of tuples looks like
(sdist)
[('name', 'pulp-python'), ('version', '3.0.0a1.dev0'), ('filetype', 'sdist'), ('pyversion', None), ('metadata_version', '1.0'), ('summary', 'pulp-python plugin for the Pulp Project'), ('home_page', 'http://www.pulpproject.org'), ('author', 'Pulp Project Developers'), ('author_email', 'pulp-list@redhat.com'), ('maintainer', None), ('maintainer_email', None), ('license', 'GPLv2+'), ('description', None), ('keywords', None), ('platform', 'UNKNOWN'), ('download_url', None), ('comment', None), ('md5_digest', '1488f866e0a86455e3a90ed8152167bb'), ('sha256_digest', '41b0233eb20db4324c0285720f3c206b78df3219d6e636c09e85cfa22751d857'), ('blake2_256_digest', '9445dbe404dd962f9af7bd915b7e8bd92bd602fa032ec42c5ac33a9c5f6d4cc2'), ('requires_python', None), (':action', 'file_upload'), ('protcol_version', '1'), ('content', ('pulp-python-3.0.0a1.dev0.tar.gz', <_io.BufferedReader name='dist/pulp-python-3.0.0a1.dev0.tar.gz'>, 'application/octet-stream'))]
(bdist_wheel)
[('name', 'pulp-python'), ('version', '3.0.0a1.dev0'), ('filetype', 'bdist_wheel'), ('pyversion', 'py3'), ('metadata_version', '2.0'), ('summary', 'pulp-python plugin for the Pulp Project'), ('home_page', 'http://www.pulpproject.org'), ('author', 'Pulp Project Developers'), ('author_email', 'pulp-list@redhat.com'), ('maintainer', None), ('maintainer_email', None), ('license', 'GPLv2+'), ('description', 'UNKNOWN\n\n\n'), ('keywords', None), ('platform', 'UNKNOWN'), ('download_url', None), ('comment', None), ('md5_digest', 'e7589d3c306f46003bcbb90107b16421'), ('sha256_digest', '880c97d59ec6a94a5e35ef49cfeb3be7161d503dc7a7a283894b61bb4b5aacc5'), ('blake2_256_digest', '8848709ab5c62da72825b76477483073e2179e1681c41c8fa9545b18bf7ef93d'), ('requires_dist', 'pulpcore-plugin'), ('requires_python', None), (':action', 'file_upload'), ('protcol_version', '1'), ('content', ('pulp_python-3.0.0a1.dev0-py3-none-any.whl', <_io.BufferedReader name='dist/pulp_python-3.0.0a1.dev0-py3-none-any.whl'>, 'application/octet-stream'))]
This is the raw bytes content of the encoder itself are attached as files. One is an upload with an sdist content and one is an upload with bdist_wheel as the content. The request is slightly different between the two. This is the closest I could get to the actual HTTP request, considering PyPI is HTTPS-only which prevents actual interception of the request (i.e. Wireshark). It's good enough for our purposes.
The POST request is created by the requests library with these options [2]. The Content-Type header is
multipart/form-data; boundary=b9762ceb450a48d98e81d94d363782c0
Where "boundary" is a uuid generated here [3]
Looks like there is also a defect in this code... [4] "protocol" is misspelled, and it is spelled correctly in the register code above. I submitted a PR to twine for this.
[0] https://github.com/pypa/twine/blob/a90be8f57f02630c25cbb7e9f3d9a89578122f6c/twine/repository.py#L120-L172
[1] https://github.com/pypa/twine/blob/a90be8f57f02630c25cbb7e9f3d9a89578122f6c/twine/repository.py#L133
[2] https://github.com/pypa/twine/blob/a90be8f57f02630c25cbb7e9f3d9a89578122f6c/twine/repository.py#L145
[3] https://github.com/requests/toolbelt/blob/master/requests_toolbelt/multipart/encoder.py#L83
[4] https://github.com/pypa/twine/blob/master/twine/repository.py#L127
Updated by dalley over 6 years ago
- File source.txt source.txt added
- File wheel.txt wheel.txt added
Updated by amacdona@redhat.com over 6 years ago
Since the twine uploads go to an "unmodified PyPI endpoint url" this one depends on making a python repository as part of a live API. Once the live API story is written, this one needs to be related to it.
Updated by amacdona@redhat.com almost 6 years ago
- Tags Pulp 3 added
+1, this is a cool issue.
We can't do this just yet, because currently publications are served by an associated distribution, which is a pulpcore feature. To allow this, the python plugin will need to field requests that are namespaced by distribution.base_path. I'm not sure if this can be done while leaving the existing code, or if we would need to alter the workflows for pulp_python to avoid the pulpcore distributions. If this is the way that we have to go, we would need to implement a live API endpoint for upload and somehow also handle the workload of distributions. I imagine this could be done in two ways, 1) we could serve static files (publications) or 2) we could implement a live API for GET requests for the JSON metadata like PyPI does itself.
Updated by amacdona@redhat.com about 5 years ago
- Sprint/Milestone deleted (
3.0 GA)
Updated by gerrod about 3 years ago
- Status changed from NEW to CLOSED - DUPLICATE
Migrated to GitHub Issues: https://github.com/pulp/pulp_python/issues/342
.