Story #140
closed
As a user, I can sync Python packages from another Pulp server
100%
Description
The PyPI repository publishes package metadata at /pypi/<package_name>/json[0]. Pulp does not yet implement this API. I am not aware of all of the uses for this API, but one use case is the ability to know which versions of a package are available. Pulp's importer uses this API to perform repository synchronizations. If we wish to enable users to synchronize Python repositories from other Pulp servers, we will need to implement this API endpoint. Deliverables:
- Modify the distributor so that it generates the /pypi/<package_name>/json API endpoint.
- Write release notes
- Write unit tests
- Ensure that Pulp can synchronize from itself
[0] For example, see https://pypi.python.org/pypi/scipy/json
Related issues
Updated by amacdona@redhat.com over 8 years ago
Pypi uses a fundamentally different model from us, and it is apparent from the JSON metadata.
{'info:
{"key": "value}
'releases: [
{"1.0": [{"data": "for"}, {"each": "file"}]
}
This means that the information stored in "info" for a distribution (like scipy) is meant to apply to each of the releases and each of their files. Because we store it in a flat structure, this means that we each file would have a duplicate of the same information. It also means that if one of these fields changes from release to release, the newest info overwrites the old, even for older releases.
In order to make a pulp to pulp sync work, we will have to be able to publish JSON that builds the whole data structure with the same hierarchy that is returned from pip, favoring the newest release where there are conflicts.
Updated by amacdona@redhat.com over 8 years ago
- Blocked by Story #1883: As a user, I can sync and publish all package types added
Updated by amacdona@redhat.com over 8 years ago
- Sprint Candidate changed from No to Yes
Updated by mhrivnak over 8 years ago
- Sprint/Milestone set to 21
- Groomed changed from No to Yes
Updated by amacdona@redhat.com over 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to amacdona@redhat.com
Added by Austin Macdonald over 8 years ago
Updated by amacdona@redhat.com over 8 years ago
- Status changed from ASSIGNED to POST
Updated by Anonymous over 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset 5cb61ae8391808d70d41d7b5923bc5f913f3ca65.
Added by Austin Macdonald about 8 years ago
Revision 5cfe2f39 | View on GitHub
Publish metadata to allow pulp to pulp sync
Publishes a json metadata file for each python project that contains metadata for each python package that belongs to it. This is a subset of the api that PyPI implements, and allows a Pulp to sync from another Pulp in exactly the same way that it would sync from PyPI itself.
closes #140
Updated by Anonymous about 8 years ago
- Status changed from POST to MODIFIED
Applied in changeset 5cfe2f39b38f1aeb218623687f64d14970b02f77.
Updated by Ichimonji10 almost 8 years ago
- Status changed from 5 to ASSIGNED
Syncing between repositories simply doesn't work. I'll add a link to an automated test later today.
Updated by Ichimonji10 almost 8 years ago
Updated by mhrivnak almost 8 years ago
Can you provide any detail on what happens when it fails?
Updated by Ichimonji10 almost 8 years ago
- Status changed from ASSIGNED to 6
Can you provide any detail on what happens when it fails?
Time to pull a 180. Pulp-to-Pulp sync works. I've made some changes to the automated test, and can verify that it passes. The test was failing due to my ignorance of how Pulp-to-Pulp syncing is supposed to work. All that said, I'd like to suggest several improvements. If I'm confused enough by the feature to think "this is broken," then our users may have the same experience.
First, please document how to use a published Python repository. I read several documents in full before writing any tests, including getting started. Despite that, I was on IRC yesterday asking questions like this:
How does one fetch content units from a published Python repo? (Answer: From
/pulp/python/web/<repo-id>/simple/.) (Actual answer: From/pulp/python/web/<repo-id>/.)
Is there a programmatic way to get that info? (Answer: No. Hard-code that path into your app.)
If I want a Python repo with id "bar" to sync from a published Python repo with id "foo", then I should set "bar"'s feed to/pulp/python/web/foo/simple/, right? (Answer: Yes.) (Actual answer: No.)
Second, consider improving either Pulp's data validation or error reporting. At one point, I was submitting the following importer configuration to Pulp:
{
'feed': '/pulp/python/web/<repo-id>/',
'package_names': 'shelf-reader',
}
Note the lack of a http://pulp.example.com at the beginning of the feed string. Pulp accepted this string, and when I asked Pulp to sync, the only information returned is that the sync failed. Nectar's traceback was completely swallowed.
Updated by semyers almost 8 years ago
Ichimonji10 wrote:
Can you provide any detail on what happens when it fails?
Time to pull a 180. Pulp-to-Pulp sync works. I've made some changes to the automated test, and can verify that it passes. The test was failing due to my ignorance of how Pulp-to-Pulp syncing is supposed to work. All that said, I'd like to suggest several improvements. If I'm confused enough by the feature to think "this is broken," then our users may have the same experience.
This was largely mhrivnak's point when I put this issue forth as a blocker for 2.12.0, and is good justification for delaying the release of python 2.0 (in my opinion).
Updated by semyers almost 8 years ago
- Status changed from 6 to MODIFIED
- Platform Release deleted (
2.12.0) - Target Release - Python set to 2.0.0
This issue has been removed from the 2.12.0 release, and returned to its MODIFIED state for inclusion in a future release of Pulp.
Updated by semyers almost 8 years ago
- Blocked by Issue #2561: pulp_python 2.0 new features are not documented added
Updated by semyers over 7 years ago
- Status changed from MODIFIED to POST
I'm moving this back to POST to flag this bug as untestable in pulp-smash. This status is also appropriate because while the changes are merged to the 2.0-dev branch of pulp_python, that entire branch was deemed unreleasable for pulp 2.12, and the situation has not improved since then.
Updated by semyers over 7 years ago
- Status changed from POST to MODIFIED
2.0-dev is now being included in Platform 2.13 builds again, returning this to a testable state. Note that any smash tests related to this issue should not be run prior to 2.13.0; this is tracked in pulp-smash issue https://github.com/PulpQE/pulp-smash/issues/588.
Updated by pcreech over 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
Publish metadata to allow pulp to pulp sync
Publishes a json metadata file for each python project that contains metadata for each python package that belongs to it. This is a subset of the api that PyPI implements, and allows a Pulp to sync from another Pulp in exactly the same way that it would sync from PyPI itself.
closes #140