Project

Profile

Help

Story #140

As a user, I can sync Python packages from another Pulp server

Added by rbarlow over 5 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.13.0
Target Release - Python:
2.0.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 8
Quarter:

Description

The PyPI repository publishes package metadata at /pypi/<package_name>/json[0]. Pulp does not yet implement this API. I am not aware of all of the uses for this API, but one use case is the ability to know which versions of a package are available. Pulp's importer uses this API to perform repository synchronizations. If we wish to enable users to synchronize Python repositories from other Pulp servers, we will need to implement this API endpoint. Deliverables:

  • Modify the distributor so that it generates the /pypi/<package_name>/json API endpoint.
  • Write release notes
  • Write unit tests
  • Ensure that Pulp can synchronize from itself

[0] For example, see https://pypi.python.org/pypi/scipy/json


Related issues

Blocked by Python Support - Story #1883: As a user, I can sync and publish all package typesCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Blocked by Python Support - Issue #2561: pulp_python 2.0 new features are not documentedCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 5cb61ae8 View on GitHub
Added by Austin Macdonald over 4 years ago

Publish metadata to allow pulp to pulp sync

Publishes a json metadata file for each python project that contains metadata for each python package that belongs to it. This is a subset of the api that PyPI implements, and allows a Pulp to sync from another Pulp in exactly the same way that it would sync from PyPI itself.

closes #140

Revision 5cfe2f39 View on GitHub
Added by Austin Macdonald almost 4 years ago

Publish metadata to allow pulp to pulp sync

Publishes a json metadata file for each python project that contains metadata for each python package that belongs to it. This is a subset of the api that PyPI implements, and allows a Pulp to sync from another Pulp in exactly the same way that it would sync from PyPI itself.

closes #140

History

#1 Updated by amacdona@redhat.com over 4 years ago

Pypi uses a fundamentally different model from us, and it is apparent from the JSON metadata.

{'info:
    {"key": "value}
 'releases: [
    {"1.0": [{"data": "for"}, {"each": "file"}]
}

This means that the information stored in "info" for a distribution (like scipy) is meant to apply to each of the releases and each of their files. Because we store it in a flat structure, this means that we each file would have a duplicate of the same information. It also means that if one of these fields changes from release to release, the newest info overwrites the old, even for older releases.

In order to make a pulp to pulp sync work, we will have to be able to publish JSON that builds the whole data structure with the same hierarchy that is returned from pip, favoring the newest release where there are conflicts.

#2 Updated by amacdona@redhat.com over 4 years ago

  • Blocked by Story #1883: As a user, I can sync and publish all package types added

#3 Updated by amacdona@redhat.com over 4 years ago

  • Sprint Candidate changed from No to Yes

#4 Updated by mhrivnak over 4 years ago

  • Sprint/Milestone set to 21
  • Groomed changed from No to Yes

#5 Updated by amacdona@redhat.com over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

#6 Updated by amacdona@redhat.com over 4 years ago

  • Status changed from ASSIGNED to POST

#7 Updated by Anonymous over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by semyers over 4 years ago

  • Status changed from MODIFIED to POST

#9 Updated by mhrivnak about 4 years ago

  • Sprint/Milestone changed from 21 to 22

#10 Updated by bmbouter about 4 years ago

  • Sprint/Milestone changed from 22 to 26

#11 Updated by Anonymous almost 4 years ago

  • Status changed from POST to MODIFIED

#12 Updated by semyers almost 4 years ago

  • Platform Release set to 2.12.0

#13 Updated by semyers over 3 years ago

  • Status changed from MODIFIED to 5

#15 Updated by Ichimonji10 over 3 years ago

  • Status changed from 5 to ASSIGNED

Syncing between repositories simply doesn't work. I'll add a link to an automated test later today.

#17 Updated by mhrivnak over 3 years ago

Can you provide any detail on what happens when it fails?

#18 Updated by Ichimonji10 over 3 years ago

  • Status changed from ASSIGNED to 6

Can you provide any detail on what happens when it fails?

Time to pull a 180. Pulp-to-Pulp sync works. I've made some changes to the automated test, and can verify that it passes. The test was failing due to my ignorance of how Pulp-to-Pulp syncing is supposed to work. All that said, I'd like to suggest several improvements. If I'm confused enough by the feature to think "this is broken," then our users may have the same experience.

First, please document how to use a published Python repository. I read several documents in full before writing any tests, including getting started. Despite that, I was on IRC yesterday asking questions like this:

How does one fetch content units from a published Python repo? (Answer: From /pulp/python/web/<repo-id>/simple/.) (Actual answer: From /pulp/python/web/<repo-id>/.)
Is there a programmatic way to get that info? (Answer: No. Hard-code that path into your app.)
If I want a Python repo with id "bar" to sync from a published Python repo with id "foo", then I should set "bar"'s feed to /pulp/python/web/foo/simple/, right? (Answer: Yes.) (Actual answer: No.)

Second, consider improving either Pulp's data validation or error reporting. At one point, I was submitting the following importer configuration to Pulp:

{
  'feed': '/pulp/python/web/<repo-id>/',
  'package_names': 'shelf-reader',
}

Note the lack of a http://pulp.example.com at the beginning of the feed string. Pulp accepted this string, and when I asked Pulp to sync, the only information returned is that the sync failed. Nectar's traceback was completely swallowed.

#19 Updated by semyers over 3 years ago

Ichimonji10 wrote:

Can you provide any detail on what happens when it fails?

Time to pull a 180. Pulp-to-Pulp sync works. I've made some changes to the automated test, and can verify that it passes. The test was failing due to my ignorance of how Pulp-to-Pulp syncing is supposed to work. All that said, I'd like to suggest several improvements. If I'm confused enough by the feature to think "this is broken," then our users may have the same experience.

This was largely mhrivnak's point when I put this issue forth as a blocker for 2.12.0, and is good justification for delaying the release of python 2.0 (in my opinion).

#20 Updated by semyers over 3 years ago

  • Status changed from 6 to MODIFIED
  • Platform Release deleted (2.12.0)
  • Target Release - Python set to 2.0.0

This issue has been removed from the 2.12.0 release, and returned to its MODIFIED state for inclusion in a future release of Pulp.

#21 Updated by semyers over 3 years ago

  • Blocked by Issue #2561: pulp_python 2.0 new features are not documented added

#22 Updated by semyers over 3 years ago

  • Status changed from MODIFIED to POST

I'm moving this back to POST to flag this bug as untestable in pulp-smash. This status is also appropriate because while the changes are merged to the 2.0-dev branch of pulp_python, that entire branch was deemed unreleasable for pulp 2.12, and the situation has not improved since then.

#23 Updated by semyers over 3 years ago

  • Status changed from POST to MODIFIED

2.0-dev is now being included in Platform 2.13 builds again, returning this to a testable state. Note that any smash tests related to this issue should not be run prior to 2.13.0; this is tracked in pulp-smash issue https://github.com/PulpQE/pulp-smash/issues/588.

#24 Updated by pcreech over 3 years ago

  • Platform Release set to 2.13.0

#25 Updated by pcreech over 3 years ago

  • Status changed from MODIFIED to 5

#26 Updated by pcreech over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#27 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 8

#28 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (26)

#29 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF