Python Support

ashbyj@imsweb.com wrote:

Warehouse looks great. How stable is it? I see they have a list_packages() query that hopefully can grab a list of packages in one query, but pulp_python may need some refactoring to download multiple packages in one shot instead of looping and downloading each package as a separate request. The mirroring support in Warehouse looks helpful as well.

Yeah it really does look nice. I'm not sure how stable it is yet, other than that PyPI is not using it yet and they are still developing it. I have considered starting a branch to test the Python plugin against the current deployment, but I'd like to know the answer to your question before going too far with that ☺

As for the refactor - I think that's a good idea!

From the original post: Basically, I'd like to be able to set up an internal pypi mirror. From our list discussion:

From: pulp-list On Behalf Of Randy Barlow
Sent: Wednesday, May 13, 2015 9:00 AM
To: pulp-list
Subject: Re: [Pulp-list] Sync all packages from PyPi with pulp_python plugin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 05/13/2015 08:21 AM, Ashby, Jason (IMS) wrote:
> I’m looking to set up a pypi mirror with pulp.  I’m currently
> using Bandersnatch for this, but it’d be nice to drop it and use
> pulp instead.  Per the docs*, I see that you can sync specific
> packages from pypi, e.g.
> 
> pulp-admin python repo create --repo-id pypi --feed 
> https://pypi.python.org/ --package-names numpy,scipy
> 
> but I can’t seem to sync ALL packages.  I tried leaving off the 
> --package-names option, but a sync downloads 0 packages.   Should
> I submit an issue/feature request at 
> https://pulp.plan.io/projects/pulp_python/issues?

Hi Jason!

The problem is that PyPI does not have one single manifest file for
the available package versions, but rather one manifest per package
name. Due to this, in order to sync all packages from PyPI it would be
necessary to make around 45-50,000 web requests just to find out what
would need to be downloaded, and then of course we would need to
perform the actual package downloads.

That said, we are working on a plan to have Pulp be able to lazy fetch
packages as they are requested. This plan will take a long time to
implement (so don't expect it in any of our close releases) but I
think it will solve this problem in a performant way.

Another possible solution may be Warehouse[0]. I've been talking to
the PyPA developers about this problem, and they are aware that it
needs to be solved. They may fix it there, in which case we can get
the Python importer to be aware of all the packages.

I have also considered just doing the 50k requests anyway. I suspect
that PyPI won't like if we do that, but it is technically possible as
well.

I say go ahead and file an RFE. I'll think some more about how we
might be able to get it working. Thanks for the note, and I hope you
enjoy the plugin otherwise!

[0] https://warehouse.python.org/

Applied in changeset 5270947abc578d13c942f5cc64bf27556c212ebc.