Project

Profile

Help

Story #2444

Create a PyPI account for Pulp3 core and all plugins

Added by bmbouter about 3 years ago. Updated 8 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 20

Description

Register the following on PyPI:

pulpcore
pulpcore-cli
pulpcore-streamer
pulpcore-common
pulpcore-plugin

pulp-rpm
pulp-rpm-cli
pulp-rpm-common

pulp-puppet
pulp-puppet-cli
pulp-puppet-common

pulp-python
pulp-python-cli
pulp-python-common

pulp-docker
pulp-docker-cli
pulp-docker-common

pulp-ostree
pulp-ostree-cli
pulp-ostree-common

pulp-file
pulp-file-cli
pulp-file-common

Checklist


Related issues

Related to Pulp - Story #278: Add documentation about Pulp's setup.py requirements NEW Actions
Related to Pulp - Task #2463: Reorganize 3.0-dev branch MODIFIED Actions
Related to RPM Support - Task #2699: Ensure all default CA_PATH settings to work with multiple distribution default locations in Pulp3 NEW Actions
Related to Pulp - Issue #2531: pulp python modules use other project's namespace CLOSED - DUPLICATE Actions
Blocked by Pulp - Task #2443: create the 'ansible-pulp' project inside pulp GitHub organization CLOSED - WONTFIX Actions

Associated revisions

Revision 253ae59c View on GitHub
Added by werwty over 2 years ago

refactor pulp package to pulpcore for platform, common, and streamer

ref #2444
https://pulp.plan.io/issues/2444

Revision 253ae59c View on GitHub
Added by werwty over 2 years ago

refactor pulp package to pulpcore for platform, common, and streamer

ref #2444
https://pulp.plan.io/issues/2444

Revision a9675d7d View on GitHub
Added by werwty over 2 years ago

Move plugin to it's own package for pypi packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision a9675d7d View on GitHub
Added by werwty over 2 years ago

Move plugin to it's own package for pypi packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 6c52914c View on GitHub
Added by werwty over 2 years ago

Update vagrant to use refactored pulp package names

ref #2444
https://pulp.plan.io/issues/2444

Revision 06d70f44 View on GitHub
Added by werwty over 2 years ago

Update setup.py files for pypi packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 06d70f44 View on GitHub
Added by werwty over 2 years ago

Update setup.py files for pypi packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 59daa197 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 59daa197 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 59daa197 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 59daa197 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 411f8fc4 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 9f9712d4 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

Revision 4e4687ab View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packagingi

ref #2444
https://pulp.plan.io/issues/2444

Revision 96c90367 View on GitHub
Added by werwty over 2 years ago

Update setup.py for pulp3 PyPI packaging

ref #2444
https://pulp.plan.io/issues/2444

History

#1 Updated by bmbouter about 3 years ago

  • Blocked by Task #2443: create the 'ansible-pulp' project inside pulp GitHub organization added

#2 Updated by bmbouter about 3 years ago

Pulp today does a lot of system configuration at the RPM level. We should generalize that by relying on Ansible playbook to handle two things:

  • non Python dependency installation
  • system configuration

That Ansible work is being tracked as part of issue 2443 so I've named that as a blocker for this issue

#3 Updated by bmbouter about 3 years ago

  • Related to Story #278: Add documentation about Pulp's setup.py requirements added

#4 Updated by bmbouter about 3 years ago

In Pulp3 we will have several setup.py files. I believe to publish on PyPI each published package corresponds with 1 setup.py file. Is that correct?

#5 Updated by bmbouter about 3 years ago

  • Related to Task #2463: Reorganize 3.0-dev branch added

#6 Updated by bmbouter about 3 years ago

Answering my own question here, yes I believe each setup.py we maintain will need to be published separately on PyPI.

#7 Updated by bmbouter almost 3 years ago

There are two main questions we need answers to:

  • What are the PyPI names going to be?
  • Will all packages install under a top level directory or not?

What are the PyPI names going to be?

Pulp will need to register a lot of names on PyPI because we have many distinct Python packages. For example platform needs at least 4: one for the server, the cli, the common bits, and the streamer. This ticket is only for platform, but the decisions here will likely be applied to any given plugin the same way which have similar needs. For example, each plugin we maintain will likely need 3: one for the plugin, one for its CLI extensions, and another for the plugins common bits. With 5 actively maintained plugins that 15 packages and 4 for platform for an expected total of 19 PyPI registrations.

Through some in-person brainstorming, @mhrivnak and I came up with the ones below. All of these are available on PyPI at this time.

Idea 1: Use a prefix with underscores after it

Three prefixes we thought of are pulpproj, pulpproject, or pulp. Any $PREFIX would be used like $PREFIX_platform, $PREFIX_cli, $PREFIX_common, and $PREFIX_streamer.

For the pulpproj prefix, you would pip install with:

pip install pulpproj_platform  # Install the server which brings in pulpproj_common as a dependency
pip install pulpproj_cli  # Install the cli which brings in pulpproj_common as a dependency
pip install pulpproj_streamer  # Install the streamer which brings in pulpproj_platform as a dependency

For the pulpproject prefix, you would pip install with:

pip install pulpproject_platform  # Install the server which brings in pulpproject_common as a dependency
pip install pulpproject_cli  # Install the cli which brings in pulpproject_common as a dependency
pip install pulpproject_streamer  # Install the streamer which brings in pulpproject_platform as a dependency

For the pulp prefix, you would pip install with:

pip install pulp_platform  # Install the server which brings in pulp_common as a dependency
pip install pulp_cli  # Install the cli which brings in pulp_common as a dependency
pip install pulp_streamer  # Install the streamer which brings in pulp_platform as a dependency

Idea 2: A variation on Idea 1 ^, except don't append _platform for platform

This only works for the prefix of pulpproj or pulpproject. For that one, platform would use the following names:

pip install pulpproject  # Install the server which brings in pulpproject_common as a dependency
pip install pulpproject_cli  # Install the cli which brings in pulpproject_common as a dependency
pip install pulpproject_streamer  # Install the streamer which brings in pulpproject as a dependency

This does not work for the prefix pulp because the existing pulp on PyPI already reserves it.

Will all packages install under a top level directory or not?

This is not something we need to decide to move forward, but our name choices above will restrict our layout choices, so I include it here also. Note that the way Pulp packages are imported are mostly decided by this decision.

'yes' all packages will install under a top level dir

With this, we would have to use the pulpproj or pulpproject prefixes. For example it would look like this:

[bmbouter@dhcp129-10 ~]$ tree /usr/lib/python2.7/site-packages/pulpproject/
/usr/lib/python2.7/site-packages/pulpproject/
├── cli
├── common
├── platform
└── streamer

With this type of layout, an import in platform of common bits would be: from pulpproject.common import ...

Note, we cannot use the pulp as the top level directory because we do not own the registration for pulp on PyPI.

'no' all packages will install in their own site-packages folders

With this, every package would be installed as a top level package on its own. Any prefix would work for this. For the pulpproject prefix it would have the following paths:

/usr/lib/python2.7/site-packages/pulpproject_cli
/usr/lib/python2.7/site-packages/pulpproject_common
/usr/lib/python2.7/site-packages/pulpproject_platform
/usr/lib/python2.7/site-packages/pulpproject_streamer

In this case an import in platform importing from common would import as: from pulpproject_common import ...

#8 Updated by mhrivnak almost 3 years ago

Good summary. Two small additions:

1. The length problem really shows itself with plugins.

pip install pulp_rpm_extensions

pip install pulpproject_rpm_extensions

2. The platform packages are likely to be a little different, but those details don't matter much for this discussion. But just to illustrate it, I expect we would start from the idea of one python package and one rpm package per second-level namespace in today's tree. For example:

pulp.app becomes pulp_app
pulp.bindings becomes pulp_bindings

and so on for pulp.common, pulp.plugin, pulp.tasking, etc.

Then we will likely find that several of those namespaces always need to be delivered together, and could bundle them into one package and RPM, such as pulp_platform. I would anticipate pulp_platform to include "app", "plugin", and "tasking" namespaces at least, if not more.

#9 Updated by bizhang over 2 years ago

  • Related to Task #2699: Ensure all default CA_PATH settings to work with multiple distribution default locations in Pulp3 added

#10 Updated by bmbouter over 2 years ago

  • Subject changed from Publish Pulp on PyPI to Create PyPI accounts for Pulp3 core and all plugins
  • Description updated (diff)

Rewriting based on lots of discussion. See the recap of that discussion here: https://www.redhat.com/archives/pulp-dev/2017-May/msg00030.html

#11 Updated by ttereshc over 2 years ago

  • Groomed changed from No to Yes

#12 Updated by bmbouter over 2 years ago

  • Subject changed from Create PyPI accounts for Pulp3 core and all plugins to Create a PyPI account for Pulp3 core and all plugins
  • Description updated (diff)
  • Sprint/Milestone set to 39

We talked about the possibility of pre-registering names for more plugins. We decided that we need more info about how to delegate access if we pre-register a plugin that then needs to be pushed and maintained by a plugin writer.

#13 Updated by bizhang over 2 years ago

  • Assignee set to bizhang

#14 Updated by bizhang over 2 years ago

  • Status changed from NEW to ASSIGNED

#15 Updated by bizhang over 2 years ago

  • Description updated (diff)

#17 Updated by bizhang over 2 years ago

  • Status changed from POST to MODIFIED

#18 Updated by bmbouter over 2 years ago

  • Related to Issue #2531: pulp python modules use other project's namespace added

#19 Updated by bizhang over 2 years ago

Mhrivnak found out that it's super easy to transfer ownership of pypi packages, so I am going to preemptively register some plugins names, and we can transfer ownership to community members later:

pulp-deb
pulp-deb-common
pulp-deb-cli

#20 Updated by bmbouter over 2 years ago

-0 to registering additional plugins. I think the plugin owner should handle their own plugin start to finish. They may not even want our names.

#21 Updated by bizhang over 2 years ago

@bmbouter, yeah I only ended up registering the pulp-deb packages. mhrivnak and I could not decided on the other package names (pulp-npm vs pulp-js, pulp-ruby vs pulp-gem and pulp-whateverjavauses)

The original motivation of registering the additional packages was so no one else could encroach on our package names. but because there was no obvious name for the other plugins, I decided not to, since I didn't want squat on names that might not be used in the future.

#22 Updated by bmbouter over 2 years ago

That sounds fine; there is a pulp-deb plugin at least so whenever @misa is ready to publish on PyPI we can hand it over to him. Thanks for registering all of these. This was well done.

#23 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 20

#24 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (39)

#25 Updated by daviddavis 8 months ago

  • Sprint/Milestone set to 3.0

#26 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF