Pulp

Pulp today does a lot of system configuration at the RPM level. We should generalize that by relying on Ansible playbook to handle two things:

• non Python dependency installation
• system configuration

That Ansible work is being tracked as part of issue 2443 so I've named that as a blocker for this issue

In Pulp3 we will have several setup.py files. I believe to publish on PyPI each published package corresponds with 1 setup.py file. Is that correct?

Answering my own question here, yes I believe each setup.py we maintain will need to be published separately on PyPI.

There are two main questions we need answers to:

• What are the PyPI names going to be?
• Will all packages install under a top level directory or not?

What are the PyPI names going to be?¶

Pulp will need to register a lot of names on PyPI because we have many distinct Python packages. For example platform needs at least 4: one for the server, the cli, the common bits, and the streamer. This ticket is only for platform, but the decisions here will likely be applied to any given plugin the same way which have similar needs. For example, each plugin we maintain will likely need 3: one for the plugin, one for its CLI extensions, and another for the plugins common bits. With 5 actively maintained plugins that 15 packages and 4 for platform for an expected total of 19 PyPI registrations.

Through some in-person brainstorming, mhrivnak and I came up with the ones below. All of these are available on PyPI at this time.

Idea 1: Use a prefix with underscores after it¶

Three prefixes we thought of are pulpproj, pulpproject, or pulp. Any $PREFIX would be used like $PREFIX_platform, $PREFIX_cli, $PREFIX_common, and $PREFIX_streamer.

For the pulpproj prefix, you would pip install with:

pip install pulpproj_platform  # Install the server which brings in pulpproj_common as a dependency
pip install pulpproj_cli  # Install the cli which brings in pulpproj_common as a dependency
pip install pulpproj_streamer  # Install the streamer which brings in pulpproj_platform as a dependency

For the pulpproject prefix, you would pip install with:

pip install pulpproject_platform  # Install the server which brings in pulpproject_common as a dependency
pip install pulpproject_cli  # Install the cli which brings in pulpproject_common as a dependency
pip install pulpproject_streamer  # Install the streamer which brings in pulpproject_platform as a dependency

For the pulp prefix, you would pip install with:

pip install pulp_platform  # Install the server which brings in pulp_common as a dependency
pip install pulp_cli  # Install the cli which brings in pulp_common as a dependency
pip install pulp_streamer  # Install the streamer which brings in pulp_platform as a dependency

Idea 2: A variation on Idea 1 ^, except don't append _platform for platform¶

This only works for the prefix of pulpproj or pulpproject. For that one, platform would use the following names:

pip install pulpproject  # Install the server which brings in pulpproject_common as a dependency
pip install pulpproject_cli  # Install the cli which brings in pulpproject_common as a dependency
pip install pulpproject_streamer  # Install the streamer which brings in pulpproject as a dependency

This does not work for the prefix pulp because the existing pulp on PyPI already reserves it.

Will all packages install under a top level directory or not?¶

This is not something we need to decide to move forward, but our name choices above will restrict our layout choices, so I include it here also. Note that the way Pulp packages are imported are mostly decided by this decision.

'yes' all packages will install under a top level dir¶

With this, we would have to use the pulpproj or pulpproject prefixes. For example it would look like this:

[bmbouter@dhcp129-10 ~]$ tree /usr/lib/python2.7/site-packages/pulpproject/
/usr/lib/python2.7/site-packages/pulpproject/
├── cli
├── common
├── platform
└── streamer

With this type of layout, an import in platform of common bits would be: from pulpproject.common import ...

Note, we cannot use the pulp as the top level directory because we do not own the registration for pulp on PyPI.

'no' all packages will install in their own site-packages folders¶

With this, every package would be installed as a top level package on its own. Any prefix would work for this. For the pulpproject prefix it would have the following paths:

/usr/lib/python2.7/site-packages/pulpproject_cli
/usr/lib/python2.7/site-packages/pulpproject_common
/usr/lib/python2.7/site-packages/pulpproject_platform
/usr/lib/python2.7/site-packages/pulpproject_streamer

In this case an import in platform importing from common would import as: from pulpproject_common import ...

Good summary. Two small additions:

1. The length problem really shows itself with plugins.

pip install pulp_rpm_extensions

pip install pulpproject_rpm_extensions

2. The platform packages are likely to be a little different, but those details don't matter much for this discussion. But just to illustrate it, I expect we would start from the idea of one python package and one rpm package per second-level namespace in today's tree. For example:

pulp.app becomes pulp_app
pulp.bindings becomes pulp_bindings

and so on for pulp.common, pulp.plugin, pulp.tasking, etc.

Then we will likely find that several of those namespaces always need to be delivered together, and could bundle them into one package and RPM, such as pulp_platform. I would anticipate pulp_platform to include "app", "plugin", and "tasking" namespaces at least, if not more.

Mhrivnak found out that it's super easy to transfer ownership of pypi packages, so I am going to preemptively register some plugins names, and we can transfer ownership to community members later:

pulp-deb
pulp-deb-common
pulp-deb-cli

-0 to registering additional plugins. I think the plugin owner should handle their own plugin start to finish. They may not even want our names.

bmbouter, yeah I only ended up registering the pulp-deb packages. mhrivnak and I could not decided on the other package names (pulp-npm vs pulp-js, pulp-ruby vs pulp-gem and pulp-whateverjavauses)

The original motivation of registering the additional packages was so no one else could encroach on our package names. but because there was no obvious name for the other plugins, I decided not to, since I didn't want squat on names that might not be used in the future.

That sounds fine; there is a pulp-deb plugin at least so whenever @misa is ready to publish on PyPI we can hand it over to him. Thanks for registering all of these. This was well done.