RPM Support

This could work by introducing a new type of unit understood by pulp_rpm, which points to a single modulemd YAML document.

Using yum_distributor to publish a repo with those units would publish the concatenated modulemd YAML documents as a repodata file of type "modules".

@ralph, rmcgover, could you provide use cases you'd like to see supported by Pulp with regards to this new metadata file?
The description of this RFE is helpful for understanding what this modular metadata is, thanks! However, knowing specific use cases is also needed.

I think right now, without any changes to Pulp, one can sync and publish such repo, basically mirror it. The new metadata file won't be parsed or analysed but published as-is.

Does this RFE take into consideration changes described in this blog post: https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/ ?
Does the provided link point to the simplified version or not? http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/27/Server/x86_64/os/repodata/

ttereshc wrote:

@ralph, rmcgover, could you provide use cases you'd like to see supported by Pulp with regards to this new metadata file?
The description of this RFE is helpful for understanding what this modular metadata is, thanks! However, knowing specific use cases is also needed.

rmcgover will have to answer this one.

I think right now, without any changes to Pulp, one can sync and publish such repo, basically mirror it. The new metadata file won't be parsed or analysed but published as-is.

Right, we're doing this now as a workaround but, we would like Pulp to be able to re-assemble the metadata file from fragments (like it can do with an updateinfo.xml file).

Does this RFE take into consideration changes described in this blog post: https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/ ?

It does. The changes there don't impact the details of the metadata handling here.

Does the provided link point to the simplified version or not? http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/27/Server/x86_64/os/repodata/

Yes, the repo you're looking at is from before the blog post linked above, but there is no change at that level.

Regarding use-cases:

• As RCM, I want to import modulemd documents into a Pulp repo, and have them published together in a "modules" file, to enable modularity features in client tooling (DNF).

• As RCM, I want to be able to determine if a certain modulemd document is already present in a Pulp repo, so I can determine whether a module has been shipped.

And things I'm not sure of - @ralph, need your help:

• Do we need to be able to remove modulemd documents? (I assume answer should be "yes", even if in normal circumstances we don't do this, simply because something could be accidentally added and there should be a way to undo that)

• Do we need to "upgrade" modulemd documents (e.g. "add new version of python2-ecosystem module and remove old version")

• Do we need any restrictions on which kinds of modulemd documents are allowed to exist in a repo at the same time? At least I suspect that we don't want to allow duplicates of (name, version) ... are there others?

I also suspect that our publishing tools might find it useful to put some fields in pulp_user_metadata for each modulemd document, but I don't have specific cases in mind yet.

Currently it looks like http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/testing/27/Server/x86_64/repodata/ is a better example of a repo with modules, as it has more than one (while the other repo only had a single modulemd document in the modules file when I checked).

rmcgover wrote:

• Do we need to be able to remove modulemd documents? (I assume answer should be "yes", even if in normal circumstances we don't do this, simply because something could be accidentally added and there should be a way to undo that)

Yes, we don't normally want to do this, but we need the ability to do it in extreme circumstances.

• Do we need to "upgrade" modulemd documents (e.g. "add new version of python2-ecosystem module and remove old version")

No, new ones just get added to the mix. This is because a client may stay on the old version for some long time. The module data in their repo needs to continue to point to their old NVRs on disk for when they eventually do upgrade.

• Do we need any restrictions on which kinds of modulemd documents are allowed to exist in a repo at the same time? At least I suspect that we don't want to allow duplicates of (name, version) ... are there others?

Good question... and I think the answer is yes. We shouldn't allow duplicates of "name, stream, version, context". We should allow duplicates of "name, stream" and "name, stream, version", though. I cannot think of any other restrictions that should be imposed at the Pulp level.

Checking in. Is any further information needed here?

modulemd metadata is handled by the following library:
https://github.com/fedora-modularity/libmodulemd

It has Python bindings.
This is an example how DNF uses the bindings:
https://github.com/rpm-software-management/dnf/commit/fc3cc097234222ffecc14eaf0913db739517b1b0

In case of parsing individual modulemd chunks naming policy doc might be helpful: https://docs.pagure.org/modularity/development/building-modules/naming-policy.html (probably need N:S:V:C:A for indexing the metadata; ignore Profile)

Does this story also cover the accessing of this information in pulp (i.e. via api), or does that use case need a separate story?

smcdowel wrote:

Does this story also cover the accessing of this information in pulp (i.e. via api), or does that use case need a separate story?

From a recent email thread, it looks like (yes) this information will need to be exposed in the API so that it can be consumed by other tools in the RH content-delivery ecosystem. I do not know if this story as currently written covers that.

This feature was already tested. Pulp 2 Test Case