Story #3206
closedSupport for module metadata in pulp_rpm
Added by ralph over 6 years ago. Updated over 5 years ago.
0%
Description
We need support in Pulp and the pulp_rpm plugin for "modulemd" module metadata.
For an overview of the "modularity" project, see https://docs.pagure.org/modularity/
You can see an example of a yum repo with a published modules metadata file associated with the repo metadata here: https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-28/compose/Modular/x86_64/os/
Details on the modulemd spec can be found here: https://pagure.io/modulemd
The file is similar to the updateinfo.xml file in this way:
- When constructing an update repo, each change to the repo is associated with some text or other metadata. The updateinfo.xml file is constructed from the sum of all these changes to the repo.
- For a repo housing modules, each change to the repo will be associated with multiple modulemd files. These files should be stored and treated as fragments. They should be concatenated together to produce the final SHASUM-modules.yaml.gz file, linked to from the repomd.xml file.
The modulemd data does not replace the updateinfo.xml, but sits next to it.
As a result of this story, Pulp Rpm will support modules for sync, upload, publish and copy.
Related issues
Updated by rmcgover over 6 years ago
This could work by introducing a new type of unit understood by pulp_rpm, which points to a single modulemd YAML document.
Using yum_distributor to publish a repo with those units would publish the concatenated modulemd YAML documents as a repodata file of type "modules".
Updated by amacdona@redhat.com over 6 years ago
- Project changed from Pulp to RPM Support
Updated by ttereshc over 6 years ago
@ralph, rmcgover, could you provide use cases you'd like to see supported by Pulp with regards to this new metadata file?
The description of this RFE is helpful for understanding what this modular metadata is, thanks! However, knowing specific use cases is also needed.
I think right now, without any changes to Pulp, one can sync and publish such repo, basically mirror it. The new metadata file won't be parsed or analysed but published as-is.
Updated by ttereshc over 6 years ago
Does this RFE take into consideration changes described in this blog post: https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/ ?
Does the provided link point to the simplified version or not? http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/27/Server/x86_64/os/repodata/
Updated by ralph over 6 years ago
ttereshc wrote:
@ralph, rmcgover, could you provide use cases you'd like to see supported by Pulp with regards to this new metadata file?
The description of this RFE is helpful for understanding what this modular metadata is, thanks! However, knowing specific use cases is also needed.
rmcgover will have to answer this one.
I think right now, without any changes to Pulp, one can sync and publish such repo, basically mirror it. The new metadata file won't be parsed or analysed but published as-is.
Right, we're doing this now as a workaround but, we would like Pulp to be able to re-assemble the metadata file from fragments (like it can do with an updateinfo.xml file).
Does this RFE take into consideration changes described in this blog post: https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/ ?
It does. The changes there don't impact the details of the metadata handling here.
Does the provided link point to the simplified version or not? http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/27/Server/x86_64/os/repodata/
Yes, the repo you're looking at is from before the blog post linked above, but there is no change at that level.
Updated by rmcgover over 6 years ago
Regarding use-cases:
- As RCM, I want to import modulemd documents into a Pulp repo, and have them published together in a "modules" file, to enable modularity features in client tooling (DNF).
- As RCM, I want to be able to determine if a certain modulemd document is already present in a Pulp repo, so I can determine whether a module has been shipped.
And things I'm not sure of - @ralph, need your help:
- Do we need to be able to remove modulemd documents? (I assume answer should be "yes", even if in normal circumstances we don't do this, simply because something could be accidentally added and there should be a way to undo that)
- Do we need to "upgrade" modulemd documents (e.g. "add new version of python2-ecosystem module and remove old version")
- Do we need any restrictions on which kinds of modulemd documents are allowed to exist in a repo at the same time? At least I suspect that we don't want to allow duplicates of (name, version) ... are there others?
I also suspect that our publishing tools might find it useful to put some fields in pulp_user_metadata for each modulemd document, but I don't have specific cases in mind yet.
Currently it looks like http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/testing/27/Server/x86_64/repodata/ is a better example of a repo with modules, as it has more than one (while the other repo only had a single modulemd document in the modules file when I checked).
Updated by ralph over 6 years ago
rmcgover wrote:
- Do we need to be able to remove modulemd documents? (I assume answer should be "yes", even if in normal circumstances we don't do this, simply because something could be accidentally added and there should be a way to undo that)
Yes, we don't normally want to do this, but we need the ability to do it in extreme circumstances.
- Do we need to "upgrade" modulemd documents (e.g. "add new version of python2-ecosystem module and remove old version")
No, new ones just get added to the mix. This is because a client may stay on the old version for some long time. The module data in their repo needs to continue to point to their old NVRs on disk for when they eventually do upgrade.
- Do we need any restrictions on which kinds of modulemd documents are allowed to exist in a repo at the same time? At least I suspect that we don't want to allow duplicates of (name, version) ... are there others?
Good question... and I think the answer is yes. We shouldn't allow duplicates of "name, stream, version, context". We should allow duplicates of "name, stream" and "name, stream, version", though. I cannot think of any other restrictions that should be imposed at the Pulp level.
Updated by ralph over 6 years ago
Checking in. Is any further information needed here?
Updated by ttereshc over 6 years ago
- Related to Issue #3353: Missing docs on importing RPM repository module metadata added
Updated by ttereshc over 6 years ago
- Description updated (diff)
Updated description with the up-to-date link to a modular repo
Updated by ttereshc over 6 years ago
modulemd metadata is handled by the following library:
https://github.com/fedora-modularity/libmodulemd
It has Python bindings.
This is an example how DNF uses the bindings:
https://github.com/rpm-software-management/dnf/commit/fc3cc097234222ffecc14eaf0913db739517b1b0
In case of parsing individual modulemd chunks naming policy doc might be helpful: https://docs.pagure.org/modularity/development/building-modules/naming-policy.html (probably need N:S:V:C:A for indexing the metadata; ignore Profile)
Updated by smcdowel over 6 years ago
Does this story also cover the accessing of this information in pulp (i.e. via api), or does that use case need a separate story?
Updated by ralph over 6 years ago
smcdowel wrote:
Does this story also cover the accessing of this information in pulp (i.e. via api), or does that use case need a separate story?
From a recent email thread, it looks like (yes) this information will need to be exposed in the API so that it can be consumed by other tools in the RH content-delivery ecosystem. I do not know if this story as currently written covers that.
Updated by ipanova@redhat.com over 6 years ago
- Blocks Story #3659: Add a migration for transition from basic to advanced modular support added
Updated by ipanova@redhat.com over 6 years ago
- Related to Story #3657: As a user I can manage modulemd content added
Updated by ipanova@redhat.com over 6 years ago
- Blocks Story #3660: As a user I can manage modular content via pulp-admin added
Updated by ipanova@redhat.com over 6 years ago
- Related to Task #3661: Add model for modular content added
Updated by ipanova@redhat.com about 6 years ago
- Blocked by Task #3708: Add model for Modulemd-defaults added
Updated by ipanova@redhat.com about 6 years ago
- Related to Story #3766: As a user I can manage modulemd-defaults content added
Updated by dkliban@redhat.com about 6 years ago
- Blocks deleted (Story #3659: Add a migration for transition from basic to advanced modular support)
Updated by ipanova@redhat.com about 6 years ago
- Blocks deleted (Story #3660: As a user I can manage modular content via pulp-admin)
Updated by ipanova@redhat.com about 6 years ago
- Status changed from NEW to MODIFIED
Updated by dkliban@redhat.com almost 6 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
- Platform Release set to 2.17.0
Updated by kersom almost 6 years ago
This feature was already tested. Pulp 2 Test Case