Project

Profile

Help

Story #3206

Support for module metadata in pulp_rpm

Added by ralph almost 2 years ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Platform Release:
2.17.0
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

We need support in Pulp and the pulp_rpm plugin for "modulemd" module metadata.

For an overview of the "modularity" project, see https://docs.pagure.org/modularity/

You can see an example of a yum repo with a published modules metadata file associated with the repo metadata here: https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-28/compose/Modular/x86_64/os/

Details on the modulemd spec can be found here: https://pagure.io/modulemd

The file is similar to the updateinfo.xml file in this way:

- When constructing an update repo, each change to the repo is associated with some text or other metadata. The updateinfo.xml file is constructed from the sum of all these changes to the repo.
- For a repo housing modules, each change to the repo will be associated with multiple modulemd files. These files should be stored and treated as fragments. They should be concatenated together to produce the final SHASUM-modules.yaml.gz file, linked to from the repomd.xml file.

The modulemd data does not replace the updateinfo.xml, but sits next to it.

As a result of this story, Pulp Rpm will support modules for sync, upload, publish and copy.


Checklist


Related issues

Related to RPM Support - Issue #3353: Missing docs on importing RPM repository module metadata CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Story #3657: As a user I can manage modulemd content CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Task #3661: Add model for modular content CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Story #3766: As a user I can manage modulemd-defaults content CLOSED - CURRENTRELEASE Actions
Blocked by RPM Support - Task #3708: Add model for Modulemd-defaults CLOSED - CURRENTRELEASE Actions

History

#1 Updated by rmcgover almost 2 years ago

This could work by introducing a new type of unit understood by pulp_rpm, which points to a single modulemd YAML document.

Using yum_distributor to publish a repo with those units would publish the concatenated modulemd YAML documents as a repodata file of type "modules".

#2 Updated by amacdona@redhat.com almost 2 years ago

  • Project changed from Pulp to RPM Support

#4 Updated by ttereshc almost 2 years ago

@ralph, @rmcgover, could you provide use cases you'd like to see supported by Pulp with regards to this new metadata file?
The description of this RFE is helpful for understanding what this modular metadata is, thanks! However, knowing specific use cases is also needed.

I think right now, without any changes to Pulp, one can sync and publish such repo, basically mirror it. The new metadata file won't be parsed or analysed but published as-is.

#5 Updated by ttereshc almost 2 years ago

Does this RFE take into consideration changes described in this blog post: https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/ ?
Does the provided link point to the simplified version or not? http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/27/Server/x86_64/os/repodata/

#6 Updated by ralph almost 2 years ago

ttereshc wrote:

@ralph, @rmcgover, could you provide use cases you'd like to see supported by Pulp with regards to this new metadata file?
The description of this RFE is helpful for understanding what this modular metadata is, thanks! However, knowing specific use cases is also needed.

@rmcgover will have to answer this one.

I think right now, without any changes to Pulp, one can sync and publish such repo, basically mirror it. The new metadata file won't be parsed or analysed but published as-is.

Right, we're doing this now as a workaround but, we would like Pulp to be able to re-assemble the metadata file from fragments (like it can do with an updateinfo.xml file).

Does this RFE take into consideration changes described in this blog post: https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/ ?

It does. The changes there don't impact the details of the metadata handling here.

Does the provided link point to the simplified version or not? http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/27/Server/x86_64/os/repodata/

Yes, the repo you're looking at is from before the blog post linked above, but there is no change at that level.

#7 Updated by rmcgover almost 2 years ago

Regarding use-cases:

  • As RCM, I want to import modulemd documents into a Pulp repo, and have them published together in a "modules" file, to enable modularity features in client tooling (DNF).
  • As RCM, I want to be able to determine if a certain modulemd document is already present in a Pulp repo, so I can determine whether a module has been shipped.

And things I'm not sure of - @ralph, need your help:

  • Do we need to be able to remove modulemd documents? (I assume answer should be "yes", even if in normal circumstances we don't do this, simply because something could be accidentally added and there should be a way to undo that)
  • Do we need to "upgrade" modulemd documents (e.g. "add new version of python2-ecosystem module and remove old version")
  • Do we need any restrictions on which kinds of modulemd documents are allowed to exist in a repo at the same time? At least I suspect that we don't want to allow duplicates of (name, version) ... are there others?

I also suspect that our publishing tools might find it useful to put some fields in pulp_user_metadata for each modulemd document, but I don't have specific cases in mind yet.

Currently it looks like http://dl.fedoraproject.org/pub/fedora/linux/modular/updates/testing/27/Server/x86_64/repodata/ is a better example of a repo with modules, as it has more than one (while the other repo only had a single modulemd document in the modules file when I checked).

#8 Updated by ralph almost 2 years ago

rmcgover wrote:

  • Do we need to be able to remove modulemd documents? (I assume answer should be "yes", even if in normal circumstances we don't do this, simply because something could be accidentally added and there should be a way to undo that)

Yes, we don't normally want to do this, but we need the ability to do it in extreme circumstances.

  • Do we need to "upgrade" modulemd documents (e.g. "add new version of python2-ecosystem module and remove old version")

No, new ones just get added to the mix. This is because a client may stay on the old version for some long time. The module data in their repo needs to continue to point to their old NVRs on disk for when they eventually do upgrade.

  • Do we need any restrictions on which kinds of modulemd documents are allowed to exist in a repo at the same time? At least I suspect that we don't want to allow duplicates of (name, version) ... are there others?

Good question... and I think the answer is yes. We shouldn't allow duplicates of "name, stream, version, context". We should allow duplicates of "name, stream" and "name, stream, version", though. I cannot think of any other restrictions that should be imposed at the Pulp level.

#9 Updated by ralph over 1 year ago

Checking in. Is any further information needed here?

#10 Updated by ttereshc over 1 year ago

  • Related to Issue #3353: Missing docs on importing RPM repository module metadata added

#12 Updated by ttereshc over 1 year ago

  • Description updated (diff)

Updated description with the up-to-date link to a modular repo

#13 Updated by ttereshc over 1 year ago

modulemd metadata is handled by the following library:
https://github.com/fedora-modularity/libmodulemd

It has Python bindings.
This is an example how DNF uses the bindings:
https://github.com/rpm-software-management/dnf/commit/fc3cc097234222ffecc14eaf0913db739517b1b0

In case of parsing individual modulemd chunks naming policy doc might be helpful: https://docs.pagure.org/modularity/development/building-modules/naming-policy.html (probably need N:S:V:C:A for indexing the metadata; ignore Profile)

#14 Updated by ipanova@redhat.com over 1 year ago

  • Description updated (diff)

#15 Updated by ipanova@redhat.com over 1 year ago

  • Description updated (diff)

#16 Updated by smcdowel over 1 year ago

Does this story also cover the accessing of this information in pulp (i.e. via api), or does that use case need a separate story?

#17 Updated by ralph over 1 year ago

smcdowel wrote:

Does this story also cover the accessing of this information in pulp (i.e. via api), or does that use case need a separate story?

From a recent email thread, it looks like (yes) this information will need to be exposed in the API so that it can be consumed by other tools in the RH content-delivery ecosystem. I do not know if this story as currently written covers that.

#18 Updated by ipanova@redhat.com over 1 year ago

  • Blocks Story #3659: Add a migration for transition from basic to advanced modular support added

#19 Updated by ipanova@redhat.com over 1 year ago

  • Related to Story #3657: As a user I can manage modulemd content added

#20 Updated by ipanova@redhat.com over 1 year ago

  • Blocks Story #3660: As a user I can manage modular content via pulp-admin added

#21 Updated by ipanova@redhat.com over 1 year ago

  • Related to Task #3661: Add model for modular content added

#22 Updated by ipanova@redhat.com over 1 year ago

  • Blocked by Task #3708: Add model for Modulemd-defaults added

#23 Updated by ipanova@redhat.com over 1 year ago

  • Related to Story #3766: As a user I can manage modulemd-defaults content added

#24 Updated by dkliban@redhat.com over 1 year ago

  • Blocks deleted (Story #3659: Add a migration for transition from basic to advanced modular support)

#25 Updated by ipanova@redhat.com about 1 year ago

  • Blocks deleted (Story #3660: As a user I can manage modular content via pulp-admin)

#26 Updated by ipanova@redhat.com about 1 year ago

  • Status changed from NEW to MODIFIED

#27 Updated by ipanova@redhat.com about 1 year ago

  • Smash Test set to 1122

#28 Updated by dkliban@redhat.com about 1 year ago

  • Checklist item add unittests and pulp-smash tests set to Done

#29 Updated by dkliban@redhat.com about 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Platform Release set to 2.17.0

#30 Updated by kersom 12 months ago

This feature was already tested. Pulp 2 Test Case

#31 Updated by bmbouter 6 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF