Project

Profile

Help

Story #3740

Implement modularity content dependency solving

Added by milan about 2 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
2.19.0
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 47

Description

Motivation

Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.

Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).

Proposed solution

It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.

Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.

To support these usecases, a fake libsolv solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered bitmap from the libsolv library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.

References


Checklist


Subtasks

Task #3746: Investigate, whether exposing the @pool->considered@ bitmap is necessary for the modular dependencies to be resolvedCLOSED - COMPLETE

Actions

Related issues

Related to RPM Support - Story #4058: As a user, I can calculate applicability for modular contentCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependenciesCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Story #4162: As a user, I have dependency solving when copying modulesCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Test #4543: Test newly described definitions of recursive and recursive_conservative flagsCLOSED - COMPLETE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to RPM Support - Test #4364: Implement modularity content dependency solvingCLOSED - COMPLETE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 62bb34a6 View on GitHub
Added by milan over 1 year ago

Introduce modular deps processing

Modulemd and modulemd_defaults are now exposed to the libsolv solver. This allows for the (inter-)modular dependencies to be processed when copying a unit. Bot the relaxed and conservative dependency processing algorithms work with the functionality this patch introduces.

Fixes #3740 https://pulp.plan.io/issues/3740

History

#1 Updated by milan about 2 years ago

  • Description updated (diff)

#2 Updated by milan about 2 years ago

  • Description updated (diff)

#3 Updated by milan about 2 years ago

  • Description updated (diff)

#4 Updated by rchan about 2 years ago

  • Sprint/Milestone set to 2.17.0

Adding to 2.17.0 milestone. This is one of the required deliverables.

#5 Updated by ipanova@redhat.com about 2 years ago

  • Sprint/Milestone deleted (2.17.0)

#6 Updated by milan almost 2 years ago

  • Description updated (diff)

#7 Updated by milan almost 2 years ago

  • Description updated (diff)

#8 Updated by milan almost 2 years ago

  • Description updated (diff)

#9 Updated by milan almost 2 years ago

  • Description updated (diff)

#10 Updated by ipanova@redhat.com almost 2 years ago

inter-modular dependency support will lead to a model change of Modulemd. A new field should be added. called `dependencies`

  dependencies:
  - buildrequires:
      httpd: [2.4]
      platform: [el8]
    requires:
      httpd: [2.4]
      nginx: [1.14]
      platform: [el8]

This is an example of how the deps could look. As per spec [0], there might be multiple instances of Modulemd.Dependencies() object, where each of them will describe different way of build with varying build time and runtime.

During recursive copy we do not really care if those modules are required during build time or runtime, I suggest to make a union between `buildrequires` and `requires` per instance of Modulemd.Dependencies() object.

I suggest the following data structure, where each dict will be a translated Modulemd.Dependencies() object.


dependencies: [
                            {module-name1:  [stream1, stream2, stream3],
                             module-name2: [stream1, stream2]
                            },
                            {module-name2: [stream3],
                            module-name3: [stream1, stream2]
                            }
                    ]

Note** there might me inclusive and exclusive lists of stream, where exclusive is marked with a minus sign. We need to preserve the minus for later conflicts solving.

Note** the `platform` module is synthesized at runtime by dnf, we can ignore it and not include in the deps info at all.

[0] https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml#L78

#11 Updated by milan almost 2 years ago

A new field should be added. called `dependencies`

Another option could be a list of flat dictionaries like:

[
   {
     "name": "foo",
     "streams": ["stable"],
     "type": "runtime"
  },
  {
    "name": "bar",
    "streams": ["master"],
    "type": "build"
  },
  }
    "name": "foo",
    "streams": ["stable"],
    "type": "build"
  }
]

This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".

#12 Updated by milan almost 2 years ago

milan wrote:

A new field should be added. called `dependencies`

Another option could be a list of flat dictionaries like:

[...]

This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".

Actually, it won't work. It would loose the "inner build environments links" which exist so that build environments can have mutually exclusive dependencies and I guess the solver would freak out. Thanks Ina for pointing this out!

#13 Updated by ipanova@redhat.com almost 2 years ago

ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.

+1 to drop buildrequires and include in the proposed structure just the requires.

#14 Updated by ipanova@redhat.com almost 2 years ago

we should also consider possibly using https://github.com/fedora-modularity/fus

#15 Updated by milan over 1 year ago

wrote:

ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.

+1 to drop buildrequires and include in the proposed structure just the requires.

Thing is, the consumers of our published content might not be able to rebuild the modules themselves i.e we might create an installable but un-buildable repo. This isn't the case for for ursine content now, though we don't process SRPM deps recursively either.

Fus on the other hand considers the build-time dependencies.

#16 Updated by milan over 1 year ago

  • Related to Story #4058: As a user, I can calculate applicability for modular content added

#17 Updated by ttereshc over 1 year ago

To be clear for a quick reader, the relation with #4058 is weak.
#4058 requires only info of the first level module dependencies, if they are enabled on a consumer or not.

List of dependencies (requires only, not buildrequires) is added to the Modulemd model with this commit: https://github.com/pulp/pulp_rpm/commit/990f65a9bba8e3f9faf2f89165f2a88015dad1c1

#18 Updated by milan over 1 year ago

wrote:

we should also consider possibly using https://github.com/fedora-modularity/fus

There are some trade-offs with Fus to consider:

  • it is a command line tool that we'd have to integrate the way Pungi does
  • we'd have to semi-publish (all) the source and destination repositories metadata
  • this might have performance impact but we can afford experimenting with this idea

On the other hand, reimplementing some parts of both the Pungi and Fus tools seems like introducing code duplicity.
Still, the upside would be more control over how we resolve the dependencies.
There's an (undecided) Fus intgration e-mail discussion that can be queried for further information.

#19 Updated by milan over 1 year ago

  • Related to Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies added

#20 Updated by milan over 1 year ago

  • Related to Story #4162: As a user, I have dependency solving when copying modules added

#21 Updated by milan over 1 year ago

  • Status changed from NEW to POST

#22 Updated by milan over 1 year ago

  • Checklist item associating a module content unit supports the recursive behavior set to Done
  • Checklist item associating a module always considers all installation profiles set to Done
  • Checklist item associating a module considers other modular dependencies set to Done

#23 Updated by ttereshc over 1 year ago

  • Checklist item deleted (associating a module adhers to the modular upgrade semantics)
  • Checklist item deleted (file a pulp-smash issue to automate testing recursive module copies)
  • Checklist item file a test issue to automate recursive copy of modules and their dependencies added
  • Description updated (diff)

#24 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#25 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#26 Updated by dalley over 1 year ago

  • Description updated (diff)

#27 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#28 Updated by milan over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 50 to 100

#29 Updated by rchan over 1 year ago

  • Sprint set to Sprint 47

Adding to Sprint 47 since this work was done during this time.

#30 Updated by rchan over 1 year ago

  • Sprint/Milestone set to 2.19.0

Add 2.19.0 milestone since this is a new feature

#32 Updated by amacdona@redhat.com over 1 year ago

  • Assignee set to milan

#33 Updated by bherring over 1 year ago

  • Copied to Test #4364: Implement modularity content dependency solving added

#34 Updated by bherring over 1 year ago

  • Checklist item file a test issue to automate recursive copy of modules and their dependencies set to Done

#35 Updated by bherring over 1 year ago

  • Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added

#36 Updated by ttereshc over 1 year ago

  • Platform Release set to 2.19.0

#37 Updated by ttereshc over 1 year ago

  • Status changed from MODIFIED to 5

#38 Updated by ttereshc over 1 year ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#39 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF