Project

Profile

Help

Story #3740

Implement modularity content dependency solving

Added by milan over 1 year ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
2.19.0
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
Yes
Sprint:
Sprint 47

Description

Motivation

Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.

Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).

Proposed solution

It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.

Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.

To support these usecases, a fake libsolv solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered bitmap from the libsolv library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.

References


Checklist


Subtasks

External - Task #3741: File a PR against the libsolv Python bindings to expose the pool->considered bitmapCLOSED - COMPLETE

Actions
Task #3746: Investigate, whether exposing the @pool->considered@ bitmap is necessary for the modular dependencies to be resolvedCLOSED - COMPLETE

Actions

Related issues

Related to RPM Support - Story #4058: As a user, I can calculate applicability for modular content CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Story #4162: As a user, I have RPM dependency solving when copying modules NEW Actions
Related to RPM Support - Test #4543: Test newly described definitions of recursive and recursive_conservative flags CLOSED - COMPLETE Actions
Copied to RPM Support - Test #4364: Implement modularity content dependency solving CLOSED - COMPLETE Actions

Associated revisions

Revision 62bb34a6 View on GitHub
Added by milan 10 months ago

Introduce modular deps processing

Modulemd and modulemd_defaults are now exposed to the libsolv solver.
This allows for the (inter-)modular dependencies to be processed when
copying a unit. Bot the relaxed and conservative dependency processing
algorithms work with the functionality this patch introduces.

Fixes #3740
https://pulp.plan.io/issues/3740

History

#1 Updated by milan over 1 year ago

  • Description updated (diff)

#2 Updated by milan over 1 year ago

  • Description updated (diff)

#3 Updated by milan over 1 year ago

  • Description updated (diff)

#4 Updated by rchan over 1 year ago

  • Sprint/Milestone set to 2.17.0

Adding to 2.17.0 milestone. This is one of the required deliverables.

#5 Updated by ipanova@redhat.com about 1 year ago

  • Sprint/Milestone deleted (2.17.0)

#6 Updated by milan about 1 year ago

  • Description updated (diff)

#7 Updated by milan about 1 year ago

  • Description updated (diff)

#8 Updated by milan about 1 year ago

  • Description updated (diff)

#9 Updated by milan about 1 year ago

  • Description updated (diff)

#10 Updated by ipanova@redhat.com about 1 year ago

inter-modular dependency support will lead to a model change of Modulemd. A new field should be added. called `dependencies`

  dependencies:
  - buildrequires:
      httpd: [2.4]
      platform: [el8]
    requires:
      httpd: [2.4]
      nginx: [1.14]
      platform: [el8]

This is an example of how the deps could look. As per spec [0], there might be multiple instances of Modulemd.Dependencies() object, where each of them will describe different way of build with varying build time and runtime.

During recursive copy we do not really care if those modules are required during build time or runtime, I suggest to make a union between `buildrequires` and `requires` per instance of Modulemd.Dependencies() object.

I suggest the following data structure, where each dict will be a translated Modulemd.Dependencies() object.


dependencies: [
                            {module-name1:  [stream1, stream2, stream3],
                             module-name2: [stream1, stream2]
                            },
                            {module-name2: [stream3],
                            module-name3: [stream1, stream2]
                            }
                    ]

Note** there might me inclusive and exclusive lists of stream, where exclusive is marked with a minus sign. We need to preserve the minus for later conflicts solving.

Note** the `platform` module is synthesized at runtime by dnf, we can ignore it and not include in the deps info at all.

[0] https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml#L78

#11 Updated by milan about 1 year ago

A new field should be added. called `dependencies`

Another option could be a list of flat dictionaries like:

[
   {
     "name": "foo",
     "streams": ["stable"],
     "type": "runtime" 
  },
  {
    "name": "bar",
    "streams": ["master"],
    "type": "build" 
  },
  }
    "name": "foo",
    "streams": ["stable"],
    "type": "build" 
  }
]

This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".

#12 Updated by milan about 1 year ago

milan wrote:

A new field should be added. called `dependencies`

Another option could be a list of flat dictionaries like:

[...]

This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".

Actually, it won't work. It would loose the "inner build environments links" which exist so that build environments can have mutually exclusive dependencies and I guess the solver would freak out. Thanks Ina for pointing this out!

#13 Updated by ipanova@redhat.com about 1 year ago

@ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.

+1 to drop buildrequires and include in the proposed structure just the requires.

#14 Updated by ipanova@redhat.com about 1 year ago

we should also consider possibly using https://github.com/fedora-modularity/fus

#15 Updated by milan 11 months ago

wrote:

@ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.

+1 to drop buildrequires and include in the proposed structure just the requires.

Thing is, the consumers of our published content might not be able to rebuild the modules themselves i.e we might create an installable but un-buildable repo. This isn't the case for for ursine content now, though we don't process SRPM deps recursively either.

Fus on the other hand considers the build-time dependencies.

#16 Updated by milan 11 months ago

  • Related to Story #4058: As a user, I can calculate applicability for modular content added

#17 Updated by ttereshc 11 months ago

To be clear for a quick reader, the relation with #4058 is weak.
#4058 requires only info of the first level module dependencies, if they are enabled on a consumer or not.

List of dependencies (requires only, not buildrequires) is added to the Modulemd model with this commit: https://github.com/pulp/pulp_rpm/commit/990f65a9bba8e3f9faf2f89165f2a88015dad1c1

#18 Updated by milan 11 months ago

wrote:

we should also consider possibly using https://github.com/fedora-modularity/fus

There are some trade-offs with Fus to consider:

  • it is a command line tool that we'd have to integrate the way Pungi does
  • we'd have to semi-publish (all) the source and destination repositories metadata
  • this might have performance impact but we can afford experimenting with this idea

On the other hand, reimplementing some parts of both the Pungi and Fus tools seems like introducing code duplicity.
Still, the upside would be more control over how we resolve the dependencies.
There's an (undecided) Fus intgration e-mail discussion that can be queried for further information.

#19 Updated by milan 11 months ago

  • Related to Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies added

#20 Updated by milan 11 months ago

  • Related to Story #4162: As a user, I have RPM dependency solving when copying modules added

#21 Updated by milan 10 months ago

  • Status changed from NEW to POST

#22 Updated by milan 10 months ago

  • Checklist item associating a module content unit supports the recursive behavior set to Done
  • Checklist item associating a module always considers all installation profiles set to Done
  • Checklist item associating a module considers other modular dependencies set to Done

#23 Updated by ttereshc 9 months ago

  • Checklist item deleted (associating a module adhers to the modular upgrade semantics)
  • Checklist item deleted (file a pulp-smash issue to automate testing recursive module copies)
  • Checklist item file a test issue to automate recursive copy of modules and their dependencies added
  • Description updated (diff)

#24 Updated by ttereshc 9 months ago

  • Description updated (diff)

#25 Updated by ttereshc 9 months ago

  • Description updated (diff)

#26 Updated by dalley 9 months ago

  • Description updated (diff)

#27 Updated by ttereshc 9 months ago

  • Description updated (diff)

#28 Updated by milan 9 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 50 to 100

#29 Updated by rchan 9 months ago

  • Sprint set to Sprint 47

Adding to Sprint 47 since this work was done during this time.

#30 Updated by rchan 9 months ago

  • Sprint/Milestone set to 2.19.0

Add 2.19.0 milestone since this is a new feature

#31 Updated by daviddavis 9 months ago

  • Verification Required changed from No to Yes

#32 Updated by amacdona@redhat.com 9 months ago

  • Assignee set to milan

#33 Updated by bherring 9 months ago

  • Copied to Test #4364: Implement modularity content dependency solving added

#34 Updated by bherring 9 months ago

  • Checklist item file a test issue to automate recursive copy of modules and their dependencies set to Done

#35 Updated by bherring 7 months ago

  • Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added

#36 Updated by ttereshc 7 months ago

  • Platform Release set to 2.19.0

#37 Updated by ttereshc 7 months ago

  • Status changed from MODIFIED to ON_QA

#38 Updated by ttereshc 7 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#39 Updated by bmbouter 6 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF