Project

Profile

Help

Story #3740

closed

Implement modularity content dependency solving

Added by milan over 6 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
2.19.0
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 47
Quarter:

Description

Motivation

Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.

Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).

Proposed solution

It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.

Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.

To support these usecases, a fake libsolv solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered bitmap from the libsolv library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.

References


Sub-issues 1 (0 open1 closed)

Task #3746: Investigate, whether exposing the @pool->considered@ bitmap is necessary for the modular dependencies to be resolvedCLOSED - COMPLETE

Actions

Related issues

Related to RPM Support - Story #4058: As a user, I can calculate applicability for modular contentCLOSED - CURRENTRELEASEttereshc

Actions
Related to RPM Support - Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependenciesCLOSED - CURRENTRELEASEmilanActions
Related to RPM Support - Story #4162: As a user, I have dependency solving when copying modulesCLOSED - CURRENTRELEASEdalley

Actions
Related to RPM Support - Test #4543: Test newly described definitions of recursive and recursive_conservative flagsCLOSED - COMPLETEbherringActions
Copied to RPM Support - Test #4364: Implement modularity content dependency solvingCLOSED - COMPLETEbherringActions
Actions #1

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #2

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #3

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #4

Updated by rchan over 6 years ago

  • Sprint/Milestone set to 2.17.0

Adding to 2.17.0 milestone. This is one of the required deliverables.

Actions #5

Updated by ipanova@redhat.com over 6 years ago

  • Sprint/Milestone deleted (2.17.0)
Actions #6

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #7

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #8

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #9

Updated by milan over 6 years ago

  • Description updated (diff)
Actions #10

Updated by ipanova@redhat.com over 6 years ago

inter-modular dependency support will lead to a model change of Modulemd. A new field should be added. called `dependencies`

  dependencies:
  - buildrequires:
      httpd: [2.4]
      platform: [el8]
    requires:
      httpd: [2.4]
      nginx: [1.14]
      platform: [el8]

This is an example of how the deps could look. As per spec [0], there might be multiple instances of Modulemd.Dependencies() object, where each of them will describe different way of build with varying build time and runtime.

During recursive copy we do not really care if those modules are required during build time or runtime, I suggest to make a union between `buildrequires` and `requires` per instance of Modulemd.Dependencies() object.

I suggest the following data structure, where each dict will be a translated Modulemd.Dependencies() object.


dependencies: [
                            {module-name1:  [stream1, stream2, stream3],
                             module-name2: [stream1, stream2]
                            },
                            {module-name2: [stream3],
                            module-name3: [stream1, stream2]
                            }
                    ]

Note** there might me inclusive and exclusive lists of stream, where exclusive is marked with a minus sign. We need to preserve the minus for later conflicts solving.

Note** the `platform` module is synthesized at runtime by dnf, we can ignore it and not include in the deps info at all.

[0] https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml#L78

Actions #11

Updated by milan over 6 years ago

A new field should be added. called `dependencies`

Another option could be a list of flat dictionaries like:

[
   {
     "name": "foo",
     "streams": ["stable"],
     "type": "runtime"
  },
  {
    "name": "bar",
    "streams": ["master"],
    "type": "build"
  },
  }
    "name": "foo",
    "streams": ["stable"],
    "type": "build"
  }
]

This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".

Actions #12

Updated by milan over 6 years ago

milan wrote:

A new field should be added. called `dependencies`

Another option could be a list of flat dictionaries like:

[...]

This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".

Actually, it won't work. It would loose the "inner build environments links" which exist so that build environments can have mutually exclusive dependencies and I guess the solver would freak out. Thanks Ina for pointing this out!

Actions #13

Updated by ipanova@redhat.com about 6 years ago

ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.

+1 to drop buildrequires and include in the proposed structure just the requires.

Actions #14

Updated by ipanova@redhat.com about 6 years ago

we should also consider possibly using https://github.com/fedora-modularity/fus

Actions #15

Updated by milan about 6 years ago

wrote:

ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.

+1 to drop buildrequires and include in the proposed structure just the requires.

Thing is, the consumers of our published content might not be able to rebuild the modules themselves i.e we might create an installable but un-buildable repo. This isn't the case for for ursine content now, though we don't process SRPM deps recursively either.

Fus on the other hand considers the build-time dependencies.

Actions #16

Updated by milan about 6 years ago

  • Related to Story #4058: As a user, I can calculate applicability for modular content added
Actions #17

Updated by ttereshc about 6 years ago

To be clear for a quick reader, the relation with #4058 is weak.
#4058 requires only info of the first level module dependencies, if they are enabled on a consumer or not.

List of dependencies (requires only, not buildrequires) is added to the Modulemd model with this commit: https://github.com/pulp/pulp_rpm/commit/990f65a9bba8e3f9faf2f89165f2a88015dad1c1

Actions #18

Updated by milan about 6 years ago

wrote:

we should also consider possibly using https://github.com/fedora-modularity/fus

There are some trade-offs with Fus to consider:

  • it is a command line tool that we'd have to integrate the way Pungi does
  • we'd have to semi-publish (all) the source and destination repositories metadata
  • this might have performance impact but we can afford experimenting with this idea

On the other hand, reimplementing some parts of both the Pungi and Fus tools seems like introducing code duplicity.
Still, the upside would be more control over how we resolve the dependencies.
There's an (undecided) Fus intgration e-mail discussion that can be queried for further information.

Actions #19

Updated by milan about 6 years ago

  • Related to Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies added
Actions #20

Updated by milan about 6 years ago

  • Related to Story #4162: As a user, I have dependency solving when copying modules added
Actions #21

Updated by milan almost 6 years ago

  • Status changed from NEW to POST
Actions #22

Updated by milan almost 6 years ago

Added by milan almost 6 years ago

Revision 62bb34a6 | View on GitHub

Introduce modular deps processing

Modulemd and modulemd_defaults are now exposed to the libsolv solver. This allows for the (inter-)modular dependencies to be processed when copying a unit. Bot the relaxed and conservative dependency processing algorithms work with the functionality this patch introduces.

Fixes #3740 https://pulp.plan.io/issues/3740

Actions #23

Updated by ttereshc almost 6 years ago

  • Description updated (diff)
Actions #24

Updated by ttereshc almost 6 years ago

  • Description updated (diff)
Actions #25

Updated by ttereshc almost 6 years ago

  • Description updated (diff)
Actions #26

Updated by dalley almost 6 years ago

  • Description updated (diff)
Actions #27

Updated by ttereshc almost 6 years ago

  • Description updated (diff)
Actions #28

Updated by milan almost 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 50 to 100
Actions #29

Updated by rchan almost 6 years ago

  • Sprint set to Sprint 47

Adding to Sprint 47 since this work was done during this time.

Actions #30

Updated by rchan almost 6 years ago

  • Sprint/Milestone set to 2.19.0

Add 2.19.0 milestone since this is a new feature

Actions #32

Updated by amacdona@redhat.com almost 6 years ago

  • Assignee set to milan
Actions #33

Updated by bherring almost 6 years ago

  • Copied to Test #4364: Implement modularity content dependency solving added
Actions #34

Updated by bherring almost 6 years ago

Actions #35

Updated by bherring almost 6 years ago

  • Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added
Actions #36

Updated by ttereshc almost 6 years ago

  • Platform Release set to 2.19.0
Actions #37

Updated by ttereshc over 5 years ago

  • Status changed from MODIFIED to 5
Actions #38

Updated by ttereshc over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #39

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF