Story #3740
closedImplement modularity content dependency solving
100%
Description
Motivation¶
Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.
Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).
Proposed solution¶
It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.
Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.
To support these usecases, a fake libsolv
solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered
bitmap from the libsolv
library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.
References¶
- the base pulp_modularity support tracker; Issue #3206
- the Modularity project docs page
- the depsolving library investigation #3528 POC
- libsolv
- modular errata
- A cross-team, modular-depsolving document
Related issues
Updated by rchan over 6 years ago
- Sprint/Milestone set to 2.17.0
Adding to 2.17.0 milestone. This is one of the required deliverables.
Updated by ipanova@redhat.com over 6 years ago
- Sprint/Milestone deleted (
2.17.0)
Updated by ipanova@redhat.com over 6 years ago
inter-modular dependency support will lead to a model change of Modulemd. A new field should be added. called `dependencies`
dependencies:
- buildrequires:
httpd: [2.4]
platform: [el8]
requires:
httpd: [2.4]
nginx: [1.14]
platform: [el8]
This is an example of how the deps could look. As per spec [0], there might be multiple instances of Modulemd.Dependencies() object, where each of them will describe different way of build with varying build time and runtime.
During recursive copy we do not really care if those modules are required during build time or runtime, I suggest to make a union between `buildrequires` and `requires` per instance of Modulemd.Dependencies() object.
I suggest the following data structure, where each dict will be a translated Modulemd.Dependencies() object.
dependencies: [
{module-name1: [stream1, stream2, stream3],
module-name2: [stream1, stream2]
},
{module-name2: [stream3],
module-name3: [stream1, stream2]
}
]
Note** there might me inclusive and exclusive lists of stream, where exclusive is marked with a minus sign. We need to preserve the minus for later conflicts solving.
Note** the `platform` module is synthesized at runtime by dnf, we can ignore it and not include in the deps info at all.
[0] https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml#L78
Updated by milan over 6 years ago
A new field should be added. called `dependencies`
Another option could be a list of flat dictionaries like:
[
{
"name": "foo",
"streams": ["stable"],
"type": "runtime"
},
{
"name": "bar",
"streams": ["master"],
"type": "build"
},
}
"name": "foo",
"streams": ["stable"],
"type": "build"
}
]
This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".
Updated by milan over 6 years ago
milan wrote:
A new field should be added. called `dependencies`
Another option could be a list of flat dictionaries like:
[...]
This would allow to keep track of the build- vs. run-time dependencies. Plus they say "flat's better than nested".
Actually, it won't work. It would loose the "inner build environments links" which exist so that build environments can have mutually exclusive dependencies and I guess the solver would freak out. Thanks Ina for pointing this out!
Updated by ipanova@redhat.com about 6 years ago
ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.
+1 to drop buildrequires and include in the proposed structure just the requires.
Updated by ipanova@redhat.com about 6 years ago
we should also consider possibly using https://github.com/fedora-modularity/fus
Updated by milan about 6 years ago
ipanova@redhat.com wrote:
ttereshc, now that i look at my comment which i wrote month ago, i cannot recall a reasonable explanation why we would care about buildrequires, unless we build modules which we do not.
+1 to drop buildrequires and include in the proposed structure just the requires.
Thing is, the consumers of our published content might not be able to rebuild the modules themselves i.e we might create an installable but un-buildable repo. This isn't the case for for ursine content now, though we don't process SRPM deps recursively either.
Fus on the other hand considers the build-time dependencies.
Updated by milan about 6 years ago
- Related to Story #4058: As a user, I can calculate applicability for modular content added
Updated by ttereshc about 6 years ago
To be clear for a quick reader, the relation with #4058 is weak.
#4058 requires only info of the first level module dependencies, if they are enabled on a consumer or not.
List of dependencies (requires only, not buildrequires) is added to the Modulemd model with this commit: https://github.com/pulp/pulp_rpm/commit/990f65a9bba8e3f9faf2f89165f2a88015dad1c1
Updated by milan about 6 years ago
ipanova@redhat.com wrote:
we should also consider possibly using https://github.com/fedora-modularity/fus
There are some trade-offs with Fus to consider:
- it is a command line tool that we'd have to integrate the way Pungi does
- we'd have to semi-publish (all) the source and destination repositories metadata
- this might have performance impact but we can afford experimenting with this idea
On the other hand, reimplementing some parts of both the Pungi and Fus tools seems like introducing code duplicity.
Still, the upside would be more control over how we resolve the dependencies.
There's an (undecided) Fus intgration e-mail discussion that can be queried for further information.
Updated by milan about 6 years ago
- Related to Issue #4152: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies added
Updated by milan about 6 years ago
- Related to Story #4162: As a user, I have dependency solving when copying modules added
Updated by milan almost 6 years ago
- Status changed from NEW to POST
Linking the PR https://github.com/pulp/pulp_rpm/pull/1237
Added by milan almost 6 years ago
Updated by milan almost 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 50 to 100
Applied in changeset 62bb34a6f1290db6f07c372c1a0c3ee9702c3e91.
Updated by rchan almost 6 years ago
- Sprint set to Sprint 47
Adding to Sprint 47 since this work was done during this time.
Updated by rchan almost 6 years ago
- Sprint/Milestone set to 2.19.0
Add 2.19.0 milestone since this is a new feature
Updated by bherring almost 6 years ago
- Copied to Test #4364: Implement modularity content dependency solving added
Updated by bherring almost 6 years ago
- Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added
Updated by ttereshc over 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Introduce modular deps processing
Modulemd and modulemd_defaults are now exposed to the libsolv solver. This allows for the (inter-)modular dependencies to be processed when copying a unit. Bot the relaxed and conservative dependency processing algorithms work with the functionality this patch introduces.
Fixes #3740 https://pulp.plan.io/issues/3740