RPM Support

I'd like to propose using libsolv directly rather than thru libdnf, with the following reasoning

• libdnf is exposing just the hawkey API which in turn supports loading repository data only thru reading the repo metadata
• Pulp would have to dump the MongoDB records into XML metadata, consumable by libdnf, to update its pool.repo data while at the same time, updating pool.repo data with libsolv can be done directly and iteratively
• a selective, recursive copy of a unit needs checking what unit.requires are missing from the target repository (being built)
• this implies building the solver target data pool.repo on the fly, updating the pool record of each unit of the target repository
• the solver might need to perform couple of iterations to support copying of mutually-conflicting packages into the same target repository
• therefore a low-level interaction with the solver seems a better fit
• libdnf is an RPM-specific implementation that would limit reuse e.g in the Debian plugin
• libdnf might bring in more features than necessary, Pulp has a narrowed-down set of usecases for which it needs fine-grain control (see Comment 10)
• libdnf is a work-in-progress

Pulp depending on libdnf would also mean having to synchronize another project's release cycle

On the other hand, there's the open question of supporting depsolving for Modularity, which doesn't seem to be the case in either of the libraries but libdnf seems to have some minimal support PR posted

Problem statement¶

Given source and target repositories, solve what dependencies copying a unit from the source to the target repo induces:

+------------------------------------------+
| MONGO                                    |
|                foo-1.1                   |
|                 - bar                    |
|  +--------+     - baz        +--------+  |
|  | Source |                  | Target |  |
|  |  repo  |  +------------>  |  repo  |  |
|  |        |                  |        |  |
|  +---+----+                  +----+---+  |
|      |                            |      |
+------------------------------------------+
       |                            |
       |                            |
       ^                            ^
+------+------+              +------+------+
| Solv Pool   |              | Solv Pool   |
|             |              |             |
| +---------+ |              | +---------+ |
| |repo:    | |              | |repo:    | |
| | foo,1.1 | |              | | bar,1.0 | |
| | bar,0.9 | |              | +---------+ |
| | baz,1.0 | |              |             |
| +---------+ |              +-------------+
|             |
+-------------+

Note please, the target repo already contains bar, newer than the source does.
In the context of https://pulp.plan.io/issues/2478 one would expect, if dependencies of foo allow it, the bar to be kept at the target version:

+------------------------------------------+
| MONGO                                    |
|                                          |
|                                          |
|  +--------+                  +--------+  |
|  | Source |                  | Target |  |
|  |  repo  |                  |  repo  |  |
|  |        |                  |        |  |
|  +---+----+                  +----+---+  |
|      |                            |      |
+------------------------------------------+
       |                            |
       |                            |
       ^                            ^
+------+------+              +------+------+
| Solv Pool   |              | Solv Pool   |
|             |              |             |
| +---------+ |              | +---------+ |
| |repo:    | |              | |repo:    | |
| | foo,1.1 | |              | | bar,1.0 | |
| | bar,0.9 | |              | | foo,1.1 | |
| | baz,1.0 | |              | | baz,1.0 | |
| +---------+ |              | +---------+ |
|             |              |             |
+-------------+              +-------------+

Proposed solution¶

The libsolv sat solver can be used to infer steps required to copy a unit; one can request libsolv to "install" foo.
Putting the target libsolv repo into the source libsolv pool allows one to solve an optimal repo transaction log, considering latest available units:

+------------------------------------------+
| MONGO                                    |
|                foo-1.1                   |
|  +--------+     - baz,1.0    +--------+  |
|  | Source |                  | Target |  |
|  |  repo  |  +------------>  |  repo  |  |
|  |        |                  |        |  |
|  +---+----+                  +----+---+  |
|      |                            |      |
+------------------------------------------+
       |                            |
       |                            |
       ^                            ^
+------+------+              +------+------+
| Solv Pool   |              | Solv Pool   |
|             |              |             |
| +---------+ |              | +---------+ |
| |repo:    | |              | |repo:    | |
| | foo,1.1 | |              | | bar,1.0 | |
| | bar,0.9 | |              | +----+----+ |
| | baz,1.0 | |              |      |      |
| +---------+ |              +-------------+
|             |                     |
| +---------+ |                     |
| |@System: | |                     |
| | bar,1.0 <-----------------------+
| +---------+ |
+-------------+

There might be cases when multiple packages with mutually-conflicting dependencies are being requested to be "installed" by the solver.
To address this, the solver can be configured with allow_uninstall flag to give a successful transaction in such a case; the uninstall transaction steps would be ignored.
Pulling-in weak dependencies can be configured similarly and setting repository priorities will allow more fine-grained control over the solving process.
It might make sense to expose all these configuration options to the user thru the REST API.

Implementation considerations¶

The pool.repo objects don't support sharing between pool objects; one actually has to copy the units between repos one by one.
This might take significant amount of time and might require some fine-grained state keeping around the pool objects, should the implementation decide to cache those on the disk.

For the same reason, the https://pulp.plan.io/issues/2478 behavior might be optional (exposed as another copy option thru the REST API) to save resources.

Furthermore the "smaller" repo would ideally be picked to be "copied" into a pool; since nothing is ever actually installed, the pool is just a context. Repository priorities would most likely have to be reversed in that case.

In unit dependency conflict cases, the solver might require multiple iterations to resolve the "install" jobs. The implementation should consider this possibility and evaluate the jobs in a stabilization fashion; any outstanding job requests would imply unit dependency bugs and some users might be interested in being able to see those.

Both the source and target repos will most likely have to be locked during the copy task run. Out of sync solver pools&repos might happen in case cached pools are being used and the copy task has been interrupted: there might be inconsistencies between Mongo and the solver pools.

Thinking out loud about the requirements from the description section.... I think overall a sat solver isn't a good fit for what the requirements are; the main theme of a sat solver being to give a yes (no) answer with a (counter-)proof i.e steps how (not) to install a package, all the requirements here imply some adjusting of the solver for a task it wasn't really meant to be used for.

Functionality of a solver we are looking for:

list dependencies for a given rpm package (can be a part of a synchronous task)

A solver works in a context of all available packages --- a content pool. Considering the transitive nature of content unit dependencies, to give an answer, the solver has to work in the context of all available packages from all relevant repos. The solver then returns a list of jobs to fulfill to successfully install a content unit, which may in turn contain removals of obsoleted, outdated or conflicting units.
In an empty pool, the list of dependencies of a package equals to the top-level Requires field of the package but it most certainly misses dependencies of the dependencies. Unless the Requires field dependencies are all leafs in the dependency tree of the package.
Thus the answer is context-dependent and blurred by various content conflicts but maybe the question itself is the same.

list fulfilled dependencies (from a given list of available packages) for a given rpm package

In the context of a (multi-)source--target repository copy task, to give an answer, the sat solver has to be tricked to consider the target repository content to be already installed. This can be achieved by copying the target repo into the source repo Pool and marking all the units installed. An answer to an install request will then give a list of units to install. When compared to the list of requirements (from the previous bullet), one can obtain list of fulfilled dependencies. However the conflicts might blur the picture again.

validate that all the dependencies (from a given list of available packages) for a given rpm package are fulfilled.

Similar to the previous bullet; asserting the list of packages to install to be empty. Mind the conflicts though.

perform dep solving against multiple repositories

This is naturally supported by both the solvers; a pool can have multiple repos.

Note please that all the answers can be fine-tuned (and changed) by setting various solver options and/or the priorities of the pool repos.
Maybe stating the justification of the requirements could give more context to understand why those are needed and figure out how (better) to address them.

Random POC Notes¶

• not calling pool.setarch() causes a segfault (see also the note 11 above). libsolv requires setting an architecture which might be an issue as our repos can contain units with different architectures. libdnf being build on top of libsolv most likely exposes the same behavior.
• pool.parsermprichdep is needed to parse rich dependencies. Fedora26 however didn't ship with libsolv >=0.6.33 so I had to workaround on my dev box; for demo purposes, one can use e.g ignatenkobrain's copr repo
• The files attribute of an rpm unit isn't processed yet but should be trivial to add.
• Just the rpm unit type supported; drpm, srpm units should be trivial to add
• me not knowing what the errata type depsolving should look like I feel it might be a challenge to implement; we might consider evaluating this
• to support recursive copying of particular module dependencies while maintaining the user-expected module vs system upgrade workflow semantics we'd need to expose the pool->considered C-API function to Python. This function operates over a custom-sized bitmap vector of enabled--disabled content units that we'd have to implement an adapter for, between the Python and C runtimes. This would be a focused patch against the upstream libsolv repo
• libsolv exposes a decent unit/solvable lookup interface, that supports e.g the globing syntax

TODO¶

Yum content types not covered yet:¶

• Erratum (easy) POC
• srpm (easy) POC
• drpm (tricky) might not be needed
• package category (trivial), a set of groups
• package group (trivial), a set of (RPM) packages

Yum features not covered yet:¶

• weak dependencies (done in POC)
• file-provides (tricky)

Modularity/App-streams support:¶

• expose pool->considered to Python bindings (tricky)
• module as a solvable (dunno, probably tricky)