Project

Profile

Help

Issue #4152

Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies

Added by gmbnomis 7 months ago. Updated 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Severity:
2. Medium
Version:
2.17.1
Platform Release:
2.18.1
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 46

Description

Problem

In contrast to e.g. Pulp 2.15.3, recursive copy in 2.17 seems to ignore dependencies that are not fully resolvable.

Packages often haven dependencies to OS packages (like "bash"). This means that "recursive" only works if the repository actually contains these system dependencies. In our use case, this is not the case (we build application repositories from a "union" of multiple repos using recursive copy).

Reproducer

pulp-admin rpm repo create --repo-id pulp_2_17_stable --feed https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.17/7/x86_64/

pulp-admin rpm repo sync run --repo-id pulp_2_17_stable

pulp-admin rpm repo create --repo-id test 

pulp-admin rpm repo copy rpm --from-repo-id pulp_2_17_stable --to-repo-id test --recursive --str-eq=name=pulp-server

On Pulp 2.15.3, this yields:

Copied:
  pulp-selinux-2.17.1-1.el7-noarch
  pulp-server-2.17.1-1.el7-noarch
  python-bson-3.2-2.el7-x86_64
  python-gofer-2.12.1-1.el7-noarch
  python-isodate-0.5.0-4.pulp.el7-noarch
  python-mongoengine-0.10.5-1.el7-noarch
  python-nectar-1.5.6-1.el7-noarch
  python-pulp-common-2.17.1-1.el7-noarch
  python-pulp-repoauth-2.17.1-1.el7-noarch
  python-pymongo-3.2-2.el7-x86_64
  python-pymongo-gridfs-3.2-2.el7-x86_64

On Pulp 2.17:

Copied:
  pulp-server-2.17.1-1.el7-noarch

Additionally, it logs:

pulp_rpm.plugins.importers.yum.pulp_solv:WARNING: [f0572afa] Encountered problems solving: nothing provides /bin/bash needed by pulp-server-0:2.17.1-1.el7.noarch


Related issues

Related to RPM Support - Story #3740: Implement modularity content dependency solving CLOSED - CURRENTRELEASE Actions
Related to RPM Support - Task #3528: Investigate which library to use for dep solving CLOSED - COMPLETE Actions
Related to RPM Support - Test #4269: Recursive and conservative recursive copy CLOSED - COMPLETE Actions
Related to RPM Support - Task #4371: Document how to use the newly added recursive_conservative CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision cb4d33c5 View on GitHub
Added by milan 6 months ago

Fake missing dependencies with dummy solvables

This patch allows automatically retrying the dependency solving during
recursive copy if missing dependencies were detected.

Opt-in for the conservative recursive copy

This allows one to switch between a conservative and a relaxed
(compatible) recursive copy of units (the default).
To use the conservative recursive copy, add the
`"conservative_recursive_copy": true` tag into the `"override_config"`
dictionary of the REST API request body.

Fixes: #4152
https://pulp.plan.io/issues/4152

Revision 740dff16 View on GitHub
Added by milan 5 months ago

Fake missing dependencies with dummy solvables

This patch allows automatically retrying the dependency solving during
recursive copy if missing dependencies were detected.

Opt-in for the conservative recursive copy

This allows one to switch between a conservative and a relaxed
(compatible) recursive copy of units (the default).
To use the conservative recursive copy, add the
`"conservative_recursive_copy": true` tag into the `"override_config"`
dictionary of the REST API request body.

Fixes: #4152
https://pulp.plan.io/issues/4152

History

#1 Updated by CodeHeeler 7 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 45

#2 Updated by milan 7 months ago

Pulp (2.17) now uses libsolv as the solver to calculate dependencies when recursively copying units.
This solver is a yes--no solver rather than the best-effort solver we used to have before.
The motivation for the change is discussed on the Issue #3528.

The cleanest solution would be to include multiple repositories in the dependencies calculation, i.e as gmbnomis put it: "given source repo A and target repo B, consider the union of the B, C1, C2... repos as 'installed', where C1, C2... are the repos that would be enabled on the consumer system during DNF installation." Any dependency-solving failures would ideally be propagated back thru the recursive copy tasks result too. This solution would however imply both REST API changes as well as the recursive copy workflow change so it might not be the most welcomed idea for our users. Based on some early investigation of modular depsolving (downstream product implications), similar approach might have to be reused in Issue #3740: given the fact multiple source repositories have to be considered for modular content to resolve, multiple target repositories have to be considered, but the reasons for the multiple target repositories are still unclear to me now.

Another option might be to "fake" missing dependencies with dummy solvables providing those dependencies like DummySolvable(name='/bin/bash', fileprovides=['/bin/bash'], evr='', arch='noarch').
The solver might need to retry solving (multiple times?) before the units can be copied but the solver behavior would remain backwards-compatible which could be preferred by users. This option might be even considered in case of modular depsolving because modular content is kept separate from its dependencies in OS repo content, so the modular repos are "solo-uninstallable" by design.

#3 Updated by milan 7 months ago

  • Related to Story #3740: Implement modularity content dependency solving added

#4 Updated by milan 7 months ago

  • Related to Task #3528: Investigate which library to use for dep solving added

#5 Updated by milan 7 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan

#6 Updated by milan 7 months ago

  • Status changed from ASSIGNED to POST

Just pushed the first take on it; https://github.com/pulp/pulp_rpm/pull/1226
gmbnomis, mind taking a look?

Thanks!
milan

#7 Updated by rchan 7 months ago

  • Sprint changed from Sprint 45 to Sprint 46

#8 Updated by milan 6 months ago

  • Status changed from POST to MODIFIED

#9 Updated by kersom 6 months ago

  • Related to Test #4269: Recursive and conservative recursive copy added

#10 Updated by daviddavis 5 months ago

  • Platform Release set to 2.18.1

#11 Updated by milan 5 months ago

#12 Updated by kersom 5 months ago

  • Related to Task #4371: Document how to use the newly added recursive_conservative added

#13 Updated by jortel@redhat.com 4 months ago

  • Status changed from MODIFIED to ON_QA

#14 Updated by jortel@redhat.com 4 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter 2 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF