Project

Profile

Help

Issue #4152

Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies

Added by gmbnomis about 2 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.17.1
Platform Release:
2.18.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 46
Quarter:

Description

Problem

In contrast to e.g. Pulp 2.15.3, recursive copy in 2.17 seems to ignore dependencies that are not fully resolvable.

Packages often haven dependencies to OS packages (like "bash"). This means that "recursive" only works if the repository actually contains these system dependencies. In our use case, this is not the case (we build application repositories from a "union" of multiple repos using recursive copy).

Reproducer

pulp-admin rpm repo create --repo-id pulp_2_17_stable --feed https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.17/7/x86_64/

pulp-admin rpm repo sync run --repo-id pulp_2_17_stable

pulp-admin rpm repo create --repo-id test 

pulp-admin rpm repo copy rpm --from-repo-id pulp_2_17_stable --to-repo-id test --recursive --str-eq=name=pulp-server

On Pulp 2.15.3, this yields:

Copied:
  pulp-selinux-2.17.1-1.el7-noarch
  pulp-server-2.17.1-1.el7-noarch
  python-bson-3.2-2.el7-x86_64
  python-gofer-2.12.1-1.el7-noarch
  python-isodate-0.5.0-4.pulp.el7-noarch
  python-mongoengine-0.10.5-1.el7-noarch
  python-nectar-1.5.6-1.el7-noarch
  python-pulp-common-2.17.1-1.el7-noarch
  python-pulp-repoauth-2.17.1-1.el7-noarch
  python-pymongo-3.2-2.el7-x86_64
  python-pymongo-gridfs-3.2-2.el7-x86_64

On Pulp 2.17:

Copied:
  pulp-server-2.17.1-1.el7-noarch

Additionally, it logs:

pulp_rpm.plugins.importers.yum.pulp_solv:WARNING: [f0572afa] Encountered problems solving: nothing provides /bin/bash needed by pulp-server-0:2.17.1-1.el7.noarch


Related issues

Related to RPM Support - Story #3740: Implement modularity content dependency solvingCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Task #3528: Investigate which library to use for dep solvingCLOSED - COMPLETE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Test #4269: Recursive and conservative recursive copyCLOSED - COMPLETE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Task #4371: Document how to use the newly added recursive_conservativeCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision cb4d33c5 View on GitHub
Added by milan almost 2 years ago

Fake missing dependencies with dummy solvables

This patch allows automatically retrying the dependency solving during recursive copy if missing dependencies were detected.

Opt-in for the conservative recursive copy

This allows one to switch between a conservative and a relaxed (compatible) recursive copy of units (the default). To use the conservative recursive copy, add the "conservative_recursive_copy": true tag into the "override_config" dictionary of the REST API request body.

Fixes: #4152 https://pulp.plan.io/issues/4152

Revision 740dff16 View on GitHub
Added by milan almost 2 years ago

Fake missing dependencies with dummy solvables

This patch allows automatically retrying the dependency solving during recursive copy if missing dependencies were detected.

Opt-in for the conservative recursive copy

This allows one to switch between a conservative and a relaxed (compatible) recursive copy of units (the default). To use the conservative recursive copy, add the "conservative_recursive_copy": true tag into the "override_config" dictionary of the REST API request body.

Fixes: #4152 https://pulp.plan.io/issues/4152

History

#1 Updated by CodeHeeler about 2 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 45

#2 Updated by milan about 2 years ago

Pulp (2.17) now uses libsolv as the solver to calculate dependencies when recursively copying units.
This solver is a yes--no solver rather than the best-effort solver we used to have before.
The motivation for the change is discussed on the Issue #3528.

The cleanest solution would be to include multiple repositories in the dependencies calculation, i.e as gmbnomis put it: "given source repo A and target repo B, consider the union of the B, C1, C2... repos as 'installed', where C1, C2... are the repos that would be enabled on the consumer system during DNF installation." Any dependency-solving failures would ideally be propagated back thru the recursive copy tasks result too. This solution would however imply both REST API changes as well as the recursive copy workflow change so it might not be the most welcomed idea for our users. Based on some early investigation of modular depsolving (downstream product implications), similar approach might have to be reused in Issue #3740: given the fact multiple source repositories have to be considered for modular content to resolve, multiple target repositories have to be considered, but the reasons for the multiple target repositories are still unclear to me now.

Another option might be to "fake" missing dependencies with dummy solvables providing those dependencies like DummySolvable(name='/bin/bash', fileprovides=['/bin/bash'], evr='', arch='noarch').
The solver might need to retry solving (multiple times?) before the units can be copied but the solver behavior would remain backwards-compatible which could be preferred by users. This option might be even considered in case of modular depsolving because modular content is kept separate from its dependencies in OS repo content, so the modular repos are "solo-uninstallable" by design.

#3 Updated by milan about 2 years ago

  • Related to Story #3740: Implement modularity content dependency solving added

#4 Updated by milan about 2 years ago

  • Related to Task #3528: Investigate which library to use for dep solving added

#5 Updated by milan about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan

#6 Updated by milan about 2 years ago

  • Status changed from ASSIGNED to POST

Just pushed the first take on it; https://github.com/pulp/pulp_rpm/pull/1226
gmbnomis, mind taking a look?

Thanks!
milan

#7 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 45 to Sprint 46

#8 Updated by milan almost 2 years ago

  • Status changed from POST to MODIFIED

#9 Updated by kersom almost 2 years ago

  • Related to Test #4269: Recursive and conservative recursive copy added

#10 Updated by daviddavis almost 2 years ago

  • Platform Release set to 2.18.1

#11 Updated by milan almost 2 years ago

#12 Updated by kersom almost 2 years ago

  • Related to Task #4371: Document how to use the newly added recursive_conservative added

#13 Updated by jortel@redhat.com almost 2 years ago

  • Status changed from MODIFIED to 5

#14 Updated by jortel@redhat.com almost 2 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF