Project

Profile

Help

Story #3847

Introduce weak dependency solving

Added by milan 12 months ago. Updated 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
% Done:

100%

Platform Release:
2.17.0
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
Yes
Sprint:
Sprint 41

Description

Pulp lacks resolving of RPM weak forward dependency Recommends during content association in case of e.g recursively copying RPMs between repositories, which by default is respected by dnf A content gap might therefore be induced on a consumer machine when installing from a repository containing just the recursive-copy calculated dependencies of an RPM unit. This isn't a breaking discrepancy as by definition, weak dependencies missing don't cause an unit to break

The very weak/hint Suggests field is out of scope as hints are by default not processed by dnf
Bot the weak and very weak/hint backward fields: Supplements and Enhances are out of scope too.

Publishing the weak dependencies in repository metadata is out of scope of this story as this is already supported trivially.

Implementation

Examples

I've found few Recommends field samples in the Fedora28--Workstation flavor primary.xml repodata file. The dnf unit recommends installing these two items, with rich dependency conditioning:

<rpm:recommends>
  <rpm:entry name="(/usr/bin/sqlite3 if bash-completion)"/>
  <rpm:entry name="(python3-dbus if NetworkManager)"/>
</rpm:recommends>

Notes

  • the RPM content upload code needn't be updated as it reuses the XML parsing code
  • the backwards dependencies needn't be processed because the Pulp workflow is closer to a repo closure calculation than to an actual content installation
  • the processing of the Recommends field is going to be handled thru libsolv once that PR lands
  • weak dependencies processing can be switched off in dnf; see also: man dnf.conf
  • very weak dependencies/hints are most likely just completely ignored by dnf
test.log (9.32 KB) milan, 07/13/2018 02:21 PM test.log

Checklist


Subtasks

Issue #3929: Pulp content search does not show Recommends fieldCLOSED - CURRENTRELEASEmilanActions

Related issues

Related to RPM Support - Story #3715: Reimplement unit dependency solving with the libsolv library CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision 6d6c8292 View on GitHub
Added by milan 11 months ago

Introduce the Recommends dependency

Pulp has been ignoring the Recommends weak dependency when processing e.g
recursive unit copies. This patch updates the RPM model to track a
recommends unit attribute and the primary.xml parser to populate this
attribute. The libsolv dependency solver is used to process this
attribute when calculating unit dependencies.

Migration: populate the recommends attribute

Walks over the rpm units, unzipping the primary metadata XML snippets,
parsing the recommends entries and populating the recommends unit
attributes accoridingly.

Fixes: #3847
https://pulp.plan.io/issues/3847

History

#1 Updated by milan 12 months ago

  • Description updated (diff)

#2 Updated by milan 11 months ago

  • Description updated (diff)

#3 Updated by milan 11 months ago

  • Description updated (diff)

#4 Updated by ipanova@redhat.com 11 months ago

  • Platform Release deleted (2.17.0)

#5 Updated by milan 11 months ago

  • Related to Story #3715: Reimplement unit dependency solving with the libsolv library added

#6 Updated by ipanova@redhat.com 11 months ago

since there is a change in the model, shall we require a migration?

#7 Updated by milan 11 months ago

Thanks for the check; to my best knowledge a migration is unnecessary, with the following reasoning:
  • by definition, nothing should break if (during an installation) a unit Recommends dependency is missing from a repository
  • Pulp has been ignoring the Recommends dependency so far
  • therefore implementing this story without a migration won't make things worse

The migration itself would have to process all the RPM unit XML records to be able to populate the Recommends attribute which might have adverse performance effects. The unit Recommends attribute will be populated for newly-uploaded or newly-synchronized units.

However, let's document this.

#8 Updated by milan 11 months ago

  • Description updated (diff)

#9 Updated by dkliban@redhat.com 11 months ago

Without a migration the user will get into a situation where the RPMs that were synced before 2.17.0 will not have the 'recommends' field set. The RPMs that are synced after 2.17.0 release will have 'recommends' set. When the user goes to copy RPMs recursively he will notice that some RPMs bring their weak deps with them and some RPMs don't bring their weak deps with them.

#10 Updated by milan 11 months ago

wrote:

Without a migration the user will get into a situation where the RPMs that were synced before 2.17.0 will not have the 'recommends' field set. The RPMs that are synced after 2.17.0 release will have 'recommends' set. When the user goes to copy RPMs recursively he will notice that some RPMs bring their weak deps with them and some RPMs don't bring their weak deps with them.

I agree but please mind this isn't a breaking discrepancy; we should be OK to just document (and test) this behaviour.

#11 Updated by milan 11 months ago

  • Description updated (diff)

#12 Updated by milan 11 months ago

  • Description updated (diff)

#13 Updated by milan 11 months ago

  • Description updated (diff)

#14 Updated by ipanova@redhat.com 11 months ago

thanks @dkliban, this is exact reason why i asked for a migration :)
As Milan pointed out, in the migration we'd need to get from the DB for each rpm unit the xml snippet, uncompress it, parse it.
On the other hand if we won't write the migration then those units before the upgrade will forever miss the the newly added field unless the unit will be removed and added back to the DB. Nothing would break without the migration, dnf will still be happy, just the behaviour during recursive copy will be inconsistent - for some units weak deps will will be copied over and for some not.

I think i am still in favour of writing the migration even if it is a costful one, let's minimize the inconsistent behaviour as much as we can.

#15 Updated by ipanova@redhat.com 11 months ago

i think this story is ready to be groomed once we +1 or -1 on the migration.

#16 Updated by dkliban@redhat.com 11 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#17 Updated by milan 11 months ago

  • File test.log test.log added
  • Groomed changed from Yes to No
  • Sprint Candidate changed from Yes to No

To give a concrete example:
Assuming we've got 2 repos; f27, f28 with content of Fedora27, Fedora28 respectively.
Given the fact RPM unit nevra is used as unit identity key and given the fact during a sync call (eve the --force-full one) the unit records aren't updated in Mongo, the discrepancy would occur either after having deleting a repo and cleaning orphans or if a newer unit version (as in nevra) was pulled in. In these cases, one dnf unit version would have recommends attribute populated, the other wouldn't:

> db.units_rpm.find({'name': 'dnf'})[0].recommends
[
        {
                "release" : null,
                "epoch" : null,
                "version" : null,
                "flags" : null,
                "name" : "python3-dbus" 
        }
]
> db.units_rpm.find({'name': 'dnf'})[0].release
1.fc27
> db.units_rpm.find({'name': 'dnf'})[1].recommends
> db.units_rpm.find({'name': 'dnf'})[1].release
12.fc28
>

See also my test.log attachment.

#18 Updated by milan 11 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#19 Updated by milan 11 months ago

  • Assignee set to milan

#20 Updated by milan 11 months ago

  • Status changed from NEW to POST

#21 Updated by dkliban@redhat.com 11 months ago

  • Sprint set to Sprint 40

#22 Updated by milan 11 months ago

  • Description updated (diff)

Speaking about inconsistency, there might be other point of view on the migration: after it, out of the blue, users recursively copying e.g dnf-2.7.3-1.fc27.noarch.rpm will start pulling in sqlite-3 as a recommendation. This change of behavior will happen without any update to the dnf package itself. This might be surprising to the user because they might not anticipate a change of dependencies of a package without the accompanying bump of the package evr. As a consequence, recursive copies of dnf-2.7.3-1.fc27.noarch.rpm before and after the migration would differ in content.

#23 Updated by milan 11 months ago

  • Smash Test set to 1107

#24 Updated by ttereshc 11 months ago

+1 to ipanova and dkliban point to have a migration for consistency reasons.
The fact that Pulp won't error without migration is not good enough reason not to do it, imho.

#25 Updated by milan 11 months ago

It's not about Pulp erroring out I'm afraid, it's about repositories having different content if they're created thru a recursive copy before and after the migration, no matter the content version being the same. This I find a bigger discrepancy than if all the non-updated rpm versions missed the recommends field (or just had the default empty list fwiw) because it's consistent with client behavior.

#26 Updated by ttereshc 11 months ago

I'm not sure I follow "the content version being the same" part.

If users upgrade to Pulp 2.x with weak deps support, they know that now weak deps are taken into account.
If they perform recursive copy, and the result will be different from the older version of Pulp that's fine because now weak deps are respected.
Published repository will have new revision number so clients will know that content was updated and is different now.
Let me know if I misunderstood you.

Ah before or after the migration. Migrations are an essential part of installation so we should not cover the case when they were not run

#28 Updated by rchan 11 months ago

  • Sprint changed from Sprint 40 to Sprint 41

#29 Updated by milan 11 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#30 Updated by rchan 11 months ago

  • Sprint/Milestone set to 2.17.0

#31 Updated by ipanova@redhat.com 11 months ago

  • Platform Release set to 2.17.0

#32 Updated by ipanova@redhat.com 11 months ago

  • Platform Release changed from 2.17.0 to master

#33 Updated by ipanova@redhat.com 11 months ago

  • Platform Release deleted (master)

#34 Updated by ipanova@redhat.com 11 months ago

  • Platform Release set to 2.17.0

#35 Updated by ipanova@redhat.com 10 months ago

  • Status changed from MODIFIED to ON_QA

#36 Updated by kersom 10 months ago

  • Verified changed from No to Yes

#37 Updated by kersom 10 months ago

This new feature was manually verified.

Pulp copies strong and weak dependencies when using --recursive during a copy.

#38 Updated by kersom 10 months ago

  • Verified changed from Yes to No

#39 Updated by milan 10 months ago

weak rich(boolean) deps are hosed:

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo copy rpm -f f28 -t foo --recursive --str-eq 'name=bash-completion'
This command may be exited via ctrl+c without affecting the request.

[/]
Running...

Copied:
  basesystem-11-5.fc28-noarch
  bash-4.4.19-2.fc28-x86_64
  bash-completion-2.7-4.fc28-noarch
  fedora-gpg-keys-28-1-noarch
  fedora-release-28-1-noarch
  fedora-repos-28-1-noarch
  filesystem-3.8-2.fc28-x86_64
  glibc-2.27-8.fc28-x86_64
  glibc-all-langpacks-2.27-8.fc28-x86_64
  glibc-common-2.27-8.fc28-x86_64
  libpkgconf-1.4.2-1.fc28-x86_64
  libselinux-2.7-13.fc28-x86_64
  libsepol-2.7-6.fc28-x86_64
  ncurses-base-6.1-4.20180224.fc28-noarch
  ncurses-libs-6.1-4.20180224.fc28-x86_64
  pcre2-10.31-4.fc28-x86_64
  pkgconf-1.4.2-1.fc28-x86_64
  pkgconf-m4-1.4.2-1.fc28-noarch
  pkgconf-pkg-config-1.4.2-1.fc28-x86_64
  setup-2.11.3-1.fc28-noarch
  tzdata-2018d-1.fc28-noarch

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo copy rpm -f f28 -t foo --recursive --str-eq 'name=dnf'
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
  rpm: 145

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo content rpm --repo-id f28 --str-eq 'name=sqlite'
Arch:         x86_64
Buildhost:    buildvm-25.phx2.fedoraproject.org
Checksum:     24ee748b29e4837c41df5b9472654f5aa66858f62cde706723d4f480b3f56fb6
Checksumtype: sha256
Description:  SQLite is a C library that implements an SQL database engine. A
              large subset of SQL92 is supported. A complete database is stored
              in a single disk file. The API is designed for convenience and
              ease of use. Applications that link against SQLite can enjoy the
              power and flexibility of an SQL database without the
              administrative hassles of supporting a separate database server.
              Version 2 and version 3 binaries are named to permit each to be
              installed on a single host
Epoch:        0
Filename:     sqlite-3.22.0-4.fc28.x86_64.rpm
License:      Public Domain
Name:         sqlite
Provides:     sqlite = 3.22.0-4.fc28-0, sqlite(x86-64) = 3.22.0-4.fc28-0
Release:      4.fc28
Requires:     libdl.so.2()(64bit), libdl.so.2(GLIBC_2.2.5)(64bit),
              libm.so.6()(64bit), libm.so.6(GLIBC_2.2.5)(64bit),
              libncurses.so.6()(64bit), libpthread.so.0()(64bit),
              libpthread.so.0(GLIBC_2.2.5)(64bit), libreadline.so.7()(64bit),
              libtinfo.so.6()(64bit), libz.so.1()(64bit),
              libz.so.1(ZLIB_1.2.0)(64bit), rtld(GNU_HASH), sqlite-libs =
              3.22.0-4.fc28-0, libc.so.6(GLIBC_2.14)(64bit)
Vendor:       Fedora Project
Version:      3.22.0

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo content rpm --repo-id foo --str-eq 'name=sqlite'
# nothing found
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"recommends": {$elemMatch: {"name": /^\(/}}})[0].name'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
dnf
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"recommends": {$elemMatch: {"name": /^\(/}}})[0].recommends[0]'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
{
        "release" : null,
        "epoch" : null,
        "version" : null,
        "flags" : null,
        "name" : "(/usr/bin/sqlite3 if bash-completion)" 
}
(pulp) [vagrant@pulp2 yum]$

This works as expected with dnf (on a F28 Workstation VM):

[root@localhost ~]# rpm -q sqlite
sqlite-3.22.0-4.fc28.x86_64
[root@localhost ~]# rpm -e sqlite
error: Failed dependencies:
    sqlite >= 3.22.0 is needed by (installed) firefox-59.0.2-1.fc28.x86_64
[root@localhost ~]# rpm -e firefox
# --->%--------------------------------------
warning: file /usr/lib64/firefox/langpacks: remove failed: No such file or directory
[root@localhost ~]# rpm -e sqlite
[root@localhost ~]# rpm -q bash-completion
bash-completion-2.7-4.fc28.noarch
[root@localhost ~]# rpm -q sqlite
package sqlite is not installed
[root@localhost ~]# dnf reinstall dnf
Last metadata expiration check: 0:28:47 ago on Tue 21 Aug 2018 08:43:20 PM CEST.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                  Arch                                     Version                                         Repository                                Size
==================================================================================================================================================================================
Reinstalling:
 dnf                                      noarch                                   2.7.5-12.fc28                                   fedora                                   352 k
Installing weak dependencies:
 sqlite                                   x86_64                                   3.22.0-4.fc28                                   fedora                                   623 k

Transaction Summary
==================================================================================================================================================================================
Install  1 Package

Total download size: 975 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): dnf-2.7.5-12.fc28.noarch.rpm                                                                                                               8.2 MB/s | 352 kB     00:00
(2/2): sqlite-3.22.0-4.fc28.x86_64.rpm                                                                                                             13 MB/s | 623 kB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                             1.3 MB/s | 975 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                          1/1
  Running scriptlet: sqlite-3.22.0-4.fc28.x86_64                                                                                                                              1/1
  Installing       : sqlite-3.22.0-4.fc28.x86_64                                                                                                                              1/3
  Reinstalling     : dnf-2.7.5-12.fc28.noarch                                                                                                                                 2/3
  Running scriptlet: dnf-2.7.5-12.fc28.noarch                                                                                                                                 2/3
  Running scriptlet: dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3
  Erasing          : dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3
  Running scriptlet: dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3
  Verifying        : dnf-2.7.5-12.fc28.noarch                                                                                                                                 1/3
  Verifying        : sqlite-3.22.0-4.fc28.x86_64                                                                                                                              2/3
  Verifying        : dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3

Reinstalled:
  dnf.noarch 2.7.5-12.fc28

Installed:
  sqlite.x86_64 3.22.0-4.fc28

Complete!
[root@localhost ~]# rpm -q sqlite
sqlite-3.22.0-4.fc28.x86_64
[root@localhost ~]#

#40 Updated by kersom 10 months ago

It was manual tested.

Following commands.

pulp-admin rpm repo create --repo-id foo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-richnweak-deps/
pulp-admin rpm repo create --repo-id bar
pulp-admin rpm repo sync run --repo-id foo
[root@f28c ~]# pulp-admin rpm repo copy rpm -f foo -t bar --recursive --str-eq 'name=Cobbler'
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
  Cobbler-1-0-noarch
  contireau-2-10-noarch
  icecubes-2-3-noarch
  orange-bits-2-3-noarch
  tablespoon-sugar-1-0-noarch

Comparing same dependency resolution using DNF.

cat >/etc/yum.repos.d/weak.repo <<EOL
[weak_repo_modi]
baseurl=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-richnweak-deps/
gpgcheck=0
name=test_repo
EOL
 kersom  /  etc  yum.repos.d  sudo dnf install Cobbler
test_repo                                                                                                                                                                           19 kB/s | 3.9 kB     00:00    
Last metadata expiration check: 0:00:00 ago on Thu 23 Aug 2018 12:59:56 PM EDT.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                  Arch                                           Version                                      Repository                                              Size
===================================================================================================================================================================================================================
Installing:
 Cobbler                                                  noarch                                         1-0                                          weak_repo_modi                                         6.2 k
Installing dependencies:
 contireau                                                noarch                                         2-10                                         weak_repo_modi                                         5.8 k
 icecubes                                                 noarch                                         2-3                                          weak_repo_modi                                         5.7 k
 tablespoon-sugar                                         noarch                                         1-0                                          weak_repo_modi                                         5.7 k
Installing weak dependencies:
 orange-bits                                              noarch                                         2-3                                          weak_repo_modi                                         5.9 k

Transaction Summary
===================================================================================================================================================================================================================
Install  5 Packages

Total download size: 29 k
Installed size: 0  
Is this ok [y/N]: 

#41 Updated by kersom 10 months ago

  • Verified changed from No to Yes

#42 Updated by ipanova@redhat.com 10 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#43 Updated by bmbouter 2 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF