Story #3847
closed
Introduce weak dependency solving
Added by milan over 6 years ago. Updated almost 3 years ago.
100%
Description
Pulp lacks resolving of RPM weak forward dependency Recommends during content association in case of e.g recursively copying RPMs between repositories, which by default is respected by dnf A content gap might therefore be induced on a consumer machine when installing from a repository containing just the recursive-copy calculated dependencies of an RPM unit. This isn't a breaking discrepancy as by definition, weak dependencies missing don't cause an unit to break
The very weak/hint Suggests field is out of scope as hints are by default not processed by dnf
Bot the weak and very weak/hint backward fields: Supplements and Enhances are out of scope too.
Publishing the weak dependencies in repository metadata is out of scope of this story as this is already supported trivially.
Implementation¶
- the RPM model needs to track the following "forward" weak dependency field
Recommends - the primary XML repo metadata parsing code has to be updated in order to populate the
Recommendsfield
Examples¶
I've found few Recommends field samples in the Fedora28--Workstation flavor primary.xml repodata file. The dnf unit recommends installing these two items, with rich dependency conditioning:
<rpm:recommends>
<rpm:entry name="(/usr/bin/sqlite3 if bash-completion)"/>
<rpm:entry name="(python3-dbus if NetworkManager)"/>
</rpm:recommends>
Notes¶
- the RPM content upload code needn't be updated as it reuses the XML parsing code
- the backwards dependencies needn't be processed because the Pulp workflow is closer to a repo closure calculation than to an actual content installation
- the processing of the
Recommendsfield is going to be handled thrulibsolvonce that PR lands - weak dependencies processing can be switched off in
dnf; see also:man dnf.conf - very weak dependencies/hints are most likely just completely ignored by
dnf
Files
Related issues
Updated by ipanova@redhat.com over 6 years ago
- Platform Release deleted (
2.17.0)
Updated by milan over 6 years ago
- Related to Story #3715: Reimplement unit dependency solving with the libsolv library added
Updated by ipanova@redhat.com over 6 years ago
since there is a change in the model, shall we require a migration?
Updated by milan over 6 years ago
Thanks for the check; to my best knowledge a migration is unnecessary, with the following reasoning:
- by definition, nothing should break if (during an installation) a unit
Recommendsdependency is missing from a repository - Pulp has been ignoring the
Recommendsdependency so far - therefore implementing this story without a migration won't make things worse
The migration itself would have to process all the RPM unit XML records to be able to populate the Recommends attribute which might have adverse performance effects. The unit Recommends attribute will be populated for newly-uploaded or newly-synchronized units.
However, let's document this.
Updated by dkliban@redhat.com over 6 years ago
Without a migration the user will get into a situation where the RPMs that were synced before 2.17.0 will not have the 'recommends' field set. The RPMs that are synced after 2.17.0 release will have 'recommends' set. When the user goes to copy RPMs recursively he will notice that some RPMs bring their weak deps with them and some RPMs don't bring their weak deps with them.
Updated by milan over 6 years ago
dkliban@redhat.com wrote:
Without a migration the user will get into a situation where the RPMs that were synced before 2.17.0 will not have the 'recommends' field set. The RPMs that are synced after 2.17.0 release will have 'recommends' set. When the user goes to copy RPMs recursively he will notice that some RPMs bring their weak deps with them and some RPMs don't bring their weak deps with them.
I agree but please mind this isn't a breaking discrepancy; we should be OK to just document (and test) this behaviour.
Updated by ipanova@redhat.com over 6 years ago
thanks @dkliban, this is exact reason why i asked for a migration :)
As Milan pointed out, in the migration we'd need to get from the DB for each rpm unit the xml snippet, uncompress it, parse it.
On the other hand if we won't write the migration then those units before the upgrade will forever miss the the newly added field unless the unit will be removed and added back to the DB. Nothing would break without the migration, dnf will still be happy, just the behaviour during recursive copy will be inconsistent - for some units weak deps will will be copied over and for some not.
I think i am still in favour of writing the migration even if it is a costful one, let's minimize the inconsistent behaviour as much as we can.
Updated by ipanova@redhat.com over 6 years ago
i think this story is ready to be groomed once we +1 or -1 on the migration.
Updated by dkliban@redhat.com over 6 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by milan over 6 years ago
To give a concrete example:
Assuming we've got 2 repos; f27, f28 with content of Fedora27, Fedora28 respectively.
Given the fact RPM unit nevra is used as unit identity key and given the fact during a sync call (eve the --force-full one) the unit records aren't updated in Mongo, the discrepancy would occur either after having deleting a repo and cleaning orphans or if a newer unit version (as in nevra) was pulled in. In these cases, one dnf unit version would have recommends attribute populated, the other wouldn't:
> db.units_rpm.find({'name': 'dnf'})[0].recommends
[
{
"release" : null,
"epoch" : null,
"version" : null,
"flags" : null,
"name" : "python3-dbus"
}
]
> db.units_rpm.find({'name': 'dnf'})[0].release
1.fc27
> db.units_rpm.find({'name': 'dnf'})[1].recommends
> db.units_rpm.find({'name': 'dnf'})[1].release
12.fc28
>
See also my test.log attachment.
Updated by milan over 6 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by milan over 6 years ago
- Status changed from NEW to POST
A (WIP) PR: https://github.com/pulp/pulp_rpm/pull/1135
Updated by milan over 6 years ago
- Description updated (diff)
Speaking about inconsistency, there might be other point of view on the migration: after it, out of the blue, users recursively copying e.g dnf-2.7.3-1.fc27.noarch.rpm will start pulling in sqlite-3 as a recommendation. This change of behavior will happen without any update to the dnf package itself. This might be surprising to the user because they might not anticipate a change of dependencies of a package without the accompanying bump of the package evr. As a consequence, recursive copies of dnf-2.7.3-1.fc27.noarch.rpm before and after the migration would differ in content.
Updated by ttereshc over 6 years ago
+1 to ipanova and dkliban point to have a migration for consistency reasons.
The fact that Pulp won't error without migration is not good enough reason not to do it, imho.
Updated by milan over 6 years ago
It's not about Pulp erroring out I'm afraid, it's about repositories having different content if they're created thru a recursive copy before and after the migration, no matter the content version being the same. This I find a bigger discrepancy than if all the non-updated rpm versions missed the recommends field (or just had the default empty list fwiw) because it's consistent with client behavior.
Updated by ttereshc over 6 years ago
I'm not sure I follow "the content version being the same" part.
If users upgrade to Pulp 2.x with weak deps support, they know that now weak deps are taken into account.
If they perform recursive copy, and the result will be different from the older version of Pulp that's fine because now weak deps are respected.
Published repository will have new revision number so clients will know that content was updated and is different now.
Let me know if I misunderstood you.
Ah before or after the migration. Migrations are an essential part of installation so we should not cover the case when they were not run
Added by milan over 6 years ago
Updated by milan over 6 years ago
PR change: https://github.com/pulp/pulp_rpm/pull/1151
Updated by milan over 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset 6d6c8292d2abba37142ff4f2af73c3fe6c3cb90d.
Updated by ipanova@redhat.com over 6 years ago
- Platform Release changed from 2.17.0 to master
Updated by ipanova@redhat.com over 6 years ago
- Platform Release deleted (
master)
Updated by ipanova@redhat.com over 6 years ago
- Status changed from MODIFIED to 5
Updated by kersom over 6 years ago
This new feature was manually verified.
Pulp copies strong and weak dependencies when using --recursive during a copy.
Updated by milan over 6 years ago
weak rich(boolean) deps are hosed:
(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo copy rpm -f f28 -t foo --recursive --str-eq 'name=bash-completion'
This command may be exited via ctrl+c without affecting the request.
[/]
Running...
Copied:
basesystem-11-5.fc28-noarch
bash-4.4.19-2.fc28-x86_64
bash-completion-2.7-4.fc28-noarch
fedora-gpg-keys-28-1-noarch
fedora-release-28-1-noarch
fedora-repos-28-1-noarch
filesystem-3.8-2.fc28-x86_64
glibc-2.27-8.fc28-x86_64
glibc-all-langpacks-2.27-8.fc28-x86_64
glibc-common-2.27-8.fc28-x86_64
libpkgconf-1.4.2-1.fc28-x86_64
libselinux-2.7-13.fc28-x86_64
libsepol-2.7-6.fc28-x86_64
ncurses-base-6.1-4.20180224.fc28-noarch
ncurses-libs-6.1-4.20180224.fc28-x86_64
pcre2-10.31-4.fc28-x86_64
pkgconf-1.4.2-1.fc28-x86_64
pkgconf-m4-1.4.2-1.fc28-noarch
pkgconf-pkg-config-1.4.2-1.fc28-x86_64
setup-2.11.3-1.fc28-noarch
tzdata-2018d-1.fc28-noarch
(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo copy rpm -f f28 -t foo --recursive --str-eq 'name=dnf'
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
Copied:
rpm: 145
(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo content rpm --repo-id f28 --str-eq 'name=sqlite'
Arch: x86_64
Buildhost: buildvm-25.phx2.fedoraproject.org
Checksum: 24ee748b29e4837c41df5b9472654f5aa66858f62cde706723d4f480b3f56fb6
Checksumtype: sha256
Description: SQLite is a C library that implements an SQL database engine. A
large subset of SQL92 is supported. A complete database is stored
in a single disk file. The API is designed for convenience and
ease of use. Applications that link against SQLite can enjoy the
power and flexibility of an SQL database without the
administrative hassles of supporting a separate database server.
Version 2 and version 3 binaries are named to permit each to be
installed on a single host
Epoch: 0
Filename: sqlite-3.22.0-4.fc28.x86_64.rpm
License: Public Domain
Name: sqlite
Provides: sqlite = 3.22.0-4.fc28-0, sqlite(x86-64) = 3.22.0-4.fc28-0
Release: 4.fc28
Requires: libdl.so.2()(64bit), libdl.so.2(GLIBC_2.2.5)(64bit),
libm.so.6()(64bit), libm.so.6(GLIBC_2.2.5)(64bit),
libncurses.so.6()(64bit), libpthread.so.0()(64bit),
libpthread.so.0(GLIBC_2.2.5)(64bit), libreadline.so.7()(64bit),
libtinfo.so.6()(64bit), libz.so.1()(64bit),
libz.so.1(ZLIB_1.2.0)(64bit), rtld(GNU_HASH), sqlite-libs =
3.22.0-4.fc28-0, libc.so.6(GLIBC_2.14)(64bit)
Vendor: Fedora Project
Version: 3.22.0
(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo content rpm --repo-id foo --str-eq 'name=sqlite'
# nothing found
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"recommends": {$elemMatch: {"name": /^\(/}}})[0].name'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
dnf
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"recommends": {$elemMatch: {"name": /^\(/}}})[0].recommends[0]'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
{
"release" : null,
"epoch" : null,
"version" : null,
"flags" : null,
"name" : "(/usr/bin/sqlite3 if bash-completion)"
}
(pulp) [vagrant@pulp2 yum]$
This works as expected with dnf (on a F28 Workstation VM):
[root@localhost ~]# rpm -q sqlite
sqlite-3.22.0-4.fc28.x86_64
[root@localhost ~]# rpm -e sqlite
error: Failed dependencies:
sqlite >= 3.22.0 is needed by (installed) firefox-59.0.2-1.fc28.x86_64
[root@localhost ~]# rpm -e firefox
# --->%--------------------------------------
warning: file /usr/lib64/firefox/langpacks: remove failed: No such file or directory
[root@localhost ~]# rpm -e sqlite
[root@localhost ~]# rpm -q bash-completion
bash-completion-2.7-4.fc28.noarch
[root@localhost ~]# rpm -q sqlite
package sqlite is not installed
[root@localhost ~]# dnf reinstall dnf
Last metadata expiration check: 0:28:47 ago on Tue 21 Aug 2018 08:43:20 PM CEST.
Dependencies resolved.
==================================================================================================================================================================================
Package Arch Version Repository Size
==================================================================================================================================================================================
Reinstalling:
dnf noarch 2.7.5-12.fc28 fedora 352 k
Installing weak dependencies:
sqlite x86_64 3.22.0-4.fc28 fedora 623 k
Transaction Summary
==================================================================================================================================================================================
Install 1 Package
Total download size: 975 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): dnf-2.7.5-12.fc28.noarch.rpm 8.2 MB/s | 352 kB 00:00
(2/2): sqlite-3.22.0-4.fc28.x86_64.rpm 13 MB/s | 623 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.3 MB/s | 975 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: sqlite-3.22.0-4.fc28.x86_64 1/1
Installing : sqlite-3.22.0-4.fc28.x86_64 1/3
Reinstalling : dnf-2.7.5-12.fc28.noarch 2/3
Running scriptlet: dnf-2.7.5-12.fc28.noarch 2/3
Running scriptlet: dnf-2.7.5-12.fc28.noarch 3/3
Erasing : dnf-2.7.5-12.fc28.noarch 3/3
Running scriptlet: dnf-2.7.5-12.fc28.noarch 3/3
Verifying : dnf-2.7.5-12.fc28.noarch 1/3
Verifying : sqlite-3.22.0-4.fc28.x86_64 2/3
Verifying : dnf-2.7.5-12.fc28.noarch 3/3
Reinstalled:
dnf.noarch 2.7.5-12.fc28
Installed:
sqlite.x86_64 3.22.0-4.fc28
Complete!
[root@localhost ~]# rpm -q sqlite
sqlite-3.22.0-4.fc28.x86_64
[root@localhost ~]#
Updated by kersom about 6 years ago
It was manual tested.
Following commands.
pulp-admin rpm repo create --repo-id foo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-richnweak-deps/
pulp-admin rpm repo create --repo-id bar
pulp-admin rpm repo sync run --repo-id foo
[root@f28c ~]# pulp-admin rpm repo copy rpm -f foo -t bar --recursive --str-eq 'name=Cobbler'
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Copied:
Cobbler-1-0-noarch
contireau-2-10-noarch
icecubes-2-3-noarch
orange-bits-2-3-noarch
tablespoon-sugar-1-0-noarch
Comparing same dependency resolution using DNF.
cat >/etc/yum.repos.d/weak.repo <<EOL
[weak_repo_modi]
baseurl=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-richnweak-deps/
gpgcheck=0
name=test_repo
EOL
kersom / etc yum.repos.d sudo dnf install Cobbler
test_repo 19 kB/s | 3.9 kB 00:00
Last metadata expiration check: 0:00:00 ago on Thu 23 Aug 2018 12:59:56 PM EDT.
Dependencies resolved.
===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
Cobbler noarch 1-0 weak_repo_modi 6.2 k
Installing dependencies:
contireau noarch 2-10 weak_repo_modi 5.8 k
icecubes noarch 2-3 weak_repo_modi 5.7 k
tablespoon-sugar noarch 1-0 weak_repo_modi 5.7 k
Installing weak dependencies:
orange-bits noarch 2-3 weak_repo_modi 5.9 k
Transaction Summary
===================================================================================================================================================================================================================
Install 5 Packages
Total download size: 29 k
Installed size: 0
Is this ok [y/N]:
Updated by ipanova@redhat.com about 6 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
Introduce the Recommends dependency
Pulp has been ignoring the Recommends weak dependency when processing e.g recursive unit copies. This patch updates the RPM model to track a recommends unit attribute and the primary.xml parser to populate this attribute. The libsolv dependency solver is used to process this attribute when calculating unit dependencies.
Migration: populate the recommends attribute
Walks over the rpm units, unzipping the primary metadata XML snippets, parsing the recommends entries and populating the recommends unit attributes accoridingly.
Fixes: #3847 https://pulp.plan.io/issues/3847