Actions
Story #3715
closed
Reimplement unit dependency solving with the libsolv library
Added by milan almost 6 years ago. Updated over 2 years ago.
Start date:
Due date:
% Done:
100%
Estimated time:
Platform Release:
2.17.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 40
Quarter:
Description
This story tracks tasks required to implement dependency solving using the libsolv library, based on the investigation and on the POC from the Issue #3528.
The goal is to implement a solver suitable for the importer associate interface but some refactoring of the current internal solver call interface might be necessary.
The solver should support the Yum-specific content unit types (recursive) dependency solving, namely:
- RPM
- Errata
- Package Group
- Package Category
on content unit attributes, such as provides, requires, as relevant to a particular content unit model, including boolean&weak dependencies solving, in the context of a source and target repository to Support more conservative dependency solving
A separate story will track the modular content as it requires a libsolv patch to expose the pool->considered bitmap to the Python bindings.
Implementation design is described on the Issue #3528; see the Note #8
The POC code can be used as an example.
The functionality exposed by this implementation should be feature-compatible with current Pulp2 implementation.
It implies that no support for either SRPM or DRPM is required and that the REST API must not change when it comes to reporting the units copied.
Related issues
Updated by ipanova@redhat.com almost 6 years ago
- Platform Release deleted (
2.17.0)
Updated by ttereshc almost 6 years ago
From the checklist, which functionality currently exists?
E.g. I don't think we perform dep solving against multiple repositories.
The list from investigation task incorporated not only existing functionality but also the potential plans and functionality for pulp 2 and/or pulp3.
Could you modify the task description and the checklist to reflect existing functionality?
As for different types of dependencies I believe apart from strict we can go ahead and add weak ones if it's one line of code and doesn't require any custom logic.
+ add to the checklist to create a pulp-smash issue.
Thanks!
Updated by ttereshc almost 6 years ago
And feel free to create a story for modularity. It may not be ready for grooming and it may contain just basic description at the moment. At least we'd have it and wouldn't forget about it (which is unlikely, of course :) )
Updated by milan almost 6 years ago
- Description updated (diff)
ttereshc wrote:
From the checklist, which functionality currently exists?
E.g. I don't think we perform dep solving against multiple repositories.
done
The list from investigation task incorporated not only existing functionality but also the potential plans and functionality for pulp 2 and/or pulp3.
Could you modify the task description and the checklist to reflect existing functionality?
What exactly is missing? The checklist is a verbose copy from the investigation issue FWIW.
As for different types of dependencies I believe apart from strict we can go ahead and add weak ones if it's one line of code and doesn't require any custom logic.
done
+ add to the checklist to create a pulp-smash issue.
done
Thanks!
Updated by ttereshc almost 6 years ago
- Groomed changed from No to Yes
Thanks, I added a couple of items to the checklist.
Updated by milan almost 6 years ago
- Status changed from ASSIGNED to POST
Updated by milan almost 6 years ago
Pulp smash issue: https://github.com/PulpQE/pulp-smash/issues/901
Updated by ipanova@redhat.com almost 6 years ago
- Related to Story #2478: Support more conservative dependency solving added
Updated by milan almost 6 years ago
- Related to Story #3847: Introduce weak dependency solving added
Updated by dkliban@redhat.com almost 6 years ago
- Sprint changed from Sprint 39 to Sprint 40
Added by milan almost 6 years ago
Updated by milan almost 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset f85007c0e6cc93e4320f12dfd9c7568707c25759.
Updated by ipanova@redhat.com almost 6 years ago
- Related to Task #3859: Remove code of old solver and its' unittests added
Added by ipanova@redhat.com over 5 years ago
Revision b99f97cc | View on GitHub
Merge pull request #1156 from dparalen/issues_3903
Share solver between recursive associate calls
Updated by ipanova@redhat.com over 5 years ago
- Status changed from MODIFIED to 5
Updated by milan over 5 years ago
Unfortunately, I've just realized the rich dependencies is hosed in Pulp; see for yourself:
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo content rpm --repo-id f28 --str-eq 'name=annobin'
Arch: x86_64
Buildhost: buildvm-20.phx2.fedoraproject.org
Checksum: 4c0674435ce645cbbcf02892f1d6a786628698470e606fc412fb65673e062b68
Checksumtype: sha256
Description: A plugin for GCC that records extra information in the files that
it compiles, and a set of scripts that analyze the recorded
information. These scripts can determine things ABI clashes in
compiled binaries, or the absence of required hardening options.
Note - the plugin is enabled in gcc builds via flags provided by
the redhat-rpm-macros package, and the analysis tools rely upon
the readelf program from the binutils package.
Epoch: 0
Filename: annobin-5.2-1.fc28.x86_64.rpm
License: GPLv3+
Name: annobin
Provides: annobin = 5.2-1.fc28-0, annobin(x86-64) = 5.2-1.fc28-0
Release: 1.fc28
Requires: /bin/bash, /bin/sh, /bin/sh, /sbin/install-info,
/sbin/install-info, gcc, libgcc_s.so.1()(64bit),
libm.so.6()(64bit), libstdc++.so.6()(64bit), rtld(GNU_HASH),
libc.so.6(GLIBC_2.4)(64bit)
Vendor: Fedora Project
Version: 5.2
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo create --repo-id bar
Successfully created repository [bar]
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo copy rpm -f f28 -t bar --recursive --str-eq='name=gcc'
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
Copied:
libacl-2.2.52-20.fc28-x86_64
libattr-2.4.47-23.fc28-x86_64
libcap-2.25-9.fc28-x86_64
libffi-3.1-16.fc28-x86_64
libgcc-8.0.1-0.20.fc28-x86_64
libgomp-8.0.1-0.20.fc28-x86_64
libmpc-1.0.2-9.fc28-x86_64
libpkgconf-1.4.2-1.fc28-x86_64
libselinux-2.7-13.fc28-x86_64
libsepol-2.7-6.fc28-x86_64
libtasn1-4.13-2.fc28-x86_64
libxcrypt-4.0.0-5.fc28-x86_64
libxcrypt-devel-4.0.0-5.fc28-x86_64
mpfr-3.1.6-1.fc28-x86_64
ncurses-6.1-4.20180224.fc28-x86_64
ncurses-base-6.1-4.20180224.fc28-noarch
ncurses-libs-6.1-4.20180224.fc28-x86_64
openssl-libs-1.1.0h-3.fc28-x86_64
p11-kit-0.23.10-1.fc28-x86_64
p11-kit-trust-0.23.10-1.fc28-x86_64
pcre-8.42-1.fc28-x86_64
pcre2-10.31-4.fc28-x86_64
pkgconf-1.4.2-1.fc28-x86_64
pkgconf-m4-1.4.2-1.fc28-noarch
pkgconf-pkg-config-1.4.2-1.fc28-x86_64
popt-1.16-14.fc28-x86_64
sed-4.5-1.fc28-x86_64
setup-2.11.3-1.fc28-noarch
tzdata-2018d-1.fc28-noarch
zlib-1.2.11-7.fc28-x86_64
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo content rpm --repo-id bar --str-eq 'name=annobin'
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo content rpm --repo-id bar --str-eq 'name=gcc'
Arch: x86_64
Buildhost: buildvm-11.phx2.fedoraproject.org
Checksum: cfd151e0a68a81f59df4176af6c1f010e3f388470529db88fcc2a826c5da12fa
Checksumtype: sha256
Description: The gcc package contains the GNU Compiler Collection version 8.
You'll need this package in order to compile C code.
Epoch: 0
Filename: gcc-8.0.1-0.20.fc28.x86_64.rpm
License: GPLv3+ and GPLv3+ with exceptions and GPLv2+ with exceptions and
LGPLv2+ and BSD
Name: gcc
Provides: bundled(libiberty), gcc = 8.0.1-0.20.fc28-0, gcc(major) = 8-0,
gcc(x86-64) = 8.0.1-0.20.fc28-0, liblto_plugin.so.0()(64bit)
Release: 0.20.fc28
Requires: /bin/sh, /bin/sh, /sbin/install-info, /sbin/install-info, binutils
>= 2.24-0, cpp = 8.0.1-0.20.fc28-0, glibc-devel >= 2.2.90-12-0,
ld-linux-x86-64.so.2()(64bit),
ld-linux-x86-64.so.2(GLIBC_2.3)(64bit), libdl.so.2()(64bit),
libdl.so.2(GLIBC_2.2.5)(64bit), libgcc >= 8.0.1-0.20.fc28-0,
libgcc_s.so.1()(64bit), libgmp.so.10()(64bit), libgomp =
8.0.1-0.20.fc28-0, libgomp.so.1()(64bit), libisl.so.15()(64bit),
libm.so.6()(64bit), libm.so.6(GLIBC_2.2.5)(64bit),
libmpc.so.3()(64bit), libmpfr.so.4()(64bit), libz.so.1()(64bit),
rtld(GNU_HASH), libc.so.6(GLIBC_2.14)(64bit)
Vendor: Fedora Project
Version: 8.0.1
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo content rpm --repo-id bar --str-eq 'name=redhat-rpm-config'
# nothing found
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"requires": {$elemMatch: {"name": /^\(/}}})[1].requires[0]'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
{
"release" : null,
"epoch" : null,
"version" : null,
"flags" : null,
"name" : "(annobin if gcc)"
}
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"requires": {$elemMatch: {"name": /^\(/}}})[1].name'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
redhat-rpm-config
(pulp) [vagrant@pulp2 yum]$
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo copy rpm -f f28 -t bar --recursive --str-eq='name=redhat-rpm-config'
This command may be exited via ctrl+c without affecting the request.
[/]
Running...
Copied:
bzip2-libs-1.0.6-26.fc28-x86_64
curl-7.59.0-2.fc28-x86_64
cyrus-sasl-lib-2.1.27-0.1rc7.fc28-x86_64
dwz-0.12-7.fc28-x86_64
elfutils-libelf-0.170-11.fc28-x86_64
expat-2.2.5-3.fc28-x86_64
file-5.32-3.fc28-x86_64
file-libs-5.32-3.fc28-x86_64
findutils-4.6.0-17.fc28-x86_64
fpc-srpm-macros-1.1-4.fc28-noarch
gawk-4.2.1-1.fc28-x86_64
ghc-srpm-macros-1.4.2-7.fc28-noarch
gnat-srpm-macros-4-5.fc28-noarch
go-srpm-macros-2-16.fc28-noarch
keyutils-libs-1.5.10-6.fc28-x86_64
krb5-libs-1.16-21.fc28-x86_64
libarchive-3.3.1-4.fc28-x86_64
libcom_err-1.43.8-2.fc28-x86_64
libcurl-7.59.0-2.fc28-x86_64
libdb-5.3.28-30.fc28-x86_64
libdb-utils-5.3.28-30.fc28-x86_64
libidn2-2.0.4-7.fc28-x86_64
libmetalink-0.1.3-6.fc28-x86_64
libnghttp2-1.31.1-1.fc28-x86_64
libpsl-0.20.1-1.fc28-x86_64
libsigsegv-2.11-5.fc28-x86_64
libssh-0.7.5-7.fc28-x86_64
libunistring-0.9.9-1.fc28-x86_64
libverto-0.3.0-5.fc28-x86_64
libxml2-2.9.7-4.fc28-x86_64
libzstd-1.3.4-1.fc28-x86_64
lua-libs-5.3.4-10.fc28-x86_64
lz4-libs-1.8.1.2-4.fc28-x86_64
nim-srpm-macros-1-1.fc28-noarch
nspr-4.19.0-1.fc28-x86_64
nss-3.36.0-1.0.fc28-x86_64
nss-softokn-3.36.0-1.0.fc28-x86_64
nss-softokn-freebl-3.36.0-1.0.fc28-x86_64
nss-sysinit-3.36.0-1.0.fc28-x86_64
nss-tools-3.36.0-1.0.fc28-x86_64
nss-util-3.36.0-1.0.fc28-x86_64
ocaml-srpm-macros-5-2.fc27-noarch
openblas-srpm-macros-2-2.fc27-noarch
openldap-2.4.46-1.fc28-x86_64
perl-srpm-macros-1-25.fc28-noarch
publicsuffix-list-dafsa-20180328-1.fc28-noarch
python-srpm-macros-3-28.fc28-noarch
qt5-srpm-macros-5.10.1-1.fc28-noarch
readline-7.0-10.fc28-x86_64
redhat-rpm-config-106-1.fc28-noarch
rpm-4.14.1-7.fc28-x86_64
rpm-libs-4.14.1-7.fc28-x86_64
rpm-plugin-selinux-4.14.1-7.fc28-x86_64
rust-srpm-macros-5-2.fc28-noarch
sqlite-libs-3.22.0-4.fc28-x86_64
unzip-6.0-38.fc28-x86_64
xz-libs-5.2.3-7.fc28-x86_64
zip-3.0-21.fc28-x86_64
(pulp) [vagrant@pulp2 yum]$ pulp-admin rpm repo content rpm --repo-id bar --str-eq 'name=annobin'
(pulp) [vagrant@pulp2 yum]$ echo $?
0
(pulp) [vagrant@pulp2 yum]$
whereas dnf performs as expected:
[root@localhost ~]# dnf search annobin
Last metadata expiration check: 0:03:24 ago on Tue 21 Aug 2018 08:43:20 PM CEST.
========================================================================= Name Exactly Matched: annobin ==========================================================================
annobin.x86_64 : Binary annotation plugin for GCC
[root@localhost ~]# rpm -q gcc
gcc-8.1.1-5.fc28.x86_64
[root@localhost ~]# rpm -q annobin
package annobin is not installed
[root@localhost ~]# rpm -q redhat-rpm-config
package redhat-rpm-config is not installed
[root@localhost ~]# dnf install redhat-rpm-config
Last metadata expiration check: 0:03:55 ago on Tue 21 Aug 2018 08:43:20 PM CEST.
Dependencies resolved.
==================================================================================================================================================================================
Package Arch Version Repository Size
==================================================================================================================================================================================
Installing:
redhat-rpm-config noarch 108-1.fc28 updates 77 k
Installing dependencies:
annobin x86_64 5.7-2.fc28 updates 85 k
dwz x86_64 0.12-7.fc28 fedora 107 k
fpc-srpm-macros noarch 1.1-4.fc28 fedora 7.5 k
ghc-srpm-macros noarch 1.4.2-7.fc28 fedora 8.2 k
gnat-srpm-macros noarch 4-5.fc28 fedora 8.8 k
go-srpm-macros noarch 2-16.fc28 fedora 13 k
nim-srpm-macros noarch 1-1.fc28 fedora 7.6 k
ocaml-srpm-macros noarch 5-2.fc27 fedora 7.8 k
openblas-srpm-macros noarch 2-2.fc27 fedora 6.6 k
perl-srpm-macros noarch 1-25.fc28 fedora 9.7 k
python-srpm-macros noarch 3-30.fc28 updates 11 k
qt5-srpm-macros noarch 5.10.1-1.fc28 fedora 9.6 k
rust-srpm-macros noarch 5-2.fc28 fedora 8.1 k
Transaction Summary
==================================================================================================================================================================================
Install 14 Packages
Total download size: 368 k
Installed size: 550 k
Is this ok [y/N]: y
Downloading Packages:
(1/14): fpc-srpm-macros-1.1-4.fc28.noarch.rpm 238 kB/s | 7.5 kB 00:00
(2/14): redhat-rpm-config-108-1.fc28.noarch.rpm 2.2 MB/s | 77 kB 00:00
(3/14): dwz-0.12-7.fc28.x86_64.rpm 2.9 MB/s | 107 kB 00:00
(4/14): ghc-srpm-macros-1.4.2-7.fc28.noarch.rpm 1.5 MB/s | 8.2 kB 00:00
(5/14): gnat-srpm-macros-4-5.fc28.noarch.rpm 2.0 MB/s | 8.8 kB 00:00
(6/14): go-srpm-macros-2-16.fc28.noarch.rpm 2.7 MB/s | 13 kB 00:00
(7/14): nim-srpm-macros-1-1.fc28.noarch.rpm 1.5 MB/s | 7.6 kB 00:00
(8/14): ocaml-srpm-macros-5-2.fc27.noarch.rpm 2.0 MB/s | 7.8 kB 00:00
(9/14): openblas-srpm-macros-2-2.fc27.noarch.rpm 1.3 MB/s | 6.6 kB 00:00
(10/14): perl-srpm-macros-1-25.fc28.noarch.rpm 2.1 MB/s | 9.7 kB 00:00
(11/14): qt5-srpm-macros-5.10.1-1.fc28.noarch.rpm 1.9 MB/s | 9.6 kB 00:00
(12/14): rust-srpm-macros-5-2.fc28.noarch.rpm 1.6 MB/s | 8.1 kB 00:00
(13/14): python-srpm-macros-3-30.fc28.noarch.rpm 2.6 MB/s | 11 kB 00:00
(14/14): annobin-5.7-2.fc28.x86_64.rpm 11 MB/s | 85 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 225 kB/s | 368 kB 00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : python-srpm-macros-3-30.fc28.noarch 1/14
Installing : annobin-5.7-2.fc28.x86_64 2/14
Running scriptlet: annobin-5.7-2.fc28.x86_64 2/14
Installing : rust-srpm-macros-5-2.fc28.noarch 3/14
Installing : qt5-srpm-macros-5.10.1-1.fc28.noarch 4/14
Installing : perl-srpm-macros-1-25.fc28.noarch 5/14
Installing : openblas-srpm-macros-2-2.fc27.noarch 6/14
Installing : ocaml-srpm-macros-5-2.fc27.noarch 7/14
Installing : nim-srpm-macros-1-1.fc28.noarch 8/14
Installing : go-srpm-macros-2-16.fc28.noarch 9/14
Installing : gnat-srpm-macros-4-5.fc28.noarch 10/14
Installing : ghc-srpm-macros-1.4.2-7.fc28.noarch 11/14
Installing : fpc-srpm-macros-1.1-4.fc28.noarch 12/14
Installing : dwz-0.12-7.fc28.x86_64 13/14
Installing : redhat-rpm-config-108-1.fc28.noarch 14/14
Running scriptlet: redhat-rpm-config-108-1.fc28.noarch 14/14
Verifying : redhat-rpm-config-108-1.fc28.noarch 1/14
Verifying : dwz-0.12-7.fc28.x86_64 2/14
Verifying : fpc-srpm-macros-1.1-4.fc28.noarch 3/14
Verifying : ghc-srpm-macros-1.4.2-7.fc28.noarch 4/14
Verifying : gnat-srpm-macros-4-5.fc28.noarch 5/14
Verifying : go-srpm-macros-2-16.fc28.noarch 6/14
Verifying : nim-srpm-macros-1-1.fc28.noarch 7/14
Verifying : ocaml-srpm-macros-5-2.fc27.noarch 8/14
Verifying : openblas-srpm-macros-2-2.fc27.noarch 9/14
Verifying : perl-srpm-macros-1-25.fc28.noarch 10/14
Verifying : qt5-srpm-macros-5.10.1-1.fc28.noarch 11/14
Verifying : rust-srpm-macros-5-2.fc28.noarch 12/14
Verifying : annobin-5.7-2.fc28.x86_64 13/14
Verifying : python-srpm-macros-3-30.fc28.noarch 14/14
Installed:
redhat-rpm-config.noarch 108-1.fc28 annobin.x86_64 5.7-2.fc28 dwz.x86_64 0.12-7.fc28 fpc-srpm-macros.noarch 1.1-4.fc28
ghc-srpm-macros.noarch 1.4.2-7.fc28 gnat-srpm-macros.noarch 4-5.fc28 go-srpm-macros.noarch 2-16.fc28 nim-srpm-macros.noarch 1-1.fc28
ocaml-srpm-macros.noarch 5-2.fc27 openblas-srpm-macros.noarch 2-2.fc27 perl-srpm-macros.noarch 1-25.fc28 python-srpm-macros.noarch 3-30.fc28
qt5-srpm-macros.noarch 5.10.1-1.fc28 rust-srpm-macros.noarch 5-2.fc28
Complete!
[root@localhost ~]# rpm -q annobin
annobin-5.7-2.fc28.x86_64
[root@localhost ~]# rpm -q redhat-rpm-config
redhat-rpm-config-108-1.fc28.noarch
[root@localhost ~]# rpm -q redhat-rpm-config --requires
(annobin if gcc)
/bin/bash
/bin/sh
/usr/bin/file
/usr/bin/find
/usr/bin/grep
/usr/bin/sed
/usr/bin/xargs
coreutils
dwz >= 0.4
fpc-srpm-macros
ghc-srpm-macros
gnat-srpm-macros
go-srpm-macros
nim-srpm-macros
ocaml-srpm-macros
openblas-srpm-macros
perl-srpm-macros
python-srpm-macros
qt5-srpm-macros
rpm >= 4.11.0
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
rpmlib(RichDependencies) <= 4.12.0-1
rust-srpm-macros
zip
[root@localhost ~]#
Added by milan over 5 years ago
Revision a415ce87 | View on GitHub
Fix indentation in the solver to parse rich deps
The rich dep branch of the code was actually never accessed.
Updated by kersom over 5 years ago
This feature was manually verified. Automation will be added soon.
Updated by ipanova@redhat.com over 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
.
Resolve rpm (rich) dependencies via libsolv
Pulp now uses libsolv as a dependency solver to address recursive RPM unit dependency solving during e.g unit copy
Fixes: #3715 https://pulp.plan.io/issues/3715
Fixes: #2478 https://pulp.plan.io/issues/2478