RPM Support

since there is a change in the model, shall we require a migration?

Thanks for the check; to my best knowledge a migration is unnecessary, with the following reasoning:

• by definition, nothing should break if (during an installation) a unit Recommends dependency is missing from a repository
• Pulp has been ignoring the Recommends dependency so far
• therefore implementing this story without a migration won't make things worse

The migration itself would have to process all the RPM unit XML records to be able to populate the Recommends attribute which might have adverse performance effects. The unit Recommends attribute will be populated for newly-uploaded or newly-synchronized units.

However, let's document this.

Without a migration the user will get into a situation where the RPMs that were synced before 2.17.0 will not have the 'recommends' field set. The RPMs that are synced after 2.17.0 release will have 'recommends' set. When the user goes to copy RPMs recursively he will notice that some RPMs bring their weak deps with them and some RPMs don't bring their weak deps with them.

dkliban@redhat.com wrote:

Without a migration the user will get into a situation where the RPMs that were synced before 2.17.0 will not have the 'recommends' field set. The RPMs that are synced after 2.17.0 release will have 'recommends' set. When the user goes to copy RPMs recursively he will notice that some RPMs bring their weak deps with them and some RPMs don't bring their weak deps with them.

I agree but please mind this isn't a breaking discrepancy; we should be OK to just document (and test) this behaviour.

thanks @dkliban, this is exact reason why i asked for a migration :)
As Milan pointed out, in the migration we'd need to get from the DB for each rpm unit the xml snippet, uncompress it, parse it.
On the other hand if we won't write the migration then those units before the upgrade will forever miss the the newly added field unless the unit will be removed and added back to the DB. Nothing would break without the migration, dnf will still be happy, just the behaviour during recursive copy will be inconsistent - for some units weak deps will will be copied over and for some not.

I think i am still in favour of writing the migration even if it is a costful one, let's minimize the inconsistent behaviour as much as we can.

i think this story is ready to be groomed once we +1 or -1 on the migration.

+1 to ipanova and dkliban point to have a migration for consistency reasons.
The fact that Pulp won't error without migration is not good enough reason not to do it, imho.

It's not about Pulp erroring out I'm afraid, it's about repositories having different content if they're created thru a recursive copy before and after the migration, no matter the content version being the same. This I find a bigger discrepancy than if all the non-updated rpm versions missed the recommends field (or just had the default empty list fwiw) because it's consistent with client behavior.

I'm not sure I follow "the content version being the same" part.

If users upgrade to Pulp 2.x with weak deps support, they know that now weak deps are taken into account.
If they perform recursive copy, and the result will be different from the older version of Pulp that's fine because now weak deps are respected.
Published repository will have new revision number so clients will know that content was updated and is different now.
Let me know if I misunderstood you.

Ah before or after the migration. Migrations are an essential part of installation so we should not cover the case when they were not run

PR change: https://github.com/pulp/pulp_rpm/pull/1151

This new feature was manually verified.

Pulp copies strong and weak dependencies when using --recursive during a copy.

weak rich(boolean) deps are hosed:

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo copy rpm -f f28 -t foo --recursive --str-eq 'name=bash-completion'
This command may be exited via ctrl+c without affecting the request.

[/]
Running...

Copied:
  basesystem-11-5.fc28-noarch
  bash-4.4.19-2.fc28-x86_64
  bash-completion-2.7-4.fc28-noarch
  fedora-gpg-keys-28-1-noarch
  fedora-release-28-1-noarch
  fedora-repos-28-1-noarch
  filesystem-3.8-2.fc28-x86_64
  glibc-2.27-8.fc28-x86_64
  glibc-all-langpacks-2.27-8.fc28-x86_64
  glibc-common-2.27-8.fc28-x86_64
  libpkgconf-1.4.2-1.fc28-x86_64
  libselinux-2.7-13.fc28-x86_64
  libsepol-2.7-6.fc28-x86_64
  ncurses-base-6.1-4.20180224.fc28-noarch
  ncurses-libs-6.1-4.20180224.fc28-x86_64
  pcre2-10.31-4.fc28-x86_64
  pkgconf-1.4.2-1.fc28-x86_64
  pkgconf-m4-1.4.2-1.fc28-noarch
  pkgconf-pkg-config-1.4.2-1.fc28-x86_64
  setup-2.11.3-1.fc28-noarch
  tzdata-2018d-1.fc28-noarch

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo copy rpm -f f28 -t foo --recursive --str-eq 'name=dnf'
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
  rpm: 145

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo content rpm --repo-id f28 --str-eq 'name=sqlite'
Arch:         x86_64
Buildhost:    buildvm-25.phx2.fedoraproject.org
Checksum:     24ee748b29e4837c41df5b9472654f5aa66858f62cde706723d4f480b3f56fb6
Checksumtype: sha256
Description:  SQLite is a C library that implements an SQL database engine. A
              large subset of SQL92 is supported. A complete database is stored
              in a single disk file. The API is designed for convenience and
              ease of use. Applications that link against SQLite can enjoy the
              power and flexibility of an SQL database without the
              administrative hassles of supporting a separate database server.
              Version 2 and version 3 binaries are named to permit each to be
              installed on a single host
Epoch:        0
Filename:     sqlite-3.22.0-4.fc28.x86_64.rpm
License:      Public Domain
Name:         sqlite
Provides:     sqlite = 3.22.0-4.fc28-0, sqlite(x86-64) = 3.22.0-4.fc28-0
Release:      4.fc28
Requires:     libdl.so.2()(64bit), libdl.so.2(GLIBC_2.2.5)(64bit),
              libm.so.6()(64bit), libm.so.6(GLIBC_2.2.5)(64bit),
              libncurses.so.6()(64bit), libpthread.so.0()(64bit),
              libpthread.so.0(GLIBC_2.2.5)(64bit), libreadline.so.7()(64bit),
              libtinfo.so.6()(64bit), libz.so.1()(64bit),
              libz.so.1(ZLIB_1.2.0)(64bit), rtld(GNU_HASH), sqlite-libs =
              3.22.0-4.fc28-0, libc.so.6(GLIBC_2.14)(64bit)
Vendor:       Fedora Project
Version:      3.22.0

(pulp) [vagrant@pulp2 importers]$ pulp-admin rpm repo content rpm --repo-id foo --str-eq 'name=sqlite'
# nothing found
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"recommends": {$elemMatch: {"name": /^\(/}}})[0].name'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
dnf
(pulp) [vagrant@pulp2 yum]$ mongo pulp_database --eval 'db.units_rpm.find({"recommends": {$elemMatch: {"name": /^\(/}}})[0].recommends[0]'
MongoDB shell version v3.4.11
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.11
{
        "release" : null,
        "epoch" : null,
        "version" : null,
        "flags" : null,
        "name" : "(/usr/bin/sqlite3 if bash-completion)"
}
(pulp) [vagrant@pulp2 yum]$

This works as expected with dnf (on a F28 Workstation VM):

[root@localhost ~]# rpm -q sqlite
sqlite-3.22.0-4.fc28.x86_64
[root@localhost ~]# rpm -e sqlite
error: Failed dependencies:
    sqlite >= 3.22.0 is needed by (installed) firefox-59.0.2-1.fc28.x86_64
[root@localhost ~]# rpm -e firefox
# --->%--------------------------------------
warning: file /usr/lib64/firefox/langpacks: remove failed: No such file or directory
[root@localhost ~]# rpm -e sqlite
[root@localhost ~]# rpm -q bash-completion
bash-completion-2.7-4.fc28.noarch
[root@localhost ~]# rpm -q sqlite
package sqlite is not installed
[root@localhost ~]# dnf reinstall dnf
Last metadata expiration check: 0:28:47 ago on Tue 21 Aug 2018 08:43:20 PM CEST.
Dependencies resolved.
==================================================================================================================================================================================
 Package                                  Arch                                     Version                                         Repository                                Size
==================================================================================================================================================================================
Reinstalling:
 dnf                                      noarch                                   2.7.5-12.fc28                                   fedora                                   352 k
Installing weak dependencies:
 sqlite                                   x86_64                                   3.22.0-4.fc28                                   fedora                                   623 k

Transaction Summary
==================================================================================================================================================================================
Install  1 Package

Total download size: 975 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): dnf-2.7.5-12.fc28.noarch.rpm                                                                                                               8.2 MB/s | 352 kB     00:00
(2/2): sqlite-3.22.0-4.fc28.x86_64.rpm                                                                                                             13 MB/s | 623 kB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                             1.3 MB/s | 975 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                          1/1
  Running scriptlet: sqlite-3.22.0-4.fc28.x86_64                                                                                                                              1/1
  Installing       : sqlite-3.22.0-4.fc28.x86_64                                                                                                                              1/3
  Reinstalling     : dnf-2.7.5-12.fc28.noarch                                                                                                                                 2/3
  Running scriptlet: dnf-2.7.5-12.fc28.noarch                                                                                                                                 2/3
  Running scriptlet: dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3
  Erasing          : dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3
  Running scriptlet: dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3
  Verifying        : dnf-2.7.5-12.fc28.noarch                                                                                                                                 1/3
  Verifying        : sqlite-3.22.0-4.fc28.x86_64                                                                                                                              2/3
  Verifying        : dnf-2.7.5-12.fc28.noarch                                                                                                                                 3/3

Reinstalled:
  dnf.noarch 2.7.5-12.fc28

Installed:
  sqlite.x86_64 3.22.0-4.fc28

Complete!
[root@localhost ~]# rpm -q sqlite
sqlite-3.22.0-4.fc28.x86_64
[root@localhost ~]#

It was manual tested.

Following commands.

pulp-admin rpm repo create --repo-id foo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-richnweak-deps/
pulp-admin rpm repo create --repo-id bar
pulp-admin rpm repo sync run --repo-id foo
[root@f28c ~]# pulp-admin rpm repo copy rpm -f foo -t bar --recursive --str-eq 'name=Cobbler'
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
  Cobbler-1-0-noarch
  contireau-2-10-noarch
  icecubes-2-3-noarch
  orange-bits-2-3-noarch
  tablespoon-sugar-1-0-noarch

Comparing same dependency resolution using DNF.

cat >/etc/yum.repos.d/weak.repo <<EOL
[weak_repo_modi]
baseurl=https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-richnweak-deps/
gpgcheck=0
name=test_repo
EOL

 kersom  /  etc  yum.repos.d  sudo dnf install Cobbler
test_repo                                                                                                                                                                           19 kB/s | 3.9 kB     00:00    
Last metadata expiration check: 0:00:00 ago on Thu 23 Aug 2018 12:59:56 PM EDT.
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                  Arch                                           Version                                      Repository                                              Size
===================================================================================================================================================================================================================
Installing:
 Cobbler                                                  noarch                                         1-0                                          weak_repo_modi                                         6.2 k
Installing dependencies:
 contireau                                                noarch                                         2-10                                         weak_repo_modi                                         5.8 k
 icecubes                                                 noarch                                         2-3                                          weak_repo_modi                                         5.7 k
 tablespoon-sugar                                         noarch                                         1-0                                          weak_repo_modi                                         5.7 k
Installing weak dependencies:
 orange-bits                                              noarch                                         2-3                                          weak_repo_modi                                         5.9 k

Transaction Summary
===================================================================================================================================================================================================================
Install  5 Packages

Total download size: 29 k
Installed size: 0  
Is this ok [y/N]: