Project

Profile

Help

Issue #1979

metadata unit copy action creates incorrect unit count on repo

Added by bmbouter over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
Platform Release:
2.8.5
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 3
Quarter:

Description

This issue affects all metadata rpm types: package categories, package groups, package environments, yum_metadata_file, and package_langpacks. I'll illustrate with langpacks, but the same should be true of all of them.

There can only be 1 package_langpacks associated with a repository. You can see that when copying langpacks over and over the count continues to increment even though each copy overwrites the previous one. Consider this reproducer:

pulp-admin rpm repo create --repo-id=repo_1
pulp-admin rpm repo create --repo-id=repo_2
pulp-admin rpm repo uploads langpacks -i hyphen -n hyphen-%s --repo-id repo_1
pulp-admin rpm repo copy langpacks --from-repo-id=repo_1 --to-repo-id=repo_2
pulp-admin rpm repo copy langpacks --from-repo-id=repo_1 --to-repo-id=repo_2
pulp-admin rpm repo copy langpacks --from-repo-id=repo_1 --to-repo-id=repo_2

Now when listing the langpacks units for repo_2 I expect there still to be 1, but instead it shows 3:

[vagrant@dev ~]$ pulp-admin rpm repo list --repo-id repo_2
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                   repo_2
Display Name:         None
Description:          None
Content Unit Counts:  
  Package Langpacks: 3

There is still in fact only 1 package_langpack which I verified in the database with:

> db.units_package_langpacks.find({"repo_id": "repo_2"}).count()
1

Given ^, the machinery that handles the repo counts is not keep the count correctly. I believe that machinery is here[0].

[0]: https://github.com/pulp/pulp_rpm/blob/d52206bdff5419b5b74976089c605a6cdbeaffc4/plugins/pulp_rpm/plugins/importers/yum/associate.py#L345-L380


Checklist

Associated revisions

Revision 6ac95ae4 View on GitHub
Added by bmbouter over 4 years ago

Moves migration 31 -> 33 so migrations can be introduced in 2.8.z

This allows the fix for issue 1979 to become migration 31 and the fix for issue 1955 to become migration 32. Both of those will be introduced in 2.8-dev and merged forward to master.

https://pulp.plan.io/issues/1979 https://pulp.plan.io/issues/1955

re #1979 re #1955

Revision 077871ad View on GitHub
Added by bmbouter over 4 years ago

Overwriting a metadata unit during copy disassociates first

The metadata units have an overwrite functionaity whereby newer units overwrite older units if they have the same unit key. The unit being overwritten is deleted and the new one is added. The unit being deleted did not have disassociate called so the unit count would be wrong for these unit types after multiple copies.

This comes with a migration which regenerates the unit counts for all repos fixing any incorrect data as a result of the bug.

HEAD of pulp-smash passes and all unit tests pass

https://pulp.plan.io/issues/1979 fixes #1979

History

#1 Updated by bmbouter over 4 years ago

  • Subject changed from Langpacks copy creates incorrect unit count to metadata unit copy action creates incorrect unit count on repo
  • Description updated (diff)

This issue is not unique to langpacks so I'm updating the issue.

#2 Updated by mhrivnak over 4 years ago

  • Checklist item Confirm that this bug did not affect 2.8 added

#3 Updated by bmbouter over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
  • Priority changed from Normal to High
  • Sprint/Milestone set to 21
  • Platform Release set to 2.9.0

#4 Updated by bmbouter over 4 years ago

  • Checklist item Confirm that this bug did not affect 2.8 set to Done

#5 Updated by bmbouter over 4 years ago

I was able to reproduce the issue with Package Category types, since Package Langpacks were not available in 2.8. Also, I have verified the PR I am working on fixes the issue.

#6 Updated by bmbouter over 4 years ago

  • Status changed from ASSIGNED to POST
  • Triaged changed from No to Yes

#7 Updated by bmbouter over 4 years ago

  • Platform Release changed from 2.9.0 to 2.8.5

#8 Updated by bmbouter over 4 years ago

QE: To test this, you need to verify the migration and the fix preventing future issues.

To test the migration, you can install a Pulp that is < 2.8.5 and get a repo into a situation where two package categories are copied into the same repo. At that point you've got a repo with an incorrect unit count. Then upgrade to 2.8.5 and verify that migration has run. The list the unit counts after the upgrade and you should see they have been fixed.

To test the code, after upgrading to 2.8.5 (or a version with this fix) you can try to reproduce as mentioned in the bug report. You'll notice you won't see the unit count error as you did prior to upgrading.

#9 Updated by bmbouter over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#10 Updated by semyers over 4 years ago

  • Status changed from MODIFIED to 5

#11 Updated by pthomas@redhat.com over 4 years ago

Verified
Followed steps in #8

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[/]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client
 rpm:
  MySQL-python-1.2.3-11.el7-x86_64
  PyGreSQL-4.0-9.el7-x86_64
  libdbi-dbd-mysql-0.8.3-16.el7-x86_64
  libdbi-dbd-pgsql-0.8.3-16.el7-x86_64
  mariadb-5.5.44-2.el7-x86_64
  mariadb-bench-5.5.44-2.el7-x86_64
  mariadb-server-5.5.44-2.el7-x86_64
  mariadb-test-5.5.44-2.el7-x86_64
  mysql-connector-java-5.1.25-3.el7-noarch
  mysql-connector-odbc-5.2.5-6.el7-x86_64
  perl-DBD-MySQL-4.023-5.el7-x86_64
  perl-DBD-Pg-2.19.3-4.el7-x86_64
  postgresql-9.2.13-1.el7_1-i686
  postgresql-9.2.13-1.el7_1-x86_64
  postgresql-contrib-9.2.13-1.el7_1-x86_64
  postgresql-docs-9.2.13-1.el7_1-x86_64
  postgresql-jdbc-9.2.1002-5.el7-noarch
  postgresql-odbc-09.03.0100-2.el7-x86_64
  postgresql-plperl-9.2.13-1.el7_1-x86_64
  postgresql-plpython-9.2.13-1.el7_1-x86_64
  postgresql-pltcl-9.2.13-1.el7_1-x86_64
  postgresql-server-9.2.13-1.el7_1-x86_64
  postgresql-test-9.2.13-1.el7_1-x86_64
  python-psycopg2-2.5.1-3.el7-x86_64
  rhdb-utils-9.2.0-5.el7-x86_64
  tcl-pgtcl-2.0.0-5.el7-x86_64

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 2
  Package Group:    26
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" 
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
  databases

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 3
  Package Group:    26
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 4
  Package Group:    30
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# rpm -qa |grep pulp-server
pulp-server-2.8.5-0.1.beta.el7.noarch
[root@hp-ml350pgen8-01 ~]# 
[root@hp-ml350pgen8-01 ~]# 
[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 4
  Package Group:    30
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 4
  Package Group:    30
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# 

#12 Updated by pthomas@redhat.com over 4 years ago

  • Status changed from 5 to 6

#13 Updated by semyers over 4 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 3

#16 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (21)

#17 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF