Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1979

closed

metadata unit copy action creates incorrect unit count on repo

Added by bmbouter almost 8 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
bmbouter
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
Platform Release:
2.8.5
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 3
Quarter:

Description

This issue affects all metadata rpm types: package categories, package groups, package environments, yum_metadata_file, and package_langpacks. I'll illustrate with langpacks, but the same should be true of all of them.

There can only be 1 package_langpacks associated with a repository. You can see that when copying langpacks over and over the count continues to increment even though each copy overwrites the previous one. Consider this reproducer:

pulp-admin rpm repo create --repo-id=repo_1
pulp-admin rpm repo create --repo-id=repo_2
pulp-admin rpm repo uploads langpacks -i hyphen -n hyphen-%s --repo-id repo_1
pulp-admin rpm repo copy langpacks --from-repo-id=repo_1 --to-repo-id=repo_2
pulp-admin rpm repo copy langpacks --from-repo-id=repo_1 --to-repo-id=repo_2
pulp-admin rpm repo copy langpacks --from-repo-id=repo_1 --to-repo-id=repo_2

Now when listing the langpacks units for repo_2 I expect there still to be 1, but instead it shows 3:

[vagrant@dev ~]$ pulp-admin rpm repo list --repo-id repo_2
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                   repo_2
Display Name:         None
Description:          None
Content Unit Counts:  
  Package Langpacks: 3

There is still in fact only 1 package_langpack which I verified in the database with:

> db.units_package_langpacks.find({"repo_id": "repo_2"}).count()
1

Given ^, the machinery that handles the repo counts is not keep the count correctly. I believe that machinery is here[0].

[0]: https://github.com/pulp/pulp_rpm/blob/d52206bdff5419b5b74976089c605a6cdbeaffc4/plugins/pulp_rpm/plugins/importers/yum/associate.py#L345-L380

Actions
Copy link
 #1

Updated by bmbouter almost 8 years ago

  • Subject changed from Langpacks copy creates incorrect unit count to metadata unit copy action creates incorrect unit count on repo
  • Description updated (diff)

This issue is not unique to langpacks so I'm updating the issue.

Actions
Copy link
 #2

Updated by mhrivnak almost 8 years ago

Actions
Copy link
 #3

Updated by bmbouter almost 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
  • Priority changed from Normal to High
  • Sprint/Milestone set to 21
  • Platform Release set to 2.9.0
Actions
Copy link
 #5

Updated by bmbouter almost 8 years ago

I was able to reproduce the issue with Package Category types, since Package Langpacks were not available in 2.8. Also, I have verified the PR I am working on fixes the issue.

Actions
Copy link
 #6

Updated by bmbouter almost 8 years ago

  • Status changed from ASSIGNED to POST
  • Triaged changed from No to Yes

Added by bmbouter almost 8 years ago

Revision 6ac95ae4 | View on GitHub

Moves migration 31 -> 33 so migrations can be introduced in 2.8.z

This allows the fix for issue 1979 to become migration 31 and the fix for issue 1955 to become migration 32. Both of those will be introduced in 2.8-dev and merged forward to master.

https://pulp.plan.io/issues/1979 https://pulp.plan.io/issues/1955

re #1979 re #1955

Actions
Copy link
 #7

Updated by bmbouter almost 8 years ago

  • Platform Release changed from 2.9.0 to 2.8.5
Actions
Copy link
 #8

Updated by bmbouter almost 8 years ago

QE: To test this, you need to verify the migration and the fix preventing future issues.

To test the migration, you can install a Pulp that is < 2.8.5 and get a repo into a situation where two package categories are copied into the same repo. At that point you've got a repo with an incorrect unit count. Then upgrade to 2.8.5 and verify that migration has run. The list the unit counts after the upgrade and you should see they have been fixed.

To test the code, after upgrading to 2.8.5 (or a version with this fix) you can try to reproduce as mentioned in the bug report. You'll notice you won't see the unit count error as you did prior to upgrading.

Added by bmbouter almost 8 years ago

Revision 077871ad | View on GitHub

Overwriting a metadata unit during copy disassociates first

The metadata units have an overwrite functionaity whereby newer units overwrite older units if they have the same unit key. The unit being overwritten is deleted and the new one is added. The unit being deleted did not have disassociate called so the unit count would be wrong for these unit types after multiple copies.

This comes with a migration which regenerates the unit counts for all repos fixing any incorrect data as a result of the bug.

HEAD of pulp-smash passes and all unit tests pass

https://pulp.plan.io/issues/1979 fixes #1979

Actions
Copy link
 #9

Updated by bmbouter almost 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #10

Updated by semyers almost 8 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #11

Updated by pthomas@redhat.com almost 8 years ago

Verified
Followed steps in #8

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[/]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client
 rpm:
  MySQL-python-1.2.3-11.el7-x86_64
  PyGreSQL-4.0-9.el7-x86_64
  libdbi-dbd-mysql-0.8.3-16.el7-x86_64
  libdbi-dbd-pgsql-0.8.3-16.el7-x86_64
  mariadb-5.5.44-2.el7-x86_64
  mariadb-bench-5.5.44-2.el7-x86_64
  mariadb-server-5.5.44-2.el7-x86_64
  mariadb-test-5.5.44-2.el7-x86_64
  mysql-connector-java-5.1.25-3.el7-noarch
  mysql-connector-odbc-5.2.5-6.el7-x86_64
  perl-DBD-MySQL-4.023-5.el7-x86_64
  perl-DBD-Pg-2.19.3-4.el7-x86_64
  postgresql-9.2.13-1.el7_1-i686
  postgresql-9.2.13-1.el7_1-x86_64
  postgresql-contrib-9.2.13-1.el7_1-x86_64
  postgresql-docs-9.2.13-1.el7_1-x86_64
  postgresql-jdbc-9.2.1002-5.el7-noarch
  postgresql-odbc-09.03.0100-2.el7-x86_64
  postgresql-plperl-9.2.13-1.el7_1-x86_64
  postgresql-plpython-9.2.13-1.el7_1-x86_64
  postgresql-pltcl-9.2.13-1.el7_1-x86_64
  postgresql-server-9.2.13-1.el7_1-x86_64
  postgresql-test-9.2.13-1.el7_1-x86_64
  python-psycopg2-2.5.1-3.el7-x86_64
  rhdb-utils-9.2.0-5.el7-x86_64
  tcl-pgtcl-2.0.0-5.el7-x86_64

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 2
  Package Group:    26
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" 
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
  databases

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 3
  Package Group:    26
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 4
  Package Group:    30
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# rpm -qa |grep pulp-server
pulp-server-2.8.5-0.1.beta.el7.noarch
[root@hp-ml350pgen8-01 ~]# 
[root@hp-ml350pgen8-01 ~]# 
[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 4
  Package Group:    30
  Rpm:              494

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo copy category -f rhel7 -t  issue1979 --str-eq "id=databases" --recursive
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 package_category:
  databases
 package_group:
  mariadb
  mariadb-client
  postgresql
  postgresql-client

[root@hp-ml350pgen8-01 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  rhel7
Display Name:        None
Description:         None
Content Unit Counts: 
  Distribution:           1
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    4620
  Yum Repo Metadata File: 1

Id:                  rhel7-2
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:                1202
  Package Category:       9
  Package Environment:    6
  Package Group:          71
  Rpm:                    10864
  Yum Repo Metadata File: 1

Id:                  issue1979
Display Name:        None
Description:         None
Content Unit Counts: 
  Package Category: 4
  Package Group:    30
  Rpm:              494

[root@hp-ml350pgen8-01 ~]#
Actions
Copy link
 #12

Updated by pthomas@redhat.com almost 8 years ago

  • Status changed from 5 to 6
Actions
Copy link
 #13

Updated by semyers almost 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #15

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 3
Actions
Copy link
 #16

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (21)
Actions
Copy link
 #17

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF