Project

Profile

Help

Issue #1570

Reading an RPM repo returns incorrect rpm_unit_counts

Added by Ichimonji10 over 5 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Let's say you have a repository, and you read information about that repository by issuing an HTTP GET request to /pulp/api/v2/repositories/{repo_id}/. The server will return an object like so:

{u'_href': u'/pulp/api/v2/repositories/5f6f8439-ff54-41d7-884f-345b6269615c/',
 u'_id': {u'$oid': u'56a25837e7798903cef0ad8b'},
 u'_ns': u'repos',
 u'content_unit_counts': {u'erratum': 4,
                          u'package_category': 1,
                          u'package_group': 2,
                          u'rpm': 32},
 u'description': None,
 u'display_name': u'5f6f8439-ff54-41d7-884f-345b6269615c',
 u'id': u'5f6f8439-ff54-41d7-884f-345b6269615c',
 u'last_unit_added': u'2016-01-22T16:26:39Z',
 u'last_unit_removed': None,
 u'notes': {u'_repo-type': u'rpm-repo'},
 u'scratchpad': {u'checksum_type': u'sha256'}}

Unfortunately, the ['content_unit_counts']['rpm'] section is frequently incorrect. Here is a test run from my local test system:

$ PULP_SMASH_CONFIG_FILE=$PS_CONFIG python -m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase
...
======================================================================
FAIL: test_unit_count_on_repo (pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase) (unit_type='rpm')
Verify that the sync added the correct number of units to the repo.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/test_sync_publish.py", line 209, in test_unit_count_on_repo
    self.assertEqual(counts.get(unit_type), count)
AssertionError: 34 != 32

----------------------------------------------------------------------
Ran 4 tests in 12.259s

FAILED (failures=1)

As you can see, the "rpm" section has a count of 34, not 32 as expected. You can verify these results by going to the repository and counting the number of RPMs there.


Related issues

Related to Pulp - Issue #1467: Repository unit counts are not always accurate when the plugin authors forget to update themCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 2d8a07c8 View on GitHub
Added by ttereshc over 5 years ago

Fix rpm_unit_counts, handle upload of the existing rpm unit

closes #1570 https://pulp.plan.io/issues/1570

History

#1 Updated by ipanova@redhat.com over 5 years ago

Probably it is related to https://pulp.plan.io/issues/1467

#2 Updated by ipanova@redhat.com over 5 years ago

  • Related to Issue #1467: Repository unit counts are not always accurate when the plugin authors forget to update them added

#3 Updated by ttereshc over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

#4 Updated by ttereshc over 5 years ago

Hi Ichimonji10,
I can't reproduce this issue. I tried adding/removing rpms, syncing repo, every time counters were right.
Pulp Smash tests passed on my machine, I ran them multiple times in a loop (with skipTest commented), no failures.
Could you check that you run tests against recent Pulp version, the similar issue #1467 was fixed about 3 weeks ago.
If tests still fail, any hints how to reproduce it are greatly appreciated.

#5 Updated by Ichimonji10 over 5 years ago

250

I'm testing against a pair of local Vagrant-managed virtual machines. I leave one VM exactly as is, running the master version of Pulp, and I roll back the second to version 2.7. My procedure for spinning up a fresh VM is as follows:

mkdir pulp-dev
cd pulp-dev
urls=(git@github.com:pulp/pulp{,_{docker,puppet,python,rpm}})
for url in "${urls[@]}"; do
    git clone "${url}"
done
cd pulp
cp Vagrantfile{.example,}
vim Vagrantfile playpen/ansible/vagrant-playbook.yml  # make identifiers unique
vagrant up

My procedure for rolling back a VM to Pulp 2.7 is this script

I most recently spun up VMs this morning, and I re-ran the relvant tests just now. Before running the tests, I commented out relevant test selectors, like so:

    def test_unit_count_on_repo(self):
        """Verify that the sync added the correct number of units to the repo.

        Read the repository and examine its ``content_unit_counts`` attribute.
        Compare these attributes to metadata from the remote repository.
        Expected values are currently hard-coded into this test.
        """
        # if (self.cfg.version >= Version('2.8') and
        #         selectors.bug_is_untestable(1455)):
        #     self.skipTest('https://pulp.plan.io/issues/1455')
        counts = self.repo.get('content_unit_counts', {})
        for unit_type, count in {
                'rpm': 32,
                'erratum': 4,
                'package_group': 2,
                'package_category': 1,
        }.items():
            # if (unit_type == 'rpm' and self.cfg.version >= Version('2.8') and
            #         selectors.bug_is_untestable(1570)):
            #     continue
            with self.subTest(unit_type=unit_type):
                self.assertEqual(counts.get(unit_type), count)

I executed the tests with the following command:

PULP_SMASH_CONFIG_FILE=$PS_CONFIG \
python -m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase

The $PS_CONFIG variable shouldn't be relevant to this bug. It evaluates to a value such as pulp-dev-9.json or pulp-2.7-9.json, and it allows me to test against multiple VMs in parallel. See the attached screenshot.

#6 Updated by Ichimonji10 over 5 years ago

250

Eugh, the screenshot referenced above is incorrect. See new screenshot.

#7 Updated by jortel@redhat.com over 5 years ago

  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes

#8 Updated by jortel@redhat.com over 5 years ago

  • Priority changed from Normal to High
  • Platform Release changed from 2.8.0 to 2.8.1

#9 Updated by pthomas@redhat.com over 5 years ago

Because of how this issue is affecting satellite and for the fact that its a 2.8 regression, Can we bump the priority on this so that we can get this addressed in 2.8.

#10 Updated by Anonymous over 5 years ago

  • Platform Release changed from 2.8.1 to 2.8.0

Problem has been reproduced by ttereshc and ipanova and are close to identifying the root cause.

#11 Updated by ttereshc over 5 years ago

  • Status changed from ASSIGNED to POST

Counters were higher than expected when you tried to upload the file which was already in repo due to the absence of the uniqueness constraint in the database.
This part was fixed https://github.com/pulp/pulp/pull/2392
Then counters became lower than expected because the upload of the existing file was not handled well.

https://github.com/pulp/pulp_rpm/pull/800

#12 Updated by ttereshc over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#13 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from MODIFIED to 5

#14 Updated by pthomas@redhat.com over 5 years ago

  • Status changed from 5 to 6

verified
[root@ibm-x3550m3-10 ~]# rpm -qa |grep pulp
python-pulp-repoauth-2.8.0-0.9.rc.el7.noarch
pulp-agent-2.8.0-0.9.rc.el7.noarch
pulp-admin-client-2.8.0-0.9.rc.el7.noarch
pulp-nodes-common-2.8.0-0.9.rc.el7.noarch
python-pulp-puppet-common-2.8.0-0.9.rc.el7.noarch
pulp-rpm-consumer-extensions-2.8.0-0.9.rc.el7.noarch
pulp-nodes-child-2.8.0-0.9.rc.el7.noarch
python-pulp-rpm-common-2.8.0-0.9.rc.el7.noarch
python-pulp-bindings-2.8.0-0.9.rc.el7.noarch
python-pulp-oid_validation-2.8.0-0.9.rc.el7.noarch
pulp-rpm-plugins-2.8.0-0.9.rc.el7.noarch
pulp-puppet-admin-extensions-2.8.0-0.9.rc.el7.noarch
pulp-consumer-client-2.8.0-0.9.rc.el7.noarch
pulp-puppet-handlers-2.8.0-0.9.rc.el7.noarch
pulp-puppet-consumer-extensions-2.8.0-0.9.rc.el7.noarch
pulp-nodes-consumer-extensions-2.8.0-0.9.rc.el7.noarch
python-pulp-common-2.8.0-0.9.rc.el7.noarch
pulp-server-2.8.0-0.9.rc.el7.noarch
python-kombu-3.0.33-4.pulp.el7.noarch
pulp-docker-plugins-2.0.0-0.9.rc.el7.noarch
pulp-rpm-admin-extensions-2.8.0-0.9.rc.el7.noarch
pulp-docker-admin-extensions-2.0.0-0.9.rc.el7.noarch
python-pulp-agent-lib-2.8.0-0.9.rc.el7.noarch
pulp-rpm-yumplugins-2.8.0-0.9.rc.el7.noarch
python-pulp-docker-common-2.0.0-0.9.rc.el7.noarch
pulp-puppet-plugins-2.8.0-0.9.rc.el7.noarch
pulp-selinux-2.8.0-0.9.rc.el7.noarch
pulp-rpm-handlers-2.8.0-0.9.rc.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-client-lib-2.8.0-0.9.rc.el7.noarch
[root@ibm-x3550m3-10 ~]#

(env)[preethi@dhcp129-52 pulp-smash]$ python m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase
....
---------------------------------------------------------------------

Ran 4 tests in 29.291s

OK
(env)[preethi@dhcp129-52 pulp-smash]$

#15 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#17 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF