Project

Profile

Help

Issue #1570

closed

Reading an RPM repo returns incorrect rpm_unit_counts

Added by Ichimonji10 about 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Let's say you have a repository, and you read information about that repository by issuing an HTTP GET request to /pulp/api/v2/repositories/{repo_id}/. The server will return an object like so:

{u'_href': u'/pulp/api/v2/repositories/5f6f8439-ff54-41d7-884f-345b6269615c/',
 u'_id': {u'$oid': u'56a25837e7798903cef0ad8b'},
 u'_ns': u'repos',
 u'content_unit_counts': {u'erratum': 4,
                          u'package_category': 1,
                          u'package_group': 2,
                          u'rpm': 32},
 u'description': None,
 u'display_name': u'5f6f8439-ff54-41d7-884f-345b6269615c',
 u'id': u'5f6f8439-ff54-41d7-884f-345b6269615c',
 u'last_unit_added': u'2016-01-22T16:26:39Z',
 u'last_unit_removed': None,
 u'notes': {u'_repo-type': u'rpm-repo'},
 u'scratchpad': {u'checksum_type': u'sha256'}}

Unfortunately, the ['content_unit_counts']['rpm'] section is frequently incorrect. Here is a test run from my local test system:

$ PULP_SMASH_CONFIG_FILE=$PS_CONFIG python -m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase
...
======================================================================
FAIL: test_unit_count_on_repo (pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase) (unit_type='rpm')
Verify that the sync added the correct number of units to the repo.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/test_sync_publish.py", line 209, in test_unit_count_on_repo
    self.assertEqual(counts.get(unit_type), count)
AssertionError: 34 != 32

----------------------------------------------------------------------
Ran 4 tests in 12.259s

FAILED (failures=1)

As you can see, the "rpm" section has a count of 34, not 32 as expected. You can verify these results by going to the repository and counting the number of RPMs there.


Files


Related issues

Related to Pulp - Issue #1467: Repository unit counts are not always accurate when the plugin authors forget to update themCLOSED - CURRENTRELEASEpcreechActions
Actions #1

Updated by ipanova@redhat.com about 8 years ago

Probably it is related to https://pulp.plan.io/issues/1467

Actions #2

Updated by ipanova@redhat.com about 8 years ago

  • Related to Issue #1467: Repository unit counts are not always accurate when the plugin authors forget to update them added
Actions #3

Updated by ttereshc about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
Actions #4

Updated by ttereshc about 8 years ago

Hi Ichimonji10,
I can't reproduce this issue. I tried adding/removing rpms, syncing repo, every time counters were right.
Pulp Smash tests passed on my machine, I ran them multiple times in a loop (with skipTest commented), no failures.
Could you check that you run tests against recent Pulp version, the similar issue #1467 was fixed about 3 weeks ago.
If tests still fail, any hints how to reproduce it are greatly appreciated.

Actions #5

Updated by Ichimonji10 about 8 years ago

I'm testing against a pair of local Vagrant-managed virtual machines. I leave one VM exactly as is, running the master version of Pulp, and I roll back the second to version 2.7. My procedure for spinning up a fresh VM is as follows:

mkdir pulp-dev
cd pulp-dev
urls=(git@github.com:pulp/pulp{,_{docker,puppet,python,rpm}})
for url in "${urls[@]}"; do
    git clone "${url}"
done
cd pulp
cp Vagrantfile{.example,}
vim Vagrantfile playpen/ansible/vagrant-playbook.yml  # make identifiers unique
vagrant up

My procedure for rolling back a VM to Pulp 2.7 is this script

I most recently spun up VMs this morning, and I re-ran the relvant tests just now. Before running the tests, I commented out relevant test selectors, like so:

    def test_unit_count_on_repo(self):
        """Verify that the sync added the correct number of units to the repo.

        Read the repository and examine its ``content_unit_counts`` attribute.
        Compare these attributes to metadata from the remote repository.
        Expected values are currently hard-coded into this test.
        """
        # if (self.cfg.version >= Version('2.8') and
        #         selectors.bug_is_untestable(1455)):
        #     self.skipTest('https://pulp.plan.io/issues/1455')
        counts = self.repo.get('content_unit_counts', {})
        for unit_type, count in {
                'rpm': 32,
                'erratum': 4,
                'package_group': 2,
                'package_category': 1,
        }.items():
            # if (unit_type == 'rpm' and self.cfg.version >= Version('2.8') and
            #         selectors.bug_is_untestable(1570)):
            #     continue
            with self.subTest(unit_type=unit_type):
                self.assertEqual(counts.get(unit_type), count)

I executed the tests with the following command:

PULP_SMASH_CONFIG_FILE=$PS_CONFIG \
python -m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase

The $PS_CONFIG variable shouldn't be relevant to this bug. It evaluates to a value such as pulp-dev-9.json or pulp-2.7-9.json, and it allows me to test against multiple VMs in parallel. See the attached screenshot.

Actions #6

Updated by Ichimonji10 about 8 years ago

Eugh, the screenshot referenced above is incorrect. See new screenshot.

Actions #7

Updated by jortel@redhat.com about 8 years ago

  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes
Actions #8

Updated by jortel@redhat.com about 8 years ago

  • Priority changed from Normal to High
  • Platform Release changed from 2.8.0 to 2.8.1
Actions #9

Updated by pthomas@redhat.com about 8 years ago

Because of how this issue is affecting satellite and for the fact that its a 2.8 regression, Can we bump the priority on this so that we can get this addressed in 2.8.

Actions #10

Updated by Anonymous about 8 years ago

  • Platform Release changed from 2.8.1 to 2.8.0

Problem has been reproduced by ttereshc and ipanova and are close to identifying the root cause.

Added by ttereshc about 8 years ago

Revision 2d8a07c8 | View on GitHub

Fix rpm_unit_counts, handle upload of the existing rpm unit

closes #1570 https://pulp.plan.io/issues/1570

Actions #11

Updated by ttereshc about 8 years ago

  • Status changed from ASSIGNED to POST

Counters were higher than expected when you tried to upload the file which was already in repo due to the absence of the uniqueness constraint in the database.
This part was fixed https://github.com/pulp/pulp/pull/2392
Then counters became lower than expected because the upload of the existing file was not handled well.

https://github.com/pulp/pulp_rpm/pull/800

Actions #12

Updated by ttereshc about 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #13

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from MODIFIED to 5
Actions #14

Updated by pthomas@redhat.com about 8 years ago

  • Status changed from 5 to 6

verified
[root@ibm-x3550m3-10 ~]# rpm -qa |grep pulp
python-pulp-repoauth-2.8.0-0.9.rc.el7.noarch
pulp-agent-2.8.0-0.9.rc.el7.noarch
pulp-admin-client-2.8.0-0.9.rc.el7.noarch
pulp-nodes-common-2.8.0-0.9.rc.el7.noarch
python-pulp-puppet-common-2.8.0-0.9.rc.el7.noarch
pulp-rpm-consumer-extensions-2.8.0-0.9.rc.el7.noarch
pulp-nodes-child-2.8.0-0.9.rc.el7.noarch
python-pulp-rpm-common-2.8.0-0.9.rc.el7.noarch
python-pulp-bindings-2.8.0-0.9.rc.el7.noarch
python-pulp-oid_validation-2.8.0-0.9.rc.el7.noarch
pulp-rpm-plugins-2.8.0-0.9.rc.el7.noarch
pulp-puppet-admin-extensions-2.8.0-0.9.rc.el7.noarch
pulp-consumer-client-2.8.0-0.9.rc.el7.noarch
pulp-puppet-handlers-2.8.0-0.9.rc.el7.noarch
pulp-puppet-consumer-extensions-2.8.0-0.9.rc.el7.noarch
pulp-nodes-consumer-extensions-2.8.0-0.9.rc.el7.noarch
python-pulp-common-2.8.0-0.9.rc.el7.noarch
pulp-server-2.8.0-0.9.rc.el7.noarch
python-kombu-3.0.33-4.pulp.el7.noarch
pulp-docker-plugins-2.0.0-0.9.rc.el7.noarch
pulp-rpm-admin-extensions-2.8.0-0.9.rc.el7.noarch
pulp-docker-admin-extensions-2.0.0-0.9.rc.el7.noarch
python-pulp-agent-lib-2.8.0-0.9.rc.el7.noarch
pulp-rpm-yumplugins-2.8.0-0.9.rc.el7.noarch
python-pulp-docker-common-2.0.0-0.9.rc.el7.noarch
pulp-puppet-plugins-2.8.0-0.9.rc.el7.noarch
pulp-selinux-2.8.0-0.9.rc.el7.noarch
pulp-rpm-handlers-2.8.0-0.9.rc.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-client-lib-2.8.0-0.9.rc.el7.noarch
[root@ibm-x3550m3-10 ~]#

(env)[preethi@dhcp129-52 pulp-smash]$ python m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase
....
---------------------------------------------------------------------

Ran 4 tests in 29.291s

OK
(env)[preethi@dhcp129-52 pulp-smash]$

Actions #15

Updated by dkliban@redhat.com almost 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #17

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF