Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1570

closed

Reading an RPM repo returns incorrect rpm_unit_counts

Added by Ichimonji10 almost 9 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
ttereshc
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Let's say you have a repository, and you read information about that repository by issuing an HTTP GET request to /pulp/api/v2/repositories/{repo_id}/. The server will return an object like so:

{u'_href': u'/pulp/api/v2/repositories/5f6f8439-ff54-41d7-884f-345b6269615c/',
 u'_id': {u'$oid': u'56a25837e7798903cef0ad8b'},
 u'_ns': u'repos',
 u'content_unit_counts': {u'erratum': 4,
                          u'package_category': 1,
                          u'package_group': 2,
                          u'rpm': 32},
 u'description': None,
 u'display_name': u'5f6f8439-ff54-41d7-884f-345b6269615c',
 u'id': u'5f6f8439-ff54-41d7-884f-345b6269615c',
 u'last_unit_added': u'2016-01-22T16:26:39Z',
 u'last_unit_removed': None,
 u'notes': {u'_repo-type': u'rpm-repo'},
 u'scratchpad': {u'checksum_type': u'sha256'}}

Unfortunately, the ['content_unit_counts']['rpm'] section is frequently incorrect. Here is a test run from my local test system:

$ PULP_SMASH_CONFIG_FILE=$PS_CONFIG python -m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase
...
======================================================================
FAIL: test_unit_count_on_repo (pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase) (unit_type='rpm')
Verify that the sync added the correct number of units to the repo.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/rpm/api_v2/test_sync_publish.py", line 209, in test_unit_count_on_repo
    self.assertEqual(counts.get(unit_type), count)
AssertionError: 34 != 32

----------------------------------------------------------------------
Ran 4 tests in 12.259s

FAILED (failures=1)

As you can see, the "rpm" section has a count of 34, not 32 as expected. You can verify these results by going to the repository and counting the number of RPMs there.

Files

Download all files
Screenshot_2016-01-28_10-20-43.png (1.33 MB) Screenshot_2016-01-28_10-20-43.png Ichimonji10, 01/28/2016 04:21 PM
Screenshot_2016-01-28_10-30-05.png (1.3 MB) Screenshot_2016-01-28_10-30-05.png Ichimonji10, 01/28/2016 04:30 PM

Related issues

Related to Pulp - Issue #1467: Repository unit counts are not always accurate when the plugin authors forget to update themCLOSED - CURRENTRELEASEpcreechActions
Actions
Copy link
 #1

Updated by ipanova@redhat.com almost 9 years ago

Probably it is related to https://pulp.plan.io/issues/1467

Actions
Copy link
 #2

Updated by ipanova@redhat.com almost 9 years ago

  • Related to Issue #1467: Repository unit counts are not always accurate when the plugin authors forget to update them added
Actions
Copy link
 #3

Updated by ttereshc almost 9 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
Actions
Copy link
 #4

Updated by ttereshc almost 9 years ago

Hi Ichimonji10,
I can't reproduce this issue. I tried adding/removing rpms, syncing repo, every time counters were right.
Pulp Smash tests passed on my machine, I ran them multiple times in a loop (with skipTest commented), no failures.
Could you check that you run tests against recent Pulp version, the similar issue #1467 was fixed about 3 weeks ago.
If tests still fail, any hints how to reproduce it are greatly appreciated.

Actions
Copy link
 #5

Updated by Ichimonji10 almost 9 years ago

I'm testing against a pair of local Vagrant-managed virtual machines. I leave one VM exactly as is, running the master version of Pulp, and I roll back the second to version 2.7. My procedure for spinning up a fresh VM is as follows:

mkdir pulp-dev
cd pulp-dev
urls=(git@github.com:pulp/pulp{,_{docker,puppet,python,rpm}})
for url in "${urls[@]}"; do
    git clone "${url}"
done
cd pulp
cp Vagrantfile{.example,}
vim Vagrantfile playpen/ansible/vagrant-playbook.yml  # make identifiers unique
vagrant up

My procedure for rolling back a VM to Pulp 2.7 is this script

I most recently spun up VMs this morning, and I re-ran the relvant tests just now. Before running the tests, I commented out relevant test selectors, like so:

    def test_unit_count_on_repo(self):
        """Verify that the sync added the correct number of units to the repo.

        Read the repository and examine its ``content_unit_counts`` attribute.
        Compare these attributes to metadata from the remote repository.
        Expected values are currently hard-coded into this test.
        """
        # if (self.cfg.version >= Version('2.8') and
        #         selectors.bug_is_untestable(1455)):
        #     self.skipTest('https://pulp.plan.io/issues/1455')
        counts = self.repo.get('content_unit_counts', {})
        for unit_type, count in {
                'rpm': 32,
                'erratum': 4,
                'package_group': 2,
                'package_category': 1,
        }.items():
            # if (unit_type == 'rpm' and self.cfg.version >= Version('2.8') and
            #         selectors.bug_is_untestable(1570)):
            #     continue
            with self.subTest(unit_type=unit_type):
                self.assertEqual(counts.get(unit_type), count)

I executed the tests with the following command:

PULP_SMASH_CONFIG_FILE=$PS_CONFIG \
python -m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase

The $PS_CONFIG variable shouldn't be relevant to this bug. It evaluates to a value such as pulp-dev-9.json or pulp-2.7-9.json, and it allows me to test against multiple VMs in parallel. See the attached screenshot.

Actions
Copy link
 #6

Updated by Ichimonji10 almost 9 years ago

Eugh, the screenshot referenced above is incorrect. See new screenshot.

Actions
Copy link
 #7

Updated by jortel@redhat.com almost 9 years ago

  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes
Actions
Copy link
 #8

Updated by jortel@redhat.com almost 9 years ago

  • Priority changed from Normal to High
  • Platform Release changed from 2.8.0 to 2.8.1
Actions
Copy link
 #9

Updated by pthomas@redhat.com almost 9 years ago

Because of how this issue is affecting satellite and for the fact that its a 2.8 regression, Can we bump the priority on this so that we can get this addressed in 2.8.

Actions
Copy link
 #10

Updated by Anonymous almost 9 years ago

  • Platform Release changed from 2.8.1 to 2.8.0

Problem has been reproduced by ttereshc and ipanova and are close to identifying the root cause.

Added by ttereshc almost 9 years ago

Revision 2d8a07c8 | View on GitHub

Fix rpm_unit_counts, handle upload of the existing rpm unit

closes #1570 https://pulp.plan.io/issues/1570

Actions
Copy link
 #11

Updated by ttereshc almost 9 years ago

  • Status changed from ASSIGNED to POST

Counters were higher than expected when you tried to upload the file which was already in repo due to the absence of the uniqueness constraint in the database.
This part was fixed https://github.com/pulp/pulp/pull/2392
Then counters became lower than expected because the upload of the existing file was not handled well.

https://github.com/pulp/pulp_rpm/pull/800

Actions
Copy link
 #12

Updated by ttereshc almost 9 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #13

Updated by dkliban@redhat.com almost 9 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #14

Updated by pthomas@redhat.com almost 9 years ago

  • Status changed from 5 to 6

verified
[root@ibm-x3550m3-10 ~]# rpm -qa |grep pulp
python-pulp-repoauth-2.8.0-0.9.rc.el7.noarch
pulp-agent-2.8.0-0.9.rc.el7.noarch
pulp-admin-client-2.8.0-0.9.rc.el7.noarch
pulp-nodes-common-2.8.0-0.9.rc.el7.noarch
python-pulp-puppet-common-2.8.0-0.9.rc.el7.noarch
pulp-rpm-consumer-extensions-2.8.0-0.9.rc.el7.noarch
pulp-nodes-child-2.8.0-0.9.rc.el7.noarch
python-pulp-rpm-common-2.8.0-0.9.rc.el7.noarch
python-pulp-bindings-2.8.0-0.9.rc.el7.noarch
python-pulp-oid_validation-2.8.0-0.9.rc.el7.noarch
pulp-rpm-plugins-2.8.0-0.9.rc.el7.noarch
pulp-puppet-admin-extensions-2.8.0-0.9.rc.el7.noarch
pulp-consumer-client-2.8.0-0.9.rc.el7.noarch
pulp-puppet-handlers-2.8.0-0.9.rc.el7.noarch
pulp-puppet-consumer-extensions-2.8.0-0.9.rc.el7.noarch
pulp-nodes-consumer-extensions-2.8.0-0.9.rc.el7.noarch
python-pulp-common-2.8.0-0.9.rc.el7.noarch
pulp-server-2.8.0-0.9.rc.el7.noarch
python-kombu-3.0.33-4.pulp.el7.noarch
pulp-docker-plugins-2.0.0-0.9.rc.el7.noarch
pulp-rpm-admin-extensions-2.8.0-0.9.rc.el7.noarch
pulp-docker-admin-extensions-2.0.0-0.9.rc.el7.noarch
python-pulp-agent-lib-2.8.0-0.9.rc.el7.noarch
pulp-rpm-yumplugins-2.8.0-0.9.rc.el7.noarch
python-pulp-docker-common-2.0.0-0.9.rc.el7.noarch
pulp-puppet-plugins-2.8.0-0.9.rc.el7.noarch
pulp-selinux-2.8.0-0.9.rc.el7.noarch
pulp-rpm-handlers-2.8.0-0.9.rc.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-client-lib-2.8.0-0.9.rc.el7.noarch
[root@ibm-x3550m3-10 ~]#

(env)[preethi@dhcp129-52 pulp-smash]$ python m unittest2 pulp_smash.tests.rpm.api_v2.test_sync_publish.SyncValidFeedTestCase
....
---------------------------------------------------------------------
Ran 4 tests in 29.291s

OK
(env)[preethi@dhcp129-52 pulp-smash]$

Actions
Copy link
 #15

Updated by dkliban@redhat.com almost 9 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #17

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF