Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1621

closed

sync with "remove-missing" failing.

Added by paji@redhat.com about 8 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
jortel@redhat.com
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Get the zoo5 repo from -> https://github.com/Katello/runcible/tree/master/test/fixtures/repositories/zoo5
Steps below will assume file:///var/www/repositories/zoo5 is the destination

$ pulp-admin rpm repo create --repo-id=reintermediate --feed=file:///var/www/repositories/zoo5 --remove-missing=true
$ pulp-admin rpm repo sync run --repo-id=reintermediate

+----------------------------------------------------------------------+
               Synchronizing Repository [reintermediate]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Failed

Importer indicated a failed response

Related issues

Related to RPM Support - Issue #1610: added updated and removed count all 0 on first syncCLOSED - CURRENTRELEASEamacdona@redhat.comActions
Actions
Copy link
 #1

Updated by paji@redhat.com about 8 years ago

/var/log/message trace

Feb  2 11:05:02 katello-yoda pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[0b5ae105-36e3-4b2e-9c38-ebf7681602da]
Feb  2 11:05:02 katello-yoda pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[a2ed2eca-248c-42d9-b37e-41aa9730486f]
Feb  2 11:05:02 katello-yoda pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[30c6ee83-739d-4afa-badb-6c7d8922e82d]
Feb  2 11:05:02 katello-yoda pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[0b5ae105-36e3-4b2e-9c38-ebf7681602da] succeeded in 0.0355760769999s: None
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///var/www/test_repos/zoo/.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///var/www/test_repos/zoo/.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from file:///var/www/test_repos/zoo/.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: file:///var/www/test_repos/zoo/
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.purge:INFO: Removing missing units.
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352) 'RPM' object has no attribute 'as_named_tuple'
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352) Traceback (most recent call last):
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 227, in run
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)     self.update_content(metadata_files, url)
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 463, in update_content
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)     purge.purge_unwanted_units(metadata_files, self.conduit, self.config)
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 37, in purge_unwanted_units
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)     remove_missing_rpms(metadata_files, conduit)
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 93, in remove_missing_rpms
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)     primary.process_package_element)
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 260, in get_remote_units
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352)     named_tuple = model.as_named_tuple
Feb  2 11:05:02 katello-yoda pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2430-60352) AttributeError: 'RPM' object has no attribute 'as_named_tuple'
Feb  2 11:05:02 katello-yoda pulp: pulp.server.event.http:INFO: (2430-60352) {'call_report': {'exception': None, 'task_type': u'pulp.server.managers.repo.sync.sync', '_href': u'/pulp/api/v2/tasks/a2ed2eca-248c-42d9-b37e-41aa9730486f/', 'task_id': u'a2ed2eca-248c-42d9-b37e-41aa9730486f', 'tags': [u'pulp:repository:reintermediate', u'pulp:action:sync'], 'finish_time': None, '_ns': u'task_status', 'start_time': u'2016-02-02T16:05:02Z', 'traceback': None, 'spawned_tasks': [], 'progress_report': {u'yum_importer': {u'content': {u'size_total': 0, u'items_left': 0, u'items_total': 0, u'state': u'FAILED', u'size_left': 0, u'details': {u'rpm_total': 0, u'rpm_done': 0, u'drpm_total': 0, u'drpm_done': 0}, u'error': u"'RPM' object has no attribute 'as_named_tuple'", u'error_details': []}, u'comps': {u'state': u'NOT_STARTED'}, u'distribution': {u'items_total': 0, u'state': u'NOT_STARTED', u'error_details': [], u'items_left': 0}, u'errata': {u'state': u'NOT_STARTED'}, u'metadata': {u'state': u'FINISHED'}}}, 'state': u'running', 'worker_name': u'reserved_resource_worker-1@katello-yoda.example.com', 'result': None, 'error': None, '_id': ObjectId('56b0d3aee899ef398e4646de'), 'id': '56b0d3aee899ef398e4646de'}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': ObjectId('56b0d304de0403047a2d01b2'), 'exception': None, 'repo_id': u'reintermediate', 'traceback': None, 'started': '2016-02-02T16:05:02Z', '_ns': u'repo_sync_results', 'completed': '2016-02-02T16:05:02Z', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FAILED'}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'state': 'NOT_STARTED'}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 0, 'result': 'failed', 'updated_count': 0, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FAILED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error': "'RPM' object has no attribute 'as_named_tuple'", 'error_details': []}, 'comps': {'
Feb  2 11:05:02 katello-yoda pulp: pulp.server.event.http:INFO: (2430-60352) state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'NOT_STARTED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FINISHED'}}, 'id': '56b0d3aede0403097e5b3cfc', 'removed_count': 0}}
Actions
Copy link
 #2

Updated by jsherril@redhat.com about 8 years ago

  • Subject changed from sync with "remove-misisng" failing. to sync with "remove-missing" failing.
Actions
Copy link
 #3

Updated by ipanova@redhat.com about 8 years ago

So during mongoengine conversion as_named_tuple was removed https://github.com/pulp/pulp_rpm/commit/944a05f7c131893a3fd46f1a67a53e4a618d95d6#diff-df55e4e5c922fdb2a0fefedecd7013c5L40 and replaced with "unit_key_as_named_tuple"

But replacing as_named_tuple-->unit_key_as_named_tuple prodices another traceback:

Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/home/ipanova/pulp_development/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/purge.py", line 203, in remove_missing_units
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     for unit in get_existing_units(model, conduit.get_units):
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/home/ipanova/pulp_development/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/purge.py", line 228, in get_existing_units
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     return unit_search_func(criteria)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/home/ipanova/pulp_development/pulp/server/pulp/plugins/conduits/mixins.py", line 173, in get_units
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     return do_get_repo_units(self.repo_id, criteria, self.exception_class, as_generator)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/home/ipanova/pulp_development/pulp/server/pulp/plugins/conduits/mixins.py", line 688, in do_get_repo_units
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     units = association_query_manager.get_units(repo_id, criteria=criteria, as_generator=True)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/home/ipanova/pulp_development/pulp/server/pulp/server/managers/repo/unit_association_query.py", line 97, in get_units
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     for association in unit_associations_generator:
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     if len(self.__data) or self._refresh():
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     self.__read_concern))
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     **kwargs)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     exhaust)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     return func(*args, **kwargs)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     set_slave_okay, sock_info.is_mongos, use_find_cmd)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)   File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640)     spec, self.fields, self.codec_options)
Feb  2 17:17:42 ina pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (27636-96640) ImporterConduitException: Cannot encode object: <mongoengine.fields.StringField object at 0x7f7569dcbed0>
Actions
Copy link
 #4

Updated by amacdona@redhat.com about 8 years ago

  • Related to Issue #1610: added updated and removed count all 0 on first sync added
Actions
Copy link
 #5

Updated by amacdona@redhat.com about 8 years ago

On completion of this issue, please revisit 1610 to ensure that the removed_counts work.

Actions
Copy link
 #6

Updated by amacdona@redhat.com about 8 years ago

  • Platform Release set to 2.8.0
Actions
Copy link
 #7

Updated by jortel@redhat.com about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes
Actions
Copy link
 #8

Updated by jortel@redhat.com about 8 years ago

  • Status changed from ASSIGNED to POST

Added by jortel@redhat.com about 8 years ago

Revision e856ea13 | View on GitHub

Fixes --remove-missing issues caused by references to as_named_tuple. closes #1621

Actions
Copy link
 #9

Updated by jortel@redhat.com about 8 years ago

wrote:

On completion of this issue, please revisit 1610 to ensure that the removed_counts work.

Verified.

Actions
Copy link
 #10

Updated by jortel@redhat.com about 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #11

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #12

Updated by pthomas@redhat.com about 8 years ago

  • Status changed from 5 to 6

verified

[root@mgmt3 ~]# rpm -qa pulp-server
pulp-server-2.8.0-0.6.beta.git.40.28675fb.el7.noarch
[root@mgmt3 ~]# 

[preethi@dhcp129-52 pulp-smash]$ python -m unittest2 pulp_smash.tests.rpm.api_v2.test_remove_unit
....
----------------------------------------------------------------------
Ran 4 tests in 55.366s

OK
(env)[preethi@dhcp129-52 pulp-smash]$
Actions
Copy link
 #13

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #15

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF