Issue #435
closed
Error during sync of suse errata, missing drpms
Description
Test Repository reflecting suse repomd usage
Description of problem:
During sync of the repomd repositories of suse the following error "Error: 'NoneType' object has no attribute 'text'" appreas during errata sync. The delta rpms seem to be skipped aswell.
Version-Release number of selected component (if applicable):
rpm -qa |grep pulp
python-pulp-bindings-2.4.0-0.18.beta.el6.noarch
pulp-katello-plugins-0.3-1.el6.noarch
python-isodate-0.5.0-1.pulp.el6.noarch
python-pulp-common-2.4.0-0.18.beta.el6.noarch
pulp-nodes-parent-2.4.0-0.18.beta.el6.noarch
pulp-admin-client-2.4.0-0.18.beta.el6.noarch
pulp-selinux-2.4.0-0.18.beta.el6.noarch
createrepo-0.9.9-21.2.pulp.el6.noarch
pulp-server-2.4.0-0.18.beta.el6.noarch
python-pulp-rpm-common-2.4.0-0.18.beta.el6.noarch
python-pulp-puppet-common-2.4.0-0.18.beta.el6.noarch
python-rhsm-1.8.0-1.pulp.el6.x86_64
pulp-rpm-plugins-2.4.0-0.18.beta.el6.noarch
pulp-puppet-plugins-2.4.0-0.18.beta.el6.noarch
mod_wsgi-3.4-1.pulp.el6.x86_64
pulp-rpm-admin-extensions-2.4.0-0.18.beta.el6.noarch
pulp-nodes-common-2.4.0-0.18.beta.el6.noarch
python-pulp-client-lib-2.4.0-0.18.beta.el6.noarch
pulp-puppet-tools-2.4.0-0.18.beta.el6.noarch
m2crypto-0.21.1.pulp-8.el6.x86_64
python-kombu-3.0.15-5.pulp.el6.noarch
How reproducible:
Serve the attached test repository using http and try to sync it into pulp
Steps to Reproduce:
1. serve attached test repository using http
2. sync into pulp
3. pulp-admin rpm repo create --repo-id=test6 --display-name="test6" --feed=http://localhost/suse-test-repo/ --relative-url=test6 --serve-http=true --serve-https=true
4. run pulp-admin rpm repo sync run --repo-id=test6
5. output or task details for error
Actual results:
Sync with rpms & errors
Expected results:
Sync with drpms/rpms/erratum & no errors
- I have verified that its the erratum producing the errors by skipping it in a test.
- I have checked that the test repository works, its not properly signed though :)
Additional info:
[root@axkatello01-1 ~]# pulp-admin rpm repo create --repo-id=test6 --display-name="test6" --feed=http://localhost/suse-test-repo/ --relative-url=test6 --serve-http=true --serve-https=true --skip rpm,drpm,distribution && pulp-admin rpm repo sync run --repo-id=test6
Successfully created repository [test6]
--------------------------------------------------------------------
Synchronizing Repository [test6]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Importing errata...
[-]
... failed
Task Failed
Importer indicated a failed response
[root@axkatello01-1 ~]# pulp-admin tasks details --task-id=bf6b2d1b-b156-4083-8931-dd265af3df63
--------------------------------------------------------------------
Task Details
--------------------------------------------------------------------
Operations: sync
Resources: test6 (repository)
State: Failed
Start Time: 2014-05-23T14:38:22Z
Finish Time: 2014-05-23T14:38:25Z
Result: N/A
Task Id: bf6b2d1b-b156-4083-8931-dd265af3df63
Progress Report:
Yum Importer:
Comps:
State: NOT_STARTED
Content:
Details:
Drpm Done: 0
Drpm Total: 0
Rpm Done: 0
Rpm Total: 0
Error Details:
Items Left: 0
Items Total: 0
Size Left: 0
Size Total: 0
State: FINISHED
Distribution:
Error Details:
Items Left: 0
Items Total: 0
State: SKIPPED
Errata:
Error: 'NoneType' object has no attribute 'text'
State: FAILED
Metadata:
State: FINISHED
Traceback: Traceback (most recent call last): File
"/usr/lib/python2.6/site-packages/celery/app/trace.py", line
240, in trace_task R = retval = fun(*args, **kwargs)
File
"/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py",
line 321, in call return super(Task,
self).__call__(*args, **kwargs) File
"/usr/lib/python2.6/site-packages/celery/app/trace.py", line
437, in protected_call return self.run(*args,
**kwargs) File
"/usr/lib/python2.6/site-packages/pulp/server/tasks/repository
.py", line 210, in sync_with_auto_publish sync_result =
managers.repo_sync_manager().sync(repo_id,
sync_config_override=overrides) File
"/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sy
nc.py", line 113, in sync raise
PulpExecutionException(_('Importer indicated a failed
response')) PulpExecutionException: Importer indicated a
failed response
[root@axkatello01-1 ~]#
Without erratum:
[root@axkatello01-1 ~]# pulp-admin rpm repo create --repo-id=test6 --display-name="test6" --feed=http://localhost/suse-test-repo/ --relative-url=test6 --serve-http=true --serve-https=true --skip erratum && pulp-admin rpm repo sync run --repo-id=test6
Successfully created repository [test6]
--------------------------------------------------------------------
Synchronizing Repository [test6]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[================================ ] 64%
RPMs: 1/2 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing package groups/categories...
[-]
... completed
Task Succeeded
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[==================================================] 100%
2 of 2 items
... completed
Publishing Delta RPMs
[-]
... completed
Publishing Errata
... skipped
Publishing Comps file
[-]
... completed
Publishing Metadata.
[==================================================] 100%
2 of 2 items
... completed
Closing repo metadata
[-]
... completed
Publishing files to web
[-]
... completed
Task Succeeded
+ This bug was cloned from Bugzilla Bug #1100838 +
Files
Updated by kvedulv@kvedulv.de about 9 years ago
More debug output:
Jul 7 13:08:00 katello01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[a25f745e-d882-41aa-93fb-4f562c442942]
Jul 7 13:08:01 katello01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[a25f745e-d882-41aa-93fb-4f562c442942] succeeded in 0.027217833s: u'res
erved_resource_worker-0@katello01.local.dq'
Jul 7 13:08:01 katello01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.sync_with_auto_publish[ed248a84-19a2-40e4-9044-39b2816ec114]
Jul 7 13:08:01 katello01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_release_resource[1b66ef9a-15ed-4122-814f-8fed46c56878]
Jul 7 13:08:01 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from http://localhost/suse-test-repo/.
Jul 7 13:08:01 katello01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): localhost
Jul 7 13:08:02 katello01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): localhost
Jul 7 13:08:02 katello01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): localhost
Jul 7 13:08:02 katello01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): localhost
Jul 7 13:08:02 katello01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): localhost
Jul 7 13:08:02 katello01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): localhost
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call last):
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 133, in run
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: self.get_errata(metadata_files)
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 462, in get
errata
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: self.save_fileless_units(errata_file_handle, updateinfo.PACKAGE_TAG, updateinfo.process_packag
e_element)
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 522, in save_fileless_units
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: to_save = existing.check_repo(wanted, self.sync_conduit.get_units)
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/existing.py", line 37, in check_repo
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: sorted_units = _sort_by_type(wanted)
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/existing.py", line 162, in _sort_by_type
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: for unit in wanted:
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 520, in <genexpr>
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: wanted = (model.as_named_tuple for model in package_info_generator)
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 63, in package_list_generator
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: package_info = process_func(element)
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/updateinfo.py", line 39, in process_package_element
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: 'pkglist': map(_parse_collection, element.find('pkglist') or []),
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/updateinfo.py", line 82, in _parse_collection
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: 'name': element.find('name').text,
Jul 7 13:08:03 katello01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: AttributeError: 'NoneType' object has no attribute 'text'
Jul 7 13:08:03 katello01 pulp: pulp.server.event.http:INFO: {'call_report': {u'exception': None, u'task_type': u'pulp.server.tasks.repository.sync_with_auto_publish', u'task_id': u'ed248a84-19a2-40e4-9044-39b2816ec114', u'tags': [u'pulp:repository:test6', u'pulp:action:sync'], u'finish_time': None, u'_ns': u'task_status', u'start_time': u'2014-07-07T11:08:01Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': {u'yum_importer': {u'content': {u'size_total': 0, u'items_left': 0, u'items_total': 0, u'state': u'FINISHED', u'size_left': 0, u'details': {u'rpm_total': 0, u'rpm_done': 0, u'drpm_total': 0, u'drpm_done': 0}, u'error_details': []}, u'comps': {u'state': u'NOT_STARTED'}, u'distribution': {u'items_total': 0, u'state': u'SKIPPED', u'error_details': [], u'items_left': 0}, u'errata': {u'state': u'FAILED', u'error': u"'NoneType' object has no attribute 'text'"}, u'metadata': {u'state': u'FINISHED'}}}, u'queue': u'reserved_resource_worker-0@katello01.local.dq', u'state': u'running', u'result': None, u'error': None, u'_id': ObjectId('53ba7f911fd87d1390414324'), u'id': u'53ba7f910ab73b04e99340b0'}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'test6', 'traceback': None, 'started': '2014-07-07T13:08:01+02:00', '_ns': u'repo_sync_results', 'completed': '2014-07-07T13:08:03+02:00', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FINISHED'}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'state': 'SKIPPED'}, 'errata': {'state': 'FAILED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 2, 'result': 'failed', 'updated_count': 0, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FINISHED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'SKIPPED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'FAILED', 'error': "'NoneType
Jul 7 13:08:03 katello01 pulp: pulp.server.event.http:INFO: ' object has no attribute 'text'"}, 'metadata': {'state': 'FINISHED'}}, 'id': '53ba7f930ab73b06037945ab', 'removed_count': 0}}
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: Task pulp.server.tasks.repository.sync_with_auto_publish[ed248a84-19a2-40e4-9044-39b2816ec114] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: Traceback (most recent call last):
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs)
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 306, incall_
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs)
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in protected_call
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: return self.run(*args, **kwargs)
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py", line 210, in sync_with_auto_publish
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: sync_result = managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides)
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 113, in sync
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: raise PulpExecutionException(_('Importer indicated a failed response'))
Jul 7 13:08:03 katello01 pulp: celery.worker.job:ERROR: PulpExecutionException: Importer indicated a failed response
Jul 7 13:08:03 katello01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[346de107-6731-41c7-953c-aaa872bdee79]
Jul 7 13:08:03 katello01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_release_resource[1b66ef9a-15ed-4122-814f-8fed46c56878] succeeded in 0.0755167460001s: None
Jul 7 13:08:03 katello01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[346de107-6731-41c7-953c-aaa872bdee79] succeeded in 0.0414652109999s: None
Jul 7 13:08:05 katello01 pulp: pulp.server.event.http:WARNING: Error response from HTTP notifier: {"displayMessage":"Couldn't find repository 'test6'","errors":["Couldn't find repository 'test6'"]}
+ This comment was cloned from Bugzilla #1100838 comment 1 +
Updated by kvedulv@kvedulv.de about 9 years ago
collection in pkglist has no name attribute in that SUSE repo, that's what the parser doesn't like.
+ This comment was cloned from Bugzilla #1100838 comment 2 +
Updated by kvedulv@kvedulv.de about 9 years ago
corresponding SUSE bug at https://bugzilla.novell.com/show_bug.cgi?id=886444
+ This comment was cloned from Bugzilla #1100838 comment 3 +
Updated by kvedulv@kvedulv.de about 9 years ago
Possible fix for errata import: https://github.com/pulp/pulp_rpm/pull/525
+ This comment was cloned from Bugzilla #1100838 comment 4 +
Updated by kvedulv@kvedulv.de about 9 years ago
Seperate BZ #1120215 opened for the delta RPM problem.
+ This comment was cloned from Bugzilla #1100838 comment 5 +
Updated by bcourt about 9 years ago
The errata sync appears to have already been fixed in the 2.6
+ This comment was cloned from Bugzilla #1100838 comment 6 +
Updated by kvedulv@kvedulv.de about 9 years ago
Ah, yes... https://github.com/pulp/pulp_rpm/pull/568 was merged.
+ This comment was cloned from Bugzilla #1100838 comment 7 +
Updated by bcourt about 9 years ago
https://github.com/pulp/pulp_rpm/pull/629
+ This comment was cloned from Bugzilla #1100838 comment 8 +
Updated by pthomas@redhat.com about 9 years ago
- Status changed from 5 to ASSIGNED
Sync seems to be hanging at Importing package groups/categories and does not move to the publish step
[root@gizmo ~]# pulp-admin rpm repo create --repo-id=test6 --display-name="test6" --feed=http://ftp5.gwdg.de/pub/opensuse/discontinued/update/11.4/ --relative-url=test6 --serve-http=true --serve-https=true --skip rpm,drpm,distribution && pulp-admin rpm repo sync run --repo-id=test6
Successfully created repository [test6]
[root@gizmo ~]# pulp-admin rpm repo sync run --repo-id test6
+----------------------------------------------------------------------+
Synchronizing Repository [test6]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[\]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
,\pre>
Updated by bcourt about 9 years ago
- Status changed from ASSIGNED to 5
Hi, I just ran a sync & publish on this repo and it appears to have worked properly though the sync portion took about 6 minutes. Did you have an error in the log files?
[bcourt@bcourt pulp]$ pulp-admin rpm repo create --repo-id suse --feed http://ftp5.gwdg.de/pub/opensuse/
discontinued/update/11.4/ --relative-url suse --skip rpm,drpm,distribution
Successfully created repository [suse]
[bcourt@bcourt pulp]$ pulp-admin rpm repo sync run --repo-id suse
+----------------------------------------------------------------------+
Synchronizing Repository [suse]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Importing errata...
[|]
... completed
Importing package groups/categories...
[-]
... completed
Task Succeeded
Initializing repo metadata
[-]
... completed
Publishing Distribution files
... skipped
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
526 of 526 items
... completed
Publishing Comps file
[-]
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
[bcourt@bcourt pulp]$
Updated by pthomas@redhat.com about 9 years ago
- Status changed from 5 to 6
verified
Looks like sync did complete for me without any errors
[root@celeno ~]# pulp-admin rpm repo create --repo-id suse --feed http://ftp5.gwdg.de/pub/opensuse/discontinued/update/11.4/ --relative-url suse --skip rpm,drpm,distribution
Successfully created repository [suse]
[root@celeno ~]#
[root@celeno ~]#
[root@celeno ~]#
[root@celeno ~]# pulp-admin rpm repo sync run --repo-id suse
+----------------------------------------------------------------------+
Synchronizing Repository [suse]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[\]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Importing errata...
[|]
... completed
Importing package groups/categories...
[-]
... completed
Task Succeeded
Initializing repo metadata
[-]
... completed
Publishing Distribution files
... skipped
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
526 of 526 items
... completed
Publishing Comps file
[-]
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
[root@celeno ~]#
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
.